[experiments] Daily Experiment Report — 2026-09-16 #61311

2026-09-16T08:02:52Z

github-actions[bot]
Bot Sep 16, 2026

🧪 Daily Experiment Report — 2026-09-16

51 A/B experiments analysed across 48 workflows in github/gh-aw. All decisions are EXTEND (37) or INCONCLUSIVE (14) — no experiment currently has enough decision-quality evidence to PROMOTE or REJECT. 24 experiments have reached min_samples (READY), but native metric observations are missing for most of them.

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis	24
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0
Tracking issues configured	0 (no experiment declares an `issue:` field)

🟢 Ready for Analysis (24 experiments)

All variants reached min_samples, but 24/24 still resolve to EXTEND/INCONCLUSIVE — 20 lack native metric observations (insufficient_observations or guardrail_unsupported), 4 are multi-variant (unsupported_multi_variant, decision engine currently supports exactly 2 variants).

`prompt_compression` · `agent-performance-analyzer.md`

Readiness: 🟢 READY · Variants: caveman vs verbose · Total runs: 117 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in aic. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%

Variant	Runs	Progress
`caveman`	63	██████████ 63/20 (100%)
`verbose`	54	██████████ 54/20 (100%)

Experiment : prompt_compression
Workflow   : agent-performance-analyzer.md
Metric     : aic
min_samples: 20 per variant | Total runs: 117
  caveman              n=63    (53.8% observed vs 50.0% expected)
  verbose              n=54    (46.2% observed vs 50.0% expected)

Balance test  : chi_square=0.692, p=0.4103, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `agent-persona-explorer.md`

Readiness: 🟢 READY · Variants: batch vs per_scenario · Total runs: 117 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in aic or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss

Variant	Runs	Progress
`batch`	62	██████████ 62/20 (100%)
`per_scenario`	55	██████████ 55/20 (100%)

Experiment : sub_agent_strategy
Workflow   : agent-persona-explorer.md
Metric     : aic
min_samples: 20 per variant | Total runs: 117
  batch                n=62    (53.0% observed vs 50.0% expected)
  per_scenario         n=55    (47.0% observed vs 50.0% expected)

Balance test  : chi_square=0.419, p=0.5250, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`tone_variant` · `aw-failure-investigator.md`

Readiness: 🟢 READY · Variants: assertive vs clinical vs narrative · Total runs: 421 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.

Variant	Runs	Progress
`assertive`	145	██████████ 145/20 (100%)
`clinical`	139	██████████ 139/20 (100%)
`narrative`	137	██████████ 137/20 (100%)

Experiment : tone_variant
Workflow   : aw-failure-investigator.md
Metric     : output_length_chars
min_samples: 20 per variant | Total runs: 421
  assertive            n=145   (34.4% observed vs 33.3% expected)
  clinical             n=139   (33.0% observed vs 33.3% expected)
  narrative            n=137   (32.5% observed vs 33.3% expected)

Balance test  : chi_square=0.247, p=0.8795, is_balanced=True
Guardrails    : not configured
Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

`tone_variant` · `breaking-change-checker.md`

Readiness: 🟢 READY · Variants: neutral vs urgent · Total runs: 43 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in issue_engagement_rate. H1: urgent increases issue_engagement_rate by >=7 percentage points without degrading run_success_rate.

Variant	Runs	Progress
`neutral`	22	██████████ 22/20 (100%)
`urgent`	21	██████████ 21/20 (100%)

Experiment : tone_variant
Workflow   : breaking-change-checker.md
Metric     : issue_engagement_rate
min_samples: 20 per variant | Total runs: 43
  neutral              n=22    (51.2% observed vs 50.0% expected)
  urgent               n=21    (48.8% observed vs 50.0% expected)

Balance test  : chi_square=0.023, p=0.8519, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `ci-coach.md`

Readiness: 🟢 READY · Variants: concise vs detailed · Total runs: 88 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

Variant	Runs	Progress
`concise`	37	██████████ 37/20 (100%)
`detailed`	51	██████████ 51/20 (100%)

Experiment : prompt_style
Workflow   : ci-coach.md
Metric     : token_count_per_run
min_samples: 20 per variant | Total runs: 88
  concise              n=37    (42.0% observed vs 50.0% expected)
  detailed             n=51    (58.0% observed vs 50.0% expected)

Balance test  : chi_square=0.000, p=0.0000, is_balanced=True
Guardrails    : FAIL
Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

`sub_agent_strategy` · `daily-agentrx-trace-optimizer.md`

Readiness: 🟢 READY · Variants: single_agent vs sub_agents · Total runs: 77 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in issue quality or run success rate. H1: sub_agents variant yields higher evidence completeness score with equal or lower token cost

Variant	Runs	Progress
`single_agent`	39	██████████ 39/20 (100%)
`sub_agents`	38	██████████ 38/20 (100%)

Experiment : sub_agent_strategy
Workflow   : daily-agentrx-trace-optimizer.md
Metric     : issue_evidence_completeness
min_samples: 20 per variant | Total runs: 77
  single_agent         n=39    (50.6% observed vs 50.0% expected)
  sub_agents           n=38    (49.4% observed vs 50.0% expected)

Balance test  : chi_square=0.013, p=0.8752, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `daily-astrostylelite-markdown-spellcheck.md`

Readiness: 🟢 READY · Variants: concise vs detailed · Total runs: 135 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.

Variant	Runs	Progress
`concise`	68	██████████ 68/20 (100%)
`detailed`	67	██████████ 67/20 (100%)

Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck.md
Metric     : aic
min_samples: 20 per variant | Total runs: 135
  concise              n=68    (50.4% observed vs 50.0% expected)
  detailed             n=67    (49.6% observed vs 50.0% expected)

Balance test  : chi_square=0.007, p=0.8918, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `daily-community-attribution.md`

Readiness: 🟢 READY · Variants: concise vs verbose · Total runs: 133 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`concise`	67	██████████ 67/20 (100%)
`verbose`	66	██████████ 66/20 (100%)

Experiment : prompt_style
Workflow   : daily-community-attribution.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 133
  concise              n=67    (50.4% observed vs 50.0% expected)
  verbose              n=66    (49.6% observed vs 50.0% expected)

Balance test  : chi_square=0.008, p=0.8915, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`reasoning_depth` · `daily-fact.md`

Readiness: 🟢 READY · Variants: multi_candidate vs single_pass · Total runs: 88 · min_samples: 30

📈 Statistics — hypothesis, samples, balance test

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

Variant	Runs	Progress
`multi_candidate`	35	██████████ 35/30 (100%)
`single_pass`	53	██████████ 53/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-fact.md
Metric     : discussion_reaction_count
min_samples: 30 per variant | Total runs: 88
  multi_candidate      n=35    (39.8% observed vs 50.0% expected)
  single_pass          n=53    (60.2% observed vs 50.0% expected)

Balance test  : chi_square=3.682, p=0.0520, is_balanced=True
Guardrails    : FAIL
Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

`prompt_style` · `daily-news.md`

Readiness: 🟢 READY · Variants: concise vs detailed · Total runs: 104 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score

Variant	Runs	Progress
`concise`	56	██████████ 56/20 (100%)
`detailed`	48	██████████ 48/20 (100%)

Experiment : prompt_style
Workflow   : daily-news.md
Metric     : aic
min_samples: 20 per variant | Total runs: 104
  concise              n=56    (53.8% observed vs 50.0% expected)
  detailed             n=48    (46.2% observed vs 50.0% expected)

Balance test  : chi_square=0.615, p=0.4386, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`reasoning_depth` · `daily-security-red-team.md`

Readiness: 🟢 READY · Variants: iterative vs single_pass · Total runs: 127 · min_samples: 30

📈 Statistics — hypothesis, samples, balance test

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

Variant	Runs	Progress
`iterative`	77	██████████ 77/30 (100%)
`single_pass`	50	██████████ 50/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Metric     : false_positive_rate
min_samples: 30 per variant | Total runs: 127
  iterative            n=77    (60.6% observed vs 50.0% expected)
  single_pass          n=50    (39.4% observed vs 50.0% expected)

Balance test  : chi_square=5.740, p=0.0158, is_balanced=False
Guardrails    : FAIL
Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

`output_format` · `deep-report.md`

Readiness: 🟢 READY · Variants: annotated_brief vs executive_brief vs full_briefing vs ste · Total runs: 193 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in discussion engagement or token cost. H1: executive_brief reduces token usage by ≥20% without reducing engagement; annotated_brief improves actionability; ste improves clarity while reducing token usage.

Variant	Runs	Progress
`annotated_brief`	46	██████████ 46/20 (100%)
`executive_brief`	70	██████████ 70/20 (100%)
`full_briefing`	52	██████████ 52/20 (100%)
`ste`	25	██████████ 25/20 (100%)

Experiment : output_format
Workflow   : deep-report.md
Metric     : token_count
min_samples: 20 per variant | Total runs: 193
  annotated_brief      n=46    (23.8% observed vs 25.0% expected)
  executive_brief      n=70    (36.3% observed vs 25.0% expected)
  full_briefing        n=52    (26.9% observed vs 25.0% expected)
  ste                  n=25    (13.0% observed vs 25.0% expected)

Balance test  : chi_square=21.404, p=0.0001, is_balanced=False
Guardrails    : not configured
Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

`tool_verbosity` · `gpclean.md`

Readiness: 🟢 READY · Variants: full_bash vs minimal_toolset · Total runs: 114 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)

Variant	Runs	Progress
`full_bash`	53	██████████ 53/20 (100%)
`minimal_toolset`	61	██████████ 61/20 (100%)

Experiment : tool_verbosity
Workflow   : gpclean.md
Metric     : aic
min_samples: 20 per variant | Total runs: 114
  full_bash            n=53    (46.5% observed vs 50.0% expected)
  minimal_toolset      n=61    (53.5% observed vs 50.0% expected)

Balance test  : chi_square=0.561, p=0.4601, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `issue-arborist.md`

Readiness: 🟢 READY · Variants: concise vs detailed · Total runs: 73 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run

Variant	Runs	Progress
`concise`	41	██████████ 41/20 (100%)
`detailed`	32	██████████ 32/20 (100%)

Experiment : prompt_style
Workflow   : issue-arborist.md
Metric     : links_created
min_samples: 20 per variant | Total runs: 73
  concise              n=41    (56.2% observed vs 50.0% expected)
  detailed             n=32    (43.8% observed vs 50.0% expected)

Balance test  : chi_square=1.110, p=0.2925, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `smokeantigravity`

Readiness: 🟢 READY · Variants: single_agent vs sub_agents · Total runs: 208 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`single_agent`	99	██████████ 99/20 (100%)
`sub_agents`	109	██████████ 109/20 (100%)

Experiment : sub_agent_strategy
Workflow   : smokeantigravity
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 208
  single_agent         n=99    (47.6% observed vs 50.0% expected)
  sub_agents           n=109   (52.4% observed vs 50.0% expected)

Balance test  : chi_square=0.481, p=0.4952, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smoke-copilot.md`

Readiness: 🟢 READY · Variants: no vs yes · Total runs: 361 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`no`	181	██████████ 181/20 (100%)
`yes`	180	██████████ 180/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 361
  no                   n=181   (50.1% observed vs 50.0% expected)
  yes                  n=180   (49.9% observed vs 50.0% expected)

Balance test  : chi_square=0.003, p=0.9118, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smoke-copilot.md`

Readiness: 🟢 READY · Variants: large vs small · Total runs: 341 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`large`	170	██████████ 170/20 (100%)
`small`	171	██████████ 171/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 341
  large                n=170   (49.9% observed vs 50.0% expected)
  small                n=171   (50.1% observed vs 50.0% expected)

Balance test  : chi_square=0.003, p=0.9109, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smoke-copilot-aoai-apikey.md`

Readiness: 🟢 READY · Variants: no vs yes · Total runs: 183 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`no`	91	██████████ 91/20 (100%)
`yes`	92	██████████ 92/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot-aoai-apikey.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 183
  no                   n=91    (49.7% observed vs 50.0% expected)
  yes                  n=92    (50.3% observed vs 50.0% expected)

Balance test  : chi_square=0.005, p=0.8991, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smoke-copilot-aoai-apikey.md`

Readiness: 🟢 READY · Variants: large vs small · Total runs: 183 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`large`	91	██████████ 91/20 (100%)
`small`	92	██████████ 92/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot-aoai-apikey.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 183
  large                n=91    (49.7% observed vs 50.0% expected)
  small                n=92    (50.3% observed vs 50.0% expected)

Balance test  : chi_square=0.005, p=0.8991, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smoke-copilot-aoai-entra.md`

Readiness: 🟢 READY · Variants: no vs yes · Total runs: 156 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`no`	78	██████████ 78/20 (100%)
`yes`	78	██████████ 78/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot-aoai-entra.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 156
  no                   n=78    (50.0% observed vs 50.0% expected)
  yes                  n=78    (50.0% observed vs 50.0% expected)

Balance test  : chi_square=0.000, p=1.0000, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smoke-copilot-aoai-entra.md`

Readiness: 🟢 READY · Variants: large vs small · Total runs: 156 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`large`	78	██████████ 78/20 (100%)
`small`	78	██████████ 78/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot-aoai-entra.md
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 156
  large                n=78    (50.0% observed vs 50.0% expected)
  small                n=78    (50.0% observed vs 50.0% expected)

Balance test  : chi_square=0.000, p=1.0000, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `smoke-gemini.md`

Readiness: 🟢 READY · Variants: single_agent vs sub_agents · Total runs: 266 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in aic. H1: sub_agents reduces tokens by >=20%

Variant	Runs	Progress
`single_agent`	126	██████████ 126/20 (100%)
`sub_agents`	140	██████████ 140/20 (100%)

Experiment : sub_agent_strategy
Workflow   : smoke-gemini.md
Metric     : aic
min_samples: 20 per variant | Total runs: 266
  single_agent         n=126   (47.4% observed vs 50.0% expected)
  sub_agents           n=140   (52.6% observed vs 50.0% expected)

Balance test  : chi_square=0.737, p=0.3951, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_decomposition` · `smokepi`

Readiness: 🟢 READY · Variants: parallel_sub_agents vs single_agent · Total runs: 76 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

(not specified)

Variant	Runs	Progress
`parallel_sub_agents`	42	██████████ 42/20 (100%)
`single_agent`	34	██████████ 34/20 (100%)

Experiment : sub_agent_decomposition
Workflow   : smokepi
Metric     : (not specified)
min_samples: 20 per variant | Total runs: 76
  parallel_sub_agents  n=42    (55.3% observed vs 50.0% expected)
  single_agent         n=34    (44.7% observed vs 50.0% expected)

Balance test  : chi_square=0.842, p=0.3619, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`tone_style` · `typist.md`

Readiness: 🟢 READY · Variants: conversational vs formal · Total runs: 81 · min_samples: 20

📈 Statistics — hypothesis, samples, balance test

H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality

Variant	Runs	Progress
`conversational`	44	██████████ 44/20 (100%)
`formal`	37	██████████ 37/20 (100%)

Experiment : tone_style
Workflow   : typist.md
Metric     : discussion_engagement_score
min_samples: 20 per variant | Total runs: 81
  conversational       n=44    (54.3% observed vs 50.0% expected)
  formal               n=37    (45.7% observed vs 50.0% expected)

Balance test  : chi_square=0.605, p=0.4427, is_balanced=True
Guardrails    : not configured
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

🟡 Collecting Samples (27 experiments)

Below the min_samples threshold for at least one variant. Progress bars show current/target counts.

Experiment	Workflow	Total runs	Variant progress	Decision
`sub_agent_strategy`	architectureguardian	38	`single_agent`=17/20, `sub_agents`=21/20	EXTEND (`insufficient_samples`)
`audit_decomposition`	audit-workflows.md	73	`phased_sub_agents`=41/264, `single_agent`=32/264	EXTEND (`insufficient_samples`)
`prompt_style`	blog-auditor.md	9	`concise`=3/20, `detailed`=6/20	EXTEND (`insufficient_samples`)
`output_format`	copilot-agent-analysis.md	80	`prose`=34/30, `ste`=3/30, `structured`=43/30	INCONCLUSIVE (`unsupported_multi_variant`)
`detail_level`	daily-architecture-diagram.md	17	`brief`=11/20, `comprehensive`=6/20	EXTEND (`insufficient_samples`)
`model_size`	daily-cache-strategy-analyzer.md	10	`gpt-5.3-codex`=5/20, `gpt-5.3-codex-spark`=5/20	EXTEND (`insufficient_samples`)
`model_size`	daily-caveman-optimizer.md	104	`agent`=1/20, `claude-haiku-4.5`=61/20, `claude-sonnet-4.6`=34/20, `claude-sonnet-5`=6/20, `small-agent`=2/20	INCONCLUSIVE (`unsupported_multi_variant`)
`output_format`	daily-code-metrics.md	120	`executive_summary`=50/20, `full_detail`=57/20, `ste`=13/20	INCONCLUSIVE (`unsupported_multi_variant`)
`output_format`	daily-compiler-quality.md	122	`concise`=51/20, `detailed`=59/20, `ste`=12/20	INCONCLUSIVE (`unsupported_multi_variant`)
`model_size`	daily-doc-healer.md	104	`agent`=2/20, `claude-haiku-4.5`=53/20, `claude-sonnet-4.6`=43/20, `claude-sonnet-5`=5/20, `small-agent`=1/20	INCONCLUSIVE (`unsupported_multi_variant`)
`model_size`	daily-doc-updater.md	101	`agent`=1/20, `claude-haiku-4.5`=44/20, `claude-sonnet-4.6`=47/20, `claude-sonnet-5`=7/20, `small-agent`=2/20	INCONCLUSIVE (`unsupported_multi_variant`)
`model_size`	dailyfunctionnamer	3		EXTEND (`insufficient_observations`)
`output_format`	daily-issues-report.md	133	`collapsible`=56/20, `inline`=60/20, `ste`=17/20	INCONCLUSIVE (`unsupported_multi_variant`)
`remove_redundant_context_v1`	daily-rendering-scripts-verifier.md	1		EXTEND (`insufficient_observations`)
`log_fetch_strategy`	daily-safe-output-optimizer.md	1		EXTEND (`insufficient_observations`)
`semgrep_output_format`	daily-semgrep-scan.md	77	`bullet_list`=28/30, `prose`=27/30, `structured_sections`=22/30	INCONCLUSIVE (`unsupported_multi_variant`)
`timeout_setting`	dailysubagentoptimizer	4	`default`=2/20, `relaxed`=1/20, `tight`=1/20	INCONCLUSIVE (`unsupported_multi_variant`)
`caveman_mode`	dataflow-pr-discussion-dataset.md	14	`no`=5/20, `yes`=9/20	EXTEND (`insufficient_samples`)
`summary_detail`	dependabotcampaign	12	`brief`=6/20, `detailed`=6/20	EXTEND (`insufficient_samples`)
`prompt_style`	dependabot-go-checker.md	47	`concise`=17/20, `detailed`=17/20, `step_by_step`=13/20	INCONCLUSIVE (`unsupported_multi_variant`)
`reasoning_depth`	plan.md	7	`baseline`=4/20, `deep`=2/20, `shallow`=1/20	INCONCLUSIVE (`unsupported_multi_variant`)
`remove_redundant_context_v1`	pr-sous-chef.md	1		EXTEND (`insufficient_observations`)
`sub_agent_strategy`	smoke-copilot-sub-agents.md	43	`delegated_sequential`=15/30, `inline_strict`=12/30, `single_agent_control`=16/30	INCONCLUSIVE (`unsupported_multi_variant`)
`prompt_style_test`	smoke-project.md	1		EXTEND (`insufficient_observations`)
`sub_agent_strategy`	smoke-temporary-id.md	1		EXTEND (`insufficient_observations`)
`model_size`	test-quality-sentinel.md	1		EXTEND (`insufficient_observations`)
`prefetch_strategy`	weekly-blog-post-writer.md	14	`eager`=5/20, `lazy`=9/20	EXTEND (`insufficient_samples`)

📊 Summary

View Full Experiments Table (all 51)

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
`prompt_compression`	agent-performance-analyzer.md	`caveman`	`verbose`	p=0.410	N/A	EXTEND	EXTEND
`sub_agent_strategy`	agent-persona-explorer.md	`batch`	`per_scenario`	p=0.525	N/A	EXTEND	EXTEND
`tone_variant`	aw-failure-investigator.md	`assertive`	`clinical / narrative`	p=0.880	N/A	INCONCLUSIVE	INCONCLUSIVE
`tone_variant`	breaking-change-checker.md	`neutral`	`urgent`	p=0.852	N/A	EXTEND	EXTEND
`prompt_style`	ci-coach.md	`concise`	`detailed`	p=0.000	FAIL	EXTEND	EXTEND
`sub_agent_strategy`	daily-agentrx-trace-optimizer.md	`single_agent`	`sub_agents`	p=0.875	N/A	EXTEND	EXTEND
`prompt_style`	daily-astrostylelite-markdown-spellcheck.md	`concise`	`detailed`	p=0.892	N/A	EXTEND	EXTEND
`prompt_style`	daily-community-attribution.md	`concise`	`verbose`	p=0.891	N/A	EXTEND	EXTEND
`reasoning_depth`	daily-fact.md	`multi_candidate`	`single_pass`	p=0.052	FAIL	EXTEND	EXTEND
`prompt_style`	daily-news.md	`concise`	`detailed`	p=0.439	N/A	EXTEND	EXTEND
`reasoning_depth`	daily-security-red-team.md	`iterative`	`single_pass`	p=0.016	FAIL	EXTEND	EXTEND
`output_format`	deep-report.md	`annotated_brief`	`executive_brief / full_briefing / ste`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`tool_verbosity`	gpclean.md	`full_bash`	`minimal_toolset`	p=0.460	N/A	EXTEND	EXTEND
`prompt_style`	issue-arborist.md	`concise`	`detailed`	p=0.292	N/A	EXTEND	EXTEND
`sub_agent_strategy`	smokeantigravity	`single_agent`	`sub_agents`	p=0.495	N/A	EXTEND	EXTEND
`caveman`	smoke-copilot.md	`no`	`yes`	p=0.912	N/A	EXTEND	EXTEND
`subagent_model`	smoke-copilot.md	`large`	`small`	p=0.911	N/A	EXTEND	EXTEND
`caveman`	smoke-copilot-aoai-apikey.md	`no`	`yes`	p=0.899	N/A	EXTEND	EXTEND
`subagent_model`	smoke-copilot-aoai-apikey.md	`large`	`small`	p=0.899	N/A	EXTEND	EXTEND
`caveman`	smoke-copilot-aoai-entra.md	`no`	`yes`	p=1.000	N/A	EXTEND	EXTEND
`subagent_model`	smoke-copilot-aoai-entra.md	`large`	`small`	p=1.000	N/A	EXTEND	EXTEND
`sub_agent_strategy`	smoke-gemini.md	`single_agent`	`sub_agents`	p=0.395	N/A	EXTEND	EXTEND
`sub_agent_decomposition`	smokepi	`parallel_sub_agents`	`single_agent`	p=0.362	N/A	EXTEND	EXTEND
`tone_style`	typist.md	`conversational`	`formal`	p=0.443	N/A	EXTEND	EXTEND
`sub_agent_strategy`	architectureguardian	`single_agent`	`sub_agents`	p=0.524	N/A	EXTEND	EXTEND
`audit_decomposition`	audit-workflows.md	`phased_sub_agents`	`single_agent`	p=0.292	FAIL	EXTEND	EXTEND
`prompt_style`	blog-auditor.md	`concise`	`detailed`	p=0.319	N/A	EXTEND	EXTEND
`output_format`	copilot-agent-analysis.md	`prose`	`ste / structured`	p=0.000	FAIL	INCONCLUSIVE	INCONCLUSIVE
`detail_level`	daily-architecture-diagram.md	`brief`	`comprehensive`	p=0.223	N/A	EXTEND	EXTEND
`model_size`	daily-cache-strategy-analyzer.md	`gpt-5.3-codex`	`gpt-5.3-codex-spark`	p=1.000	FAIL	EXTEND	EXTEND
`model_size`	daily-caveman-optimizer.md	`agent`	`claude-haiku-4.5 / claude-sonnet-4.6 / claude-sonnet-5 / small-agent`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`output_format`	daily-code-metrics.md	`executive_summary`	`full_detail / ste`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`output_format`	daily-compiler-quality.md	`concise`	`detailed / ste`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`model_size`	daily-doc-healer.md	`agent`	`claude-haiku-4.5 / claude-sonnet-4.6 / claude-sonnet-5 / small-agent`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`model_size`	daily-doc-updater.md	`agent`	`claude-haiku-4.5 / claude-sonnet-4.6 / claude-sonnet-5 / small-agent`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`model_size`	dailyfunctionnamer	`n/a`	`n/a`	p=0.000	N/A	EXTEND	EXTEND
`output_format`	daily-issues-report.md	`collapsible`	`inline / ste`	p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
`remove_redundant_context_v1`	daily-rendering-scripts-verifier.md	`n/a`	`n/a`	p=0.000	N/A	EXTEND	EXTEND
`log_fetch_strategy`	daily-safe-output-optimizer.md	`n/a`	`n/a`	p=0.000	FAIL	EXTEND	EXTEND
`semgrep_output_format`	daily-semgrep-scan.md	`bullet_list`	`prose / structured_sections`	p=0.023	FAIL	INCONCLUSIVE	INCONCLUSIVE
`timeout_setting`	dailysubagentoptimizer	`default`	`relaxed / tight`	p=0.781	N/A	INCONCLUSIVE	INCONCLUSIVE
`caveman_mode`	dataflow-pr-discussion-dataset.md	`no`	`yes`	p=0.285	N/A	EXTEND	EXTEND
`summary_detail`	dependabotcampaign	`brief`	`detailed`	p=1.000	N/A	EXTEND	EXTEND
`prompt_style`	dependabot-go-checker.md	`concise`	`detailed / step_by_step`	p=0.716	N/A	INCONCLUSIVE	INCONCLUSIVE
`reasoning_depth`	plan.md	`baseline`	`deep / shallow`	p=0.369	N/A	INCONCLUSIVE	INCONCLUSIVE
`remove_redundant_context_v1`	pr-sous-chef.md	`n/a`	`n/a`	p=0.000	N/A	EXTEND	EXTEND
`sub_agent_strategy`	smoke-copilot-sub-agents.md	`delegated_sequential`	`inline_strict / single_agent_control`	p=0.694	FAIL	INCONCLUSIVE	INCONCLUSIVE
`prompt_style_test`	smoke-project.md	`n/a`	`n/a`	p=0.000	N/A	EXTEND	EXTEND
`sub_agent_strategy`	smoke-temporary-id.md	`n/a`	`n/a`	p=0.000	FAIL	EXTEND	EXTEND
`model_size`	test-quality-sentinel.md	`n/a`	`n/a`	p=0.000	N/A	EXTEND	EXTEND
`prefetch_strategy`	weekly-blog-post-writer.md	`eager`	`lazy`	p=0.285	N/A	EXTEND	EXTEND

Descriptive window: local experiments/* git branch state per workflow (no tracking issues configured; outcome-metric correlation from the last 30 runs was not performed because no metric observations are recorded in git branch state for most experiments) · Decision thresholds: min_samples=20 default, confidence=0.95, minimum_effect=0, regression_tolerance=0 (per-workflow decision_policy)
Run: 35070670953

🔧 Self-Tuning Continuation Plan

View continuation plan

Top 3 experiment actions

Instrument native metric observations for READY experiments blocked by insufficient_observations (20 experiments, e.g. agent-performance-analyzer.prompt_compression, daily-news.prompt_style, issue-arborist.prompt_style) — these already have balanced samples ≥ min_samples but the declared metric: field (e.g. aic, token_count) is never written into the git branch state or an eval/grader artifact the core analyzer can read. Add a pick_experiment/grader hook that records the primary metric per run so these can resolve to PROMOTE/REJECT instead of parking indefinitely at EXTEND.
Add native two-sided support for guardrail metrics on cicoach.prompt_style, daily-fact.reasoning_depth, daily-security-red-team.reasoning_depth — all three are READY but blocked by guardrail_unsupported; their configured guardrail_metrics reference fields the current pipeline can't evaluate per-run. Wire these guardrails into the grader/eval observation source before the next report.
Extend the decision engine to K ≥ 3 variants for deep-report.output_format and 3 other unsupported_multi_variant experiments — deep-report has a statistically significant imbalance (chi_square=21.4, p=0.0001) worth investigating, but the core decision layer only supports exactly-two-variant comparisons today. Prioritize adding a multi-variant ANOVA/Kruskal-Wallis path so 4-variant experiments like this stop parking at INCONCLUSIVE.

Top 3 eval actions

Tighten eval questions tied to the aic/token-cost metrics used by agent-performance-analyzer, agent-persona-explorer, and smoke-copilot* experiments — these prompts currently produce no scoreable observations feeding the primary metric, so eval graders should assert token/aic capture explicitly per run.
Add an eval check for empty_output_rate and issue_creation_success_rate guardrails referenced by deep-report.output_format, so guardrail pass/fail becomes computable instead of defaulting to "not configured".
For daily-fact.reasoning_depth and daily-security-red-team.reasoning_depth, add eval grading for the declared guardrail metrics so guardrail_unsupported converts to a real pass/fail signal.

Decision-pipeline gaps for the next PR

20 READY experiments are stuck on insufficient_observations / guardrail_unsupported — the single largest blocker to converging on any PROMOTE/REJECT decision in this repository today.
Multi-variant (K ≥ 3) decision support (unsupported_multi_variant, 14 experiments) is the second-largest gap; deep-report.output_format shows a significant balance-test signal that the core analyzer cannot yet act on.
No experiment declares an issue: tracking field, so Step 8/9 (issue notifications and lifecycle labels) had nothing to act on this run — consider recommending tracking issues be added for experiments nearing READY status.
Interaction diagnostics (K1×K2 cells) were not triggered because no workflow currently has a PROMOTE decision with sparse cells; this remains a reporting-only safety net until a genuine multi-experiment PROMOTE case arises.

Generated by 🧪 daily-experiment-report · copilot · auto · 122.9 AIC · ⌖ 16 AIC · ⊞ 16.7K · ◷

expires on Sep 19, 2026, 12:02 AM UTC-08:00

2026-09-17T08:09:53Z

github-actions[bot]
Bot Sep 17, 2026
Author

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #61560.

0 replies

Navigation Menu

[experiments] Daily Experiment Report — 2026-09-16 #61311

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[experiments] Daily Experiment Report — 2026-09-16 #61311

Uh oh!

github-actions[bot] Bot Sep 16, 2026

🧪 Daily Experiment Report — 2026-09-16

⚡ Quick Stats

🟢 Ready for Analysis (24 experiments)

prompt_compression · agent-performance-analyzer.md

sub_agent_strategy · agent-persona-explorer.md

tone_variant · aw-failure-investigator.md

tone_variant · breaking-change-checker.md

prompt_style · ci-coach.md

sub_agent_strategy · daily-agentrx-trace-optimizer.md

prompt_style · daily-astrostylelite-markdown-spellcheck.md

prompt_style · daily-community-attribution.md

reasoning_depth · daily-fact.md

prompt_style · daily-news.md

reasoning_depth · daily-security-red-team.md

output_format · deep-report.md

tool_verbosity · gpclean.md

prompt_style · issue-arborist.md

sub_agent_strategy · smokeantigravity

caveman · smoke-copilot.md

subagent_model · smoke-copilot.md

caveman · smoke-copilot-aoai-apikey.md

subagent_model · smoke-copilot-aoai-apikey.md

caveman · smoke-copilot-aoai-entra.md

subagent_model · smoke-copilot-aoai-entra.md

sub_agent_strategy · smoke-gemini.md

sub_agent_decomposition · smokepi

tone_style · typist.md

🟡 Collecting Samples (27 experiments)

📊 Summary

🔧 Self-Tuning Continuation Plan

Replies: 1 comment

Uh oh!

github-actions[bot] Bot Sep 17, 2026 Author

github-actions[bot]
Bot Sep 16, 2026

`prompt_compression` · `agent-performance-analyzer.md`

`sub_agent_strategy` · `agent-persona-explorer.md`

`tone_variant` · `aw-failure-investigator.md`

`tone_variant` · `breaking-change-checker.md`

`prompt_style` · `ci-coach.md`

`sub_agent_strategy` · `daily-agentrx-trace-optimizer.md`

`prompt_style` · `daily-astrostylelite-markdown-spellcheck.md`

`prompt_style` · `daily-community-attribution.md`

`reasoning_depth` · `daily-fact.md`

`prompt_style` · `daily-news.md`

`reasoning_depth` · `daily-security-red-team.md`

`output_format` · `deep-report.md`

`tool_verbosity` · `gpclean.md`

`prompt_style` · `issue-arborist.md`

`sub_agent_strategy` · `smokeantigravity`

`caveman` · `smoke-copilot.md`

`subagent_model` · `smoke-copilot.md`

`caveman` · `smoke-copilot-aoai-apikey.md`

`subagent_model` · `smoke-copilot-aoai-apikey.md`

`caveman` · `smoke-copilot-aoai-entra.md`

`subagent_model` · `smoke-copilot-aoai-entra.md`

`sub_agent_strategy` · `smoke-gemini.md`

`sub_agent_decomposition` · `smokepi`

`tone_style` · `typist.md`

github-actions[bot]
Bot Sep 17, 2026
Author