[experiments] Daily Experiment Report — 2026-09-18 #61760

2026-09-18T08:09:42Z

github-actions[bot]
Bot Sep 18, 2026

🧪 Daily Experiment Report — 2026-09-18

51 experiments analysed across 48 workflows in github/gh-aw. 24 have reached min_samples for at least descriptive comparison. Core decisions: ✅ PROMOTE 0 · 🟡 EXTEND 37 · ❌ REJECT 0 · ⚪ INCONCLUSIVE 14. No PROMOTE or REJECT decisions today — the majority remain in evidence-collection or unsupported-multi-variant status; no tracking issues are configured so no lifecycle labels or issue comments apply.

⚡ Quick Stats

Metric	Value
Active experiments	51 (across 48 workflows)
Ready for analysis (readiness=READY)	24
Decisions with sufficient evidence (PROMOTE/REJECT)	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	3 (smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra — all sparse-cell, no PROMOTE affected)

Note on outcome metrics: Success rate, mean duration, and 95% CI columns are derived only from the runs present in each experiment's recent_runs snapshot (last 10 assignment records returned by gh aw experiments analyze), not the full state-tracked variant counts shown in the Progress column. For high-volume experiments (e.g. tone_variant in aw-failure-investigator.md, output_format in deep-report.md) this means outcome n ≪ assignment n — treat outcome stats as a small recent sample, not the full experiment population.

🟢 Experiments Ready for Analysis (24)

`prompt_compression` · `agentperformanceanalyzer`

Readiness: 🟢 READY · Variants: caveman vs verbose · min_samples: 20 · Total runs: 119
Hypothesis: H0: no change in aic. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`caveman`	63	██████████ 63/20 (100%)	80%	1072s	[743,1402]
`verbose`	56	██████████ 56/20 (100%)	80%	877s	[452,1302]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `agentpersonaexplorer`

Readiness: 🟢 READY · Variants: batch vs per_scenario · min_samples: 20 · Total runs: 119
Hypothesis: H0: no change in aic or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`batch`	62	██████████ 62/20 (100%)	80%	605s	[470,740]
`per_scenario`	57	██████████ 57/20 (100%)	40%	449s	[375,524]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`tone_variant` · `awfailureinvestigator`

Readiness: 🟢 READY · Variants: assertive vs clinical vs narrative · min_samples: 20 · Total runs: 431
Hypothesis: H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`assertive`	148	██████████ 148/20 (100%)	0%	462s	[325,598]
`clinical`	142	██████████ 142/20 (100%)	0%	480s	[424,537]
`narrative`	141	██████████ 141/20 (100%)	25%	592s	[408,775]

Evidence: analysis_type=n/a, metric=n/a

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

`tone_variant` · `breakingchangechecker`

Readiness: 🟢 READY · Variants: neutral vs urgent · min_samples: 20 · Total runs: 45
Hypothesis: H0: no change in issue_engagement_rate. H1: urgent increases issue_engagement_rate by >=7 percentage points without degrading run_success_rate.

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`neutral`	24	██████████ 24/20 (100%)	50%	540s	[278,801]
`urgent`	21	██████████ 21/20 (100%)	50%	869s	[656,1082]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `cicoach`

Readiness: 🟢 READY · Variants: concise vs detailed · min_samples: 20 · Total runs: 90
Hypothesis: H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`concise`	37	██████████ 37/20 (100%)	100%	1280s	N/A
`detailed`	53	██████████ 53/20 (100%)	78%	1494s	[987,2001]

Guardrails: run_success_rate >=0.85: UNSUPPORTED; empty_output_rate <=0.05: UNSUPPORTED

Evidence: analysis_type=mann_whitney, metric=token_count_per_run

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

`sub_agent_strategy` · `dailyagentrxtraceoptimizer`

Readiness: 🟢 READY · Variants: single_agent vs sub_agents · min_samples: 20 · Total runs: 79
Hypothesis: H0: no change in issue quality or run success rate. H1: sub_agents variant yields higher evidence completeness score with equal or lower token cost

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`single_agent`	39	██████████ 39/20 (100%)	25%	835s	[-6,1675]
`sub_agents`	40	██████████ 40/20 (100%)	50%	1747s	[284,3211]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `dailyastrostylelitemarkdownspellcheck`

Readiness: 🟢 READY · Variants: concise vs detailed · min_samples: 20 · Total runs: 137
Hypothesis: Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`concise`	68	██████████ 68/20 (100%)	100%	607s	[510,704]
`detailed`	69	██████████ 69/20 (100%)	57%	492s	[366,618]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `dailycommunityattribution`

Readiness: 🟢 READY · Variants: concise vs verbose · min_samples: 20 · Total runs: 135

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`concise`	68	██████████ 68/20 (100%)	67%	594s	[436,751]
`verbose`	67	██████████ 67/20 (100%)	75%	547s	[313,782]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`reasoning_depth` · `dailyfact`

Readiness: 🟢 READY · Variants: multi_candidate vs single_pass · min_samples: 30 · Total runs: 90
Hypothesis: H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`multi_candidate`	36	██████████ 36/30 (100%)	0%	328s	[306,349]
`single_pass`	54	██████████ 54/30 (100%)	0%	443s	[192,694]

Guardrails: empty_output_rate <0.05: UNSUPPORTED; run_success_rate >=0.95: UNSUPPORTED

Evidence: analysis_type=n/a, metric=discussion_reaction_count

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

`prompt_style` · `dailynews`

Readiness: 🟢 READY · Variants: concise vs detailed · min_samples: 20 · Total runs: 106
Hypothesis: H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`concise`	58	██████████ 58/20 (100%)	100%	745s	[692,799]
`detailed`	48	██████████ 48/20 (100%)	75%	742s	[699,785]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`reasoning_depth` · `dailysecurityredteam`

Readiness: 🟢 READY · Variants: iterative vs single_pass · min_samples: 30 · Total runs: 129
Hypothesis: H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`iterative`	77	██████████ 77/30 (100%)	0%	3818s	[3788,3847]
`single_pass`	52	██████████ 52/30 (100%)	0%	3825s	[3762,3887]

Guardrails: run_success_rate >=0.9: UNSUPPORTED; empty_output_rate <=0.05: UNSUPPORTED

Evidence: analysis_type=proportion_test, metric=false_positive_rate

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

`output_format` · `deepreport`

Readiness: 🟢 READY · Variants: annotated_brief vs executive_brief vs full_briefing vs ste · min_samples: 20 · Total runs: 201
Hypothesis: H0: no change in discussion engagement or token cost. H1: executive_brief reduces token usage by ≥20% without reducing engagement; annotated_brief improves actionability; ste improves clarity while reducing token usage.

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`annotated_brief`	48	██████████ 48/20 (100%)	0%	484s	[103,865]
`executive_brief`	72	██████████ 72/20 (100%)	67%	827s	[-81,1735]
`full_briefing`	53	██████████ 53/20 (100%)	0%	428s	N/A
`ste`	28	██████████ 28/20 (100%)	25%	555s	[346,763]

Evidence: analysis_type=n/a, metric=n/a

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

`tool_verbosity` · `gpclean`

Readiness: 🟢 READY · Variants: full_bash vs minimal_toolset · min_samples: 20 · Total runs: 116
Hypothesis: H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`full_bash`	54	██████████ 54/20 (100%)	60%	399s	[321,477]
`minimal_toolset`	62	██████████ 62/20 (100%)	20%	348s	[284,412]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`prompt_style` · `issuearborist`

Readiness: 🟢 READY · Variants: concise vs detailed · min_samples: 20 · Total runs: 75
Hypothesis: H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`concise`	41	██████████ 41/20 (100%)	75%	376s	[231,520]
`detailed`	34	██████████ 34/20 (100%)	33%	362s	[271,454]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `smokeantigravity`

Readiness: 🟢 READY · Variants: single_agent vs sub_agents · min_samples: 20 · Total runs: 208

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`single_agent`	99	██████████ 99/20 (100%)	100%	617s	[235,999]
`sub_agents`	109	██████████ 109/20 (100%)	100%	720s	[215,1226]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smokecopilot`

Readiness: 🟢 READY · Variants: no vs yes · min_samples: 20 · Total runs: 363

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`no`	182	██████████ 182/20 (100%)	33%	701s	[503,899]
`yes`	181	██████████ 181/20 (100%)	25%	656s	[343,969]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smokecopilot`

Readiness: 🟢 READY · Variants: large vs small · min_samples: 20 · Total runs: 343

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`large`	172	██████████ 172/20 (100%)	33%	696s	[523,870]
`small`	171	██████████ 171/20 (100%)	25%	663s	[297,1029]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smokecopilotaoaiapikey`

Readiness: 🟢 READY · Variants: no vs yes · min_samples: 20 · Total runs: 184

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`no`	92	██████████ 92/20 (100%)	20%	685s	[265,1105]
`yes`	92	██████████ 92/20 (100%)	60%	676s	[422,930]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smokecopilotaoaiapikey`

Readiness: 🟢 READY · Variants: large vs small · min_samples: 20 · Total runs: 184

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`large`	92	██████████ 92/20 (100%)	40%	711s	[363,1059]
`small`	92	██████████ 92/20 (100%)	40%	650s	[309,991]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`caveman` · `smokecopilotaoaientra`

Readiness: 🟢 READY · Variants: no vs yes · min_samples: 20 · Total runs: 157

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`no`	78	██████████ 78/20 (100%)	25%	769s	[110,1428]
`yes`	79	██████████ 79/20 (100%)	50%	711s	[483,939]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`subagent_model` · `smokecopilotaoaientra`

Readiness: 🟢 READY · Variants: large vs small · min_samples: 20 · Total runs: 157

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`large`	78	██████████ 78/20 (100%)	40%	797s	[412,1183]
`small`	79	██████████ 79/20 (100%)	40%	671s	[312,1029]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_strategy` · `smokegemini`

Readiness: 🟢 READY · Variants: single_agent vs sub_agents · min_samples: 20 · Total runs: 267
Hypothesis: H0: no change in aic. H1: sub_agents reduces tokens by >=20%

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`single_agent`	126	██████████ 126/20 (100%)	0%	482s	[114,849]
`sub_agents`	141	██████████ 141/20 (100%)	0%	350s	[303,397]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`sub_agent_decomposition` · `smokepi`

Readiness: 🟢 READY · Variants: parallel_sub_agents vs single_agent · min_samples: 20 · Total runs: 76

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`parallel_sub_agents`	42	██████████ 42/20 (100%)	100%	259s	[144,374]
`single_agent`	34	██████████ 34/20 (100%)	100%	199s	[173,224]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

`tone_style` · `typist`

Readiness: 🟢 READY · Variants: conversational vs formal · min_samples: 20 · Total runs: 83
Hypothesis: H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality

Variant	n (state)	Progress	Succ %	Mean dur	95% CI (s)
`conversational`	45	██████████ 45/20 (100%)	80%	923s	[570,1276]
`formal`	38	██████████ 38/20 (100%)	60%	807s	[265,1348]

Evidence: analysis_type=n/a, metric=n/a

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

🟡 Experiments Still Collecting Samples (27)

View all COLLECTING experiments

Experiment	Workflow	Sample progress	Decision
`sub_agent_strategy`	`architectureguardian`	`single_agent` ████████░░ 17/20 (85%); `sub_agents` ██████████ 21/20 (100%)	🟡 EXTEND (`insufficient_samples`)
`audit_decomposition`	`auditworkflows`	`phased_sub_agents` ██░░░░░░░░ 41/264 (16%); `single_agent` █░░░░░░░░░ 34/264 (13%)	🟡 EXTEND (`insufficient_samples`)
`prompt_style`	`blogauditor`	`concise` ██░░░░░░░░ 3/20 (15%); `detailed` ████░░░░░░ 7/20 (35%)	🟡 EXTEND (`insufficient_samples`)
`output_format`	`copilotagentanalysis`	`prose` ██████████ 34/30 (100%); `ste` █░░░░░░░░░ 3/30 (10%); `structured` ██████████ 43/30 (100%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`detail_level`	`dailyarchitecturediagram`	`brief` ██████░░░░ 11/20 (55%); `comprehensive` ███░░░░░░░ 6/20 (30%)	🟡 EXTEND (`insufficient_samples`)
`model_size`	`dailycachestrategyanalyzer`	`gpt-5.3-codex` ██░░░░░░░░ 5/20 (25%); `gpt-5.3-codex-spark` ████░░░░░░ 7/20 (35%)	🟡 EXTEND (`insufficient_samples`)
`model_size`	`dailycavemanoptimizer`	`agent` ░░░░░░░░░░ 1/20 (5%); `claude-haiku-4.5` ██████████ 63/20 (100%); `claude-sonnet-4.6` ██████████ 34/20 (100%); `claude-sonnet-5` ███░░░░░░░ 6/20 (30%); `small-agent` █░░░░░░░░░ 2/20 (10%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`output_format`	`dailycodemetrics`	`executive_summary` ██████████ 51/20 (100%); `full_detail` ██████████ 57/20 (100%); `ste` ███████░░░ 14/20 (70%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`output_format`	`dailycompilerquality`	`concise` ██████████ 52/20 (100%); `detailed` ██████████ 59/20 (100%); `ste` ██████░░░░ 13/20 (65%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`model_size`	`dailydochealer`	`agent` █░░░░░░░░░ 2/20 (10%); `claude-haiku-4.5` ██████████ 55/20 (100%); `claude-sonnet-4.6` ██████████ 43/20 (100%); `claude-sonnet-5` ██░░░░░░░░ 5/20 (25%); `small-agent` ░░░░░░░░░░ 1/20 (5%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`model_size`	`dailydocupdater`	`agent` ░░░░░░░░░░ 1/20 (5%); `claude-haiku-4.5` ██████████ 44/20 (100%); `claude-sonnet-4.6` ██████████ 47/20 (100%); `claude-sonnet-5` ████░░░░░░ 9/20 (45%); `small-agent` █░░░░░░░░░ 2/20 (10%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`model_size`	`dailyfunctionnamer`		🟡 EXTEND (`insufficient_observations`)
`output_format`	`dailyissuesreport`	`collapsible` ██████████ 58/20 (100%); `inline` ██████████ 60/20 (100%); `ste` ████████░░ 17/20 (85%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`remove_redundant_context_v1`	`dailyrenderingscriptsverifier`		🟡 EXTEND (`insufficient_observations`)
`log_fetch_strategy`	`dailysafeoutputoptimizer`		🟡 EXTEND (`insufficient_observations`)
`semgrep_output_format`	`dailysemgrepscan`	`bullet_list` █████████░ 28/30 (93%); `prose` █████████░ 27/30 (90%); `structured_sections` ███████░░░ 22/30 (73%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`timeout_setting`	`dailysubagentoptimizer`	`default` █░░░░░░░░░ 2/20 (10%); `relaxed` ░░░░░░░░░░ 1/20 (5%); `tight` ░░░░░░░░░░ 1/20 (5%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`caveman_mode`	`dataflowprdiscussiondataset`	`no` ███░░░░░░░ 6/20 (30%); `yes` ████░░░░░░ 9/20 (45%)	🟡 EXTEND (`insufficient_samples`)
`summary_detail`	`dependabotcampaign`	`brief` ███░░░░░░░ 6/20 (30%); `detailed` ███░░░░░░░ 6/20 (30%)	🟡 EXTEND (`insufficient_samples`)
`prompt_style`	`dependabotgochecker`	`concise` █████████░ 18/20 (90%); `detailed` ████████░░ 17/20 (85%); `step_by_step` ██████░░░░ 13/20 (65%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`reasoning_depth`	`plan`	`baseline` ██░░░░░░░░ 4/20 (20%); `deep` █░░░░░░░░░ 2/20 (10%); `shallow` ░░░░░░░░░░ 1/20 (5%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`remove_redundant_context_v1`	`prsouschef`		🟡 EXTEND (`insufficient_observations`)
`sub_agent_strategy`	`smokecopilotsubagents`	`delegated_sequential` █████░░░░░ 15/30 (50%); `inline_strict` ████░░░░░░ 13/30 (43%); `single_agent_control` █████░░░░░ 16/30 (53%)	⚪ INCONCLUSIVE (`unsupported_multi_variant`)
`prompt_style_test`	`smokeproject`		🟡 EXTEND (`insufficient_observations`)
`sub_agent_strategy`	`smoketemporaryid`		🟡 EXTEND (`insufficient_observations`)
`model_size`	`testqualitysentinel`		🟡 EXTEND (`insufficient_observations`)
`prefetch_strategy`	`weeklyblogpostwriter`	`eager` ██░░░░░░░░ 5/20 (25%); `lazy` ████░░░░░░ 9/20 (45%)	🟡 EXTEND (`insufficient_samples`)

🔀 Interaction Diagnostics (multi-experiment workflows)

Three workflows (smoke-copilot, smoke-copilot-aoai-apikey, smoke-copilot-aoai-entra) run caveman × subagent_model factorial experiments simultaneously. All observed cells have n < min_samples (20), so interaction_risk_status = SPARSE_CELL_RISK in every case. No PROMOTE decisions are affected today since all component decisions are EXTEND.

🔀 Interaction diagnostics: `caveman` × `subagent_model` (smokecopilot)

caveman	subagent_model	n	success rate	mean duration (s)
`no`	`large`	2	50.0%	777.0
`no`	`small`	4	25.0%	663.0
`yes`	`large`	4	25.0%	656.2

interaction_p_value: 0.1472
interaction_risk_status: SPARSE_CELL_RISK (all cells n < min_samples=20)

Since both caveman and subagent_model decisions are EXTEND (not PROMOTE), no report-level promotion safety hold is triggered here — but interaction risk remains SPARSE_CELL_RISK and must be re-checked before any future PROMOTE.

🔀 Interaction diagnostics: `caveman` × `subagent_model` (smokecopilotaoaiapikey)

caveman	subagent_model	n	success rate	mean duration (s)
`no`	`large`	2	0.0%	775.5
`no`	`small`	3	33.3%	624.3
`yes`	`large`	3	66.7%	668.0
`yes`	`small`	2	50.0%	688.0

interaction_p_value: 1.0000
interaction_risk_status: SPARSE_CELL_RISK (all cells n < min_samples=20)

Since both caveman and subagent_model decisions are EXTEND (not PROMOTE), no report-level promotion safety hold is triggered here — but interaction risk remains SPARSE_CELL_RISK and must be re-checked before any future PROMOTE.

🔀 Interaction diagnostics: `caveman` × `subagent_model` (smokecopilotaoaientra)

caveman	subagent_model	n	success rate	mean duration (s)
`no`	`large`	2	0.0%	799.5
`no`	`small`	2	50.0%	738.5
`yes`	`large`	3	66.7%	796.0
`yes`	`small`	3	33.3%	625.7

interaction_p_value: 1.0000
interaction_risk_status: SPARSE_CELL_RISK (all cells n < min_samples=20)

Since both caveman and subagent_model decisions are EXTEND (not PROMOTE), no report-level promotion safety hold is triggered here — but interaction risk remains SPARSE_CELL_RISK and must be re-checked before any future PROMOTE.

📊 Summary

View Full Experiments Table (51 experiments)

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
prompt_compression	agentperformanceanalyzer	caveman	verbose	n/a	N/A	EXTEND	EXTEND
sub_agent_strategy	agentpersonaexplorer	batch	per_scenario	n/a	N/A	EXTEND	EXTEND
sub_agent_strategy	architectureguardian	single_agent	sub_agents	n/a	N/A	EXTEND	EXTEND
audit_decomposition	auditworkflows	single_agent	phased_sub_agents	n/a	UNSUPPORTED	EXTEND	EXTEND
tone_variant	awfailureinvestigator	assertive	clinical	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	blogauditor	concise	detailed	n/a	N/A	EXTEND	EXTEND
tone_variant	breakingchangechecker	neutral	urgent	n/a	N/A	EXTEND	EXTEND
prompt_style	cicoach	detailed	concise	mann_whitney	UNSUPPORTED	EXTEND	EXTEND
output_format	copilotagentanalysis	prose	ste	t_test	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent	sub_agents	n/a	N/A	EXTEND	EXTEND
detail_level	dailyarchitecturediagram	brief	comprehensive	n/a	N/A	EXTEND	EXTEND
prompt_style	dailyastrostylelitemarkdownspellcheck	concise	detailed	n/a	N/A	EXTEND	EXTEND
model_size	dailycachestrategyanalyzer	gpt-5.3-codex	gpt-5.3-codex-spark	n/a	UNSUPPORTED	EXTEND	EXTEND
model_size	dailycavemanoptimizer	agent	claude-haiku-4.5	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
output_format	dailycodemetrics	executive_summary	full_detail	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	dailycommunityattribution	concise	verbose	n/a	N/A	EXTEND	EXTEND
output_format	dailycompilerquality	concise	detailed	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	dailydochealer	agent	claude-haiku-4.5	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	dailydocupdater	agent	claude-haiku-4.5	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
reasoning_depth	dailyfact	single_pass	multi_candidate	n/a	UNSUPPORTED	EXTEND	EXTEND
model_size	dailyfunctionnamer	n/a	n/a	n/a	N/A	EXTEND	EXTEND
output_format	dailyissuesreport	collapsible	inline	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	dailynews	concise	detailed	n/a	N/A	EXTEND	EXTEND
remove_redundant_context_v1	dailyrenderingscriptsverifier	n/a	n/a	n/a	N/A	EXTEND	EXTEND
log_fetch_strategy	dailysafeoutputoptimizer	eager	lazy	mann_whitney	UNSUPPORTED	EXTEND	EXTEND
reasoning_depth	dailysecurityredteam	single_pass	iterative	proportion_test	UNSUPPORTED	EXTEND	EXTEND
semgrep_output_format	dailysemgrepscan	bullet_list	prose	proportion_test	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer	default	relaxed	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
caveman_mode	dataflowprdiscussiondataset	no	yes	n/a	N/A	EXTEND	EXTEND
output_format	deepreport	annotated_brief	executive_brief	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
summary_detail	dependabotcampaign	brief	detailed	n/a	N/A	EXTEND	EXTEND
prompt_style	dependabotgochecker	concise	detailed	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
tool_verbosity	gpclean	full_bash	minimal_toolset	n/a	N/A	EXTEND	EXTEND
prompt_style	issuearborist	concise	detailed	n/a	N/A	EXTEND	EXTEND
reasoning_depth	plan	baseline	deep	n/a	N/A	INCONCLUSIVE	INCONCLUSIVE
remove_redundant_context_v1	prsouschef	n/a	n/a	n/a	N/A	EXTEND	EXTEND
sub_agent_strategy	smokeantigravity	single_agent	sub_agents	n/a	N/A	EXTEND	EXTEND
caveman	smokecopilot	no	yes	n/a	N/A	EXTEND	EXTEND
subagent_model	smokecopilot	large	small	n/a	N/A	EXTEND	EXTEND
caveman	smokecopilotaoaiapikey	no	yes	n/a	N/A	EXTEND	EXTEND
subagent_model	smokecopilotaoaiapikey	large	small	n/a	N/A	EXTEND	EXTEND
caveman	smokecopilotaoaientra	no	yes	n/a	N/A	EXTEND	EXTEND
subagent_model	smokecopilotaoaientra	large	small	n/a	N/A	EXTEND	EXTEND
sub_agent_strategy	smokecopilotsubagents	delegated_sequential	inline_strict	n/a	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	smokegemini	single_agent	sub_agents	n/a	N/A	EXTEND	EXTEND
sub_agent_decomposition	smokepi	parallel_sub_agents	single_agent	n/a	N/A	EXTEND	EXTEND
prompt_style_test	smokeproject	n/a	n/a	n/a	N/A	EXTEND	EXTEND
sub_agent_strategy	smoketemporaryid	single_agent	sub_agents	t_test	UNSUPPORTED	EXTEND	EXTEND
model_size	testqualitysentinel	n/a	n/a	n/a	N/A	EXTEND	EXTEND
tone_style	typist	conversational	formal	n/a	N/A	EXTEND	EXTEND
prefetch_strategy	weeklyblogpostwriter	eager	lazy	n/a	N/A	EXTEND	EXTEND

🧭 Self-Tuning Continuation Plan

🔧 Top 3 experiment actions

output_format in deep-report.md — INCONCLUSIVE (unsupported_multi_variant, 4 variants). Reduce to a 2-variant test (e.g. executive_brief vs ste) so the core decision layer can produce a deterministic PROMOTE/REJECT.
prompt_style in ci-coach.md — EXTEND (guardrail_unsupported: run_success_rate, empty_output_rate). Wire these to a supported native observation so the guardrail gate can pass/fail instead of blocking every cycle.
caveman/subagent_model in smoke-copilot*.md (3 workflows) — all EXTEND with SPARSE_CELL_RISK interaction diagnostics. Increase weekly run volume or consolidate the near-duplicate smoke workflows so factorial cells reach ≥20 samples faster.

📋 Top 3 eval actions

labels-applied (deep-report.md) — 0% pass rate (0/115). Tighten the prompt's explicit instruction to attach code-quality, automation, task-mining labels when creating issues.
output_format_goal_met (deep-report.md) — 25% pass rate (3/12). Reports don't reliably match their assigned output_format variant style (especially ste). Add per-variant style examples/checklists.
tool_verbosity_goal_met (gpclean.md) — 0% pass rate (0/3), issue-or-noop at 58% (18/31). Clarify minimal_toolset vs full_bash instructions and issue-vs-noop decision criteria.

🕳️ Decision-pipeline gaps for next PR

guardrail_unsupported blocks 3 experiments (ci-coach, daily-fact, daily-security-red-team) — run_success_rate/empty_output_rate/false_positive_rate need a native-metric resolver.
unsupported_multi_variant blocks 14 experiments (deep-report, daily-code-metrics, daily-compiler-quality, daily-doc-healer, daily-doc-updater, daily-issues-report, daily-caveman-optimizer, daily-semgrep-scan, daily-subagent-optimizer, dependabot-go-checker, plan, smoke-copilot-sub-agents) — core needs a normalized K≥3 variant decision path.
Factorial interaction risk (SPARSE_CELL_RISK) in 3 workflows is a reporting-only diagnostic today; it should become a normalized core analysis signal to natively gate promotion.

Descriptive window: last 30 runs per workflow (outcome metrics) · Decision thresholds: from decision_policy returned by gh aw experiments analyze
Run: 35321433566

Generated by 🧪 daily-experiment-report · copilot · auto · 232.1 AIC · ⌖ 18.6 AIC · ⊞ 16.9K · ◷

expires on Sep 21, 2026, 12:09 AM UTC-08:00

2026-09-19T08:09:43Z

github-actions[bot]
Bot Sep 19, 2026
Author

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #61982.

0 replies

Navigation Menu

[experiments] Daily Experiment Report — 2026-09-18 #61760

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[experiments] Daily Experiment Report — 2026-09-18 #61760

Uh oh!

github-actions[bot] Bot Sep 18, 2026

🧪 Daily Experiment Report — 2026-09-18

⚡ Quick Stats

🟢 Experiments Ready for Analysis (24)

prompt_compression · agentperformanceanalyzer

sub_agent_strategy · agentpersonaexplorer

tone_variant · awfailureinvestigator

tone_variant · breakingchangechecker

prompt_style · cicoach

sub_agent_strategy · dailyagentrxtraceoptimizer

prompt_style · dailyastrostylelitemarkdownspellcheck

prompt_style · dailycommunityattribution

reasoning_depth · dailyfact

prompt_style · dailynews

reasoning_depth · dailysecurityredteam

output_format · deepreport

tool_verbosity · gpclean

prompt_style · issuearborist

sub_agent_strategy · smokeantigravity

caveman · smokecopilot

subagent_model · smokecopilot

caveman · smokecopilotaoaiapikey

subagent_model · smokecopilotaoaiapikey

caveman · smokecopilotaoaientra

subagent_model · smokecopilotaoaientra

sub_agent_strategy · smokegemini

sub_agent_decomposition · smokepi

tone_style · typist

🟡 Experiments Still Collecting Samples (27)

🔀 Interaction Diagnostics (multi-experiment workflows)

📊 Summary

🧭 Self-Tuning Continuation Plan

Replies: 1 comment

Uh oh!

github-actions[bot] Bot Sep 19, 2026 Author

github-actions[bot]
Bot Sep 18, 2026

`prompt_compression` · `agentperformanceanalyzer`

`sub_agent_strategy` · `agentpersonaexplorer`

`tone_variant` · `awfailureinvestigator`

`tone_variant` · `breakingchangechecker`

`prompt_style` · `cicoach`

`sub_agent_strategy` · `dailyagentrxtraceoptimizer`

`prompt_style` · `dailyastrostylelitemarkdownspellcheck`

`prompt_style` · `dailycommunityattribution`

`reasoning_depth` · `dailyfact`

`prompt_style` · `dailynews`

`reasoning_depth` · `dailysecurityredteam`

`output_format` · `deepreport`

`tool_verbosity` · `gpclean`

`prompt_style` · `issuearborist`

`sub_agent_strategy` · `smokeantigravity`

`caveman` · `smokecopilot`

`subagent_model` · `smokecopilot`

`caveman` · `smokecopilotaoaiapikey`

`subagent_model` · `smokecopilotaoaiapikey`

`caveman` · `smokecopilotaoaientra`

`subagent_model` · `smokecopilotaoaientra`

`sub_agent_strategy` · `smokegemini`

`sub_agent_decomposition` · `smokepi`

`tone_style` · `typist`

github-actions[bot]
Bot Sep 19, 2026
Author