[experiments] Daily Experiment Report — 2026-09-28

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-28

48 workflows in github/gh-aw currently declare experiments: sections, running 51 active A/B experiments. 26 experiments have reached READY sample thresholds, but none have a PROMOTE or REJECT core decision yet — 37 are held at EXTEND (mostly insufficient_observations/guardrail_unsupported, since native metric/guardrail plumbing isn't wired up for most experiments) and 14 multi-variant experiments (K≥3) return INCONCLUSIVE (unsupported_multi_variant) until core analysis supports more than two variants. The smoke-copilot* family (3 workflows) runs two crossed experiments (caveman × subagent_model) with all interaction cells sparse (n<20 in every cell), so a cross-experiment safety hold is documented below even though no individual core decision is currently PROMOTE.

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	1 (smoke-copilot family, 3 workflows)

---

🔎 All 26 READY experiments with a tracking issue (detailed statistics)

prompt_compression · agent-performance-analyzer.md

Readiness: 🟢 READY
Variants: caveman vs verbose · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
caveman	70	██████████ 70/20 (100%)
verbose	59	██████████ 59/20 (100%)

Experiment : prompt_compression
Workflow   : agent-performance-analyzer.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| caveman          |    7 |    57.1% |          663.3 | [321.1 , 1005.5]   | 70/20 (100%)  |
| verbose          |    3 |    66.7% |          682.7 | [148.9 , 1216.4]   | 59/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_variant · aw-failure-investigator.md

Readiness: 🟢 READY
Variants: narrative vs assertive vs clinical · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
assertive	162	██████████ 162/20 (100%)
clinical	156	██████████ 156/20 (100%)
narrative	157	██████████ 157/20 (100%)

Experiment : tone_variant
Workflow   : aw-failure-investigator.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| narrative        |    3 |   100.0% |          482.7 | [393.1 , 572.3]    | 157/20 (100%) |
| assertive        |    5 |   100.0% |          507.6 | [450.2 , 565.0]    | 162/20 (100%) |
| clinical         |    2 |   100.0% |          464.5 | [407.3 , 521.7]    | 156/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

tone_variant · breaking-change-checker.md

Readiness: 🟢 READY
Variants: urgent vs neutral · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
neutral	27	██████████ 27/20 (100%)
urgent	24	██████████ 24/20 (100%)

Experiment : tone_variant
Workflow   : breaking-change-checker.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| urgent           |    5 |    20.0% |          702.2 | [346.2 , 1058.2]   | 24/20 (100%)  |
| neutral          |    5 |     0.0% |          408.2 | [376.4 , 440.0]    | 27/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · ci-coach.md

Readiness: 🟢 READY
Variants: detailed vs concise · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	39	██████████ 39/20 (100%)
detailed	57	██████████ 57/20 (100%)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| detailed         |    8 |    62.5% |         1352.0 | [788.4 , 1915.6]   | 57/20 (100%)  |
| concise          |    2 |   100.0% |         1262.5 | [-675.2 , 3200.2]  | 39/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

reasoning_depth · daily-fact.md

Readiness: 🟢 READY
Variants: multi_candidate vs single_pass · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Evidence: descriptive only (no supported core primary-metric observation)

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
multi_candidate	40	██████████ 40/30 (100%)
single_pass	56	██████████ 56/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-fact.md
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| multi_candidate  |    6 |     0.0% |          348.5 | [277.5 , 419.5]    | 40/30 (100%)  |
| single_pass      |    4 |     0.0% |          436.5 | [181.9 , 691.1]    | 56/30 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

---

output_format · daily-issues-report.md

Readiness: 🟢 READY
Variants: collapsible vs inline vs ste · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
collapsible	59	██████████ 59/20 (100%)
inline	63	██████████ 63/20 (100%)
ste	22	██████████ 22/20 (100%)

Experiment : output_format
Workflow   : daily-issues-report.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| collapsible      |    2 |   100.0% |          729.5 | [202.2 , 1256.8]   | 59/20 (100%)  |
| inline           |    3 |   100.0% |          707.0 | [618.1 , 795.9]    | 63/20 (100%)  |
| ste              |    5 |    80.0% |          787.2 | [490.0 , 1084.4]   | 22/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · daily-news.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	62	██████████ 62/20 (100%)
detailed	50	██████████ 50/20 (100%)

Experiment : prompt_style
Workflow   : daily-news.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    7 |   100.0% |          716.3 | [676.2 , 756.4]    | 62/20 (100%)  |
| detailed         |    3 |   100.0% |          695.0 | [547.0 , 843.0]    | 50/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

reasoning_depth · daily-security-red-team.md

Readiness: 🟢 READY
Variants: single_pass vs iterative · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Evidence: descriptive only (no supported core primary-metric observation)

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
iterative	81	██████████ 81/30 (100%)
single_pass	57	██████████ 57/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_pass      |    6 |     0.0% |         3793.0 | [3751.2 , 3834.8]  | 57/30 (100%)  |
| iterative        |    4 |     0.0% |         3819.0 | [3763.8 , 3874.2]  | 81/30 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

prompt_style · issue-arborist.md

Readiness: 🟢 READY
Variants: detailed vs concise · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	46	██████████ 46/20 (100%)
detailed	39	██████████ 39/20 (100%)

Experiment : prompt_style
Workflow   : issue-arborist.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| detailed         |    5 |     0.0% |          341.4 | [284.0 , 398.8]    | 39/20 (100%)  |
| concise          |    5 |    20.0% |          364.4 | [305.2 , 423.6]    | 46/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

model_size · test-quality-sentinel.md

Readiness: 🟢 READY
Variants: claude-sonnet-5 vs claude-haiku-4.5 · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
claude-haiku-4.5	40	██████████ 40/20 (100%)
claude-sonnet-5	39	██████████ 39/20 (100%)

Experiment : model_size
Workflow   : test-quality-sentinel.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| claude-sonnet-5  |    6 |   100.0% |          430.8 | [365.3 , 496.3]    | 39/20 (100%)  |
| claude-haiku-4.5 |    4 |    75.0% |          586.5 | [235.7 , 937.3]    | 40/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_style · typist.md

Readiness: 🟢 READY
Variants: formal vs conversational · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: descriptive only (no supported core primary-metric observation)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
conversational	47	██████████ 47/20 (100%)
formal	42	██████████ 42/20 (100%)

Experiment : tone_style
Workflow   : typist.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| formal           |    6 |    50.0% |          691.7 | [258.4 , 1125.0]   | 42/20 (100%)  |
| conversational   |    4 |    75.0% |         1013.2 | [399.6 , 1626.9]   | 47/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive outcome metrics only (n<20 duration/success samples); core primary-metric observation is unavailable/unsupported

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman × subagent_model interaction · smoke-copilot family (3 workflows)

Readiness: 🟡 COLLECTING (all interaction cells sparse)
Workflows: smoke-copilot.md, smoke-copilot-aoai-apikey.md, smoke-copilot-aoai-entra.md

Each workflow crosses two active experiments (caveman: yes/no, subagent_model: small/large). Individual per-experiment core decisions are all EXTEND (insufficient_observations), but interaction cell diagnostics are computed below as a reporting safety check.

📈 View Interaction Cell Diagnostics

smoke-copilot

Cell (caveman, subagent_model)	n	Success rate
(no, large)	5	20.0%
(yes, large)	1	100.0%
(yes, small)	4	25.0%

interaction_p_value = 0.270 · interaction_risk_status = SPARSE_CELL_RISK (all cells n<20)

smoke-copilot-aoai-apikey

Cell	n	Success rate
(no, large)	1	0.0%
(no, small)	5	0.0%
(yes, large)	3	0.0%
(yes, small)	1	0.0%

interaction_p_value = N/A (degenerate table) · interaction_risk_status = SPARSE_CELL_RISK

smoke-copilot-aoai-entra

Cell	n	Success rate
(no, large)	3	0.0%
(no, small)	2	0.0%
(yes, large)	2	0.0%
(yes, small)	3	0.0%

interaction_p_value = N/A (degenerate table) · interaction_risk_status = SPARSE_CELL_RISK

Since no individual core decision is currently PROMOTE, this hold is informational only for now — but the pattern is recorded so that if any experiment in this family later reaches PROMOTE, the sparse interaction cells will force report_action: EXTEND (interaction_underpowered) instead of an outright promotion recommendation.

---

📊 Summary

View Full Experiments Table (all 51 experiments)

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
prompt_compression	agent-performance-analyzer.md	caveman	verbose	descriptive only	n/a	EXTEND	EXTEND
sub_agent_strategy	agent-persona-explorer.md	batch	per_scenario	descriptive only	n/a	EXTEND	EXTEND
sub_agent_strategy	architectureguardian.md	single_agent	sub_agents	descriptive only	n/a	EXTEND	EXTEND
audit_decomposition	audit-workflows.md	single_agent	phased_sub_agents	descriptive only	n/a	EXTEND	EXTEND
tone_variant	aw-failure-investigator.md	assertive	clinical	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
prompt_style	blog-auditor.md	concise	detailed	descriptive only	n/a	EXTEND	EXTEND
tone_variant	breaking-change-checker.md	neutral	urgent	descriptive only	n/a	EXTEND	EXTEND
prompt_style	ci-coach.md	detailed	concise	descriptive only	n/a	EXTEND	EXTEND
output_format	copilot-agent-analysis.md	prose	ste	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	daily-agentrx-trace-optimizer.md	single_agent	sub_agents	descriptive only	n/a	EXTEND	EXTEND
detail_level	daily-architecture-diagram.md	brief	comprehensive	descriptive only	n/a	EXTEND	EXTEND
prompt_style	daily-astrostylelite-markdown-spellcheck.md	concise	detailed	descriptive only	n/a	EXTEND	EXTEND
model_size	daily-cache-strategy-analyzer.md	gpt-5.3-codex	gpt-5.3-codex-spark	descriptive only	n/a	EXTEND	EXTEND
model_size	daily-caveman-optimizer.md	agent	claude-haiku-4.5	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
output_format	daily-code-metrics.md	executive_summary	full_detail	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
prompt_style	daily-community-attribution.md	concise	verbose	descriptive only	n/a	EXTEND	EXTEND
output_format	daily-compiler-quality.md	concise	detailed	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
model_size	daily-doc-healer.md	agent	claude-haiku-4.5	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
model_size	daily-doc-updater.md	agent	claude-haiku-4.5	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
reasoning_depth	daily-fact.md	single_pass	multi_candidate	descriptive only	n/a	EXTEND	EXTEND
model_size	dailyfunctionnamer.md	n/a	n/a	descriptive only	n/a	EXTEND	EXTEND
output_format	daily-issues-report.md	collapsible	inline	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
prompt_style	daily-news.md	concise	detailed	descriptive only	n/a	EXTEND	EXTEND
remove_redundant_context_v1	daily-rendering-scripts-verifier.md	n/a	n/a	descriptive only	n/a	EXTEND	EXTEND
log_fetch_strategy	daily-safe-output-optimizer.md	eager	lazy	descriptive only	n/a	EXTEND	EXTEND
reasoning_depth	daily-security-red-team.md	single_pass	iterative	descriptive only	n/a	EXTEND	EXTEND
semgrep_output_format	daily-semgrep-scan.md	bullet_list	prose	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer.md	default	relaxed	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
caveman_mode	dataflow-pr-discussion-dataset.md	no	yes	descriptive only	n/a	EXTEND	EXTEND
output_format	deep-report.md	annotated_brief	executive_brief	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
summary_detail	dependabotcampaign.md	brief	detailed	descriptive only	n/a	EXTEND	EXTEND
prompt_style	dependabot-go-checker.md	concise	detailed	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
tool_verbosity	gpclean.md	full_bash	minimal_toolset	descriptive only	n/a	EXTEND	EXTEND
prompt_style	issue-arborist.md	concise	detailed	descriptive only	n/a	EXTEND	EXTEND
reasoning_depth	plan.md	baseline	deep	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
remove_redundant_context_v1	pr-sous-chef.md	n/a	n/a	descriptive only	n/a	EXTEND	EXTEND
sub_agent_strategy	smokeantigravity.md	single_agent	sub_agents	descriptive only	n/a	EXTEND	EXTEND
caveman	smoke-copilot.md	no	yes	descriptive only	n/a	EXTEND	EXTEND
subagent_model	smoke-copilot.md	large	small	descriptive only	n/a	EXTEND	EXTEND
caveman	smoke-copilot-aoai-apikey.md	no	yes	descriptive only	n/a	EXTEND	EXTEND
subagent_model	smoke-copilot-aoai-apikey.md	large	small	descriptive only	n/a	EXTEND	EXTEND
caveman	smoke-copilot-aoai-entra.md	no	yes	descriptive only	n/a	EXTEND	EXTEND
subagent_model	smoke-copilot-aoai-entra.md	large	small	descriptive only	n/a	EXTEND	EXTEND
sub_agent_strategy	smoke-copilot-sub-agents.md	delegated_sequential	inline_strict	descriptive only	n/a	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	smoke-gemini.md	single_agent	sub_agents	descriptive only	n/a	EXTEND	EXTEND
sub_agent_decomposition	smokepi.md	parallel_sub_agents	single_agent	descriptive only	n/a	EXTEND	EXTEND
prompt_style_test	smoke-project.md	n/a	n/a	descriptive only	n/a	EXTEND	EXTEND
sub_agent_strategy	smoke-temporary-id.md	single_agent	sub_agents	descriptive only	n/a	EXTEND	EXTEND
model_size	test-quality-sentinel.md	claude-haiku-4.5	claude-sonnet-5	descriptive only	n/a	EXTEND	EXTEND
tone_style	typist.md	conversational	formal	descriptive only	n/a	EXTEND	EXTEND
prefetch_strategy	weekly-blog-post-writer.md	eager	lazy	descriptive only	n/a	EXTEND	EXTEND

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy (min effect 0, regression tolerance 0, confidence 0.95)
Run: 36394910398

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. dailyissuesreport / deepreport / dailycodemetrics / dailycompilerquality / dailydochealer / dailydocupdater / dailycavemanoptimizer / dailysemgrepscan / dependabotgochecker / awfailureinvestigator / plan / smokecopilotsubagents / dailysubagentoptimizer / copilotagentanalysis (14 total, unsupported_multi_variant) — these experiments declare 3+ variants (e.g. output_format: [collapsible, inline, ste], model_size: [agent, claude-haiku-4.5, ...]). Core analysis currently only supports pairwise (K=2) decisions. Action: extend the core decision engine to support K≥3 comparisons (omnibus test + pairwise post-hoc with Bonferroni correction, already partially scaffolded via bonferroni_alpha), or split these into paired sub-experiments.
2. cicoach, dailyfact, dailysecurityredteam (guardrail_unsupported) — mandatory guardrail metrics (run_success_rate, empty_output_rate) are declared in frontmatter but have no supported native observation source. Action: wire these guardrail names to the existing run_success_rate/output-emptiness detectors so decision_guardrails.passed can be computed instead of blocking on EXTEND.
3. All 26 insufficient_observations experiments (majority of the READY set) — primary metric: fields (e.g. token_count_per_run, aic, discussion_engagement_score) aren't backed by a supported observation pipeline yet, even though sample counts are well past min_samples. Action: prioritize native metric collection for the highest-volume workflows first (smoke-copilot, smoke-gemini, smoke-copilot-aoai-*, dailyissuesreport) since they already have the largest run histories and would unblock a PROMOTE/REJECT decision fastest.

Top 3 eval actions

1. Workflows using output_format variants (dailyissuesreport, deepreport, dailycodemetrics, dailycompilerquality, copilotagentanalysis, dailysemgrepscan) declare "eval:output_format_goal_met" / "eval:output_format_adherence" secondary metrics — tighten these grader questions to score format compliance (e.g. STE readability, collapsible-section usage) numerically rather than pass/fail, so they can feed the core decision engine as a supported observation.
2. copilot-pr-nlp-analysis declares eval:insights_report_produced and eval:pr_conversations_analyzed — these should be validated against the workflow's actual PR-conversation-analysis prompt section to ensure the grader checks the produced artifact, not just its presence.
3. Workflows relying on aic (ai_credits) as the sole primary metric (agent-persona-explorer, blog-auditor, daily-news, gpclean, smoke-temporary-id, smoke-gemini) should add a secondary quality-eval question so cost reductions aren't rewarded at the expense of output quality — currently these can only ever answer "is it cheaper," never "is it still good."

Decision-pipeline gaps for the next PR

• insufficient_observations (26 experiments) and guardrail_unsupported (3 experiments) together account for 100% of the EXTEND decisions among READY experiments — this is the single largest lever to unlock PROMOTE/REJECT outcomes. The native-metric-to-core-analysis bridge (for token_count_per_run, aic, ai_credits_total, discussion_engagement_score, run_success_rate, empty_output_rate) is the top priority gap.
• unsupported_multi_variant (14 experiments, all K≥3) needs a normalized core analysis signal (omnibus + Bonferroni-corrected pairwise) before these experiments can ever resolve past INCONCLUSIVE.
• Cross-experiment interaction diagnostics (caveman × subagent_model in the smoke-copilot family) are currently computed only in this report as a descriptive safety check. These should become a normalized core analysis signal (interaction_p_value, SPARSE_CELL_RISK) so that a future PROMOTE decision in either experiment is automatically held pending sufficient interaction cell samples, rather than relying on manual reporting-layer logic.

Generated by 🧪 daily-experiment-report · copilot · auto · 211.2 AIC · ⌖ 20.1 AIC · ⊞ 16.3K · ◷

• expires on Oct 1, 2026, 12:18 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #64219.