[experiments] Daily Experiment Report — 2026-09-14

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-14

51 active A/B experiments were analysed across 48 workflows in github/gh-aw. 23 have reached READY sample-size thresholds; the core deterministic decision layer returned 0 PROMOTE, 37 EXTEND, 0 REJECT, and 14 INCONCLUSIVE — the platform's automatic decision logic currently only supports strict two-variant comparisons with an observed primary-metric signal, so most READY multi-variant or metric-unsupported experiments correctly stay EXTEND/INCONCLUSIVE rather than false-promoting.

⚡ Quick Stats

Metric	Value
Active experiments	51 (across 48 workflows; 3 workflows run 2 experiments each)
Ready for analysis	23
Decisions with sufficient evidence	0 (no experiment has a supported 2-variant primary-metric observation pipeline yet)
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	1 (smokecopilot family: caveman × subagent_model)

Data-pipeline gap (affects every experiment): none of the 51 experiments have a working primary-metric observation feed into the core decision engine yet — reason codes are exclusively insufficient_observations (25), unsupported_multi_variant (14), insufficient_samples (9), and guardrail_unsupported (3). Assignment/balance tracking (chi-square, is_balanced) works correctly for all experiments; the missing link is wiring each workflow's declared metric:/guardrail_metrics: into the artifact the analyzer reads. This is the single highest-leverage engineering fix (see Step 10).

---

prompt_compression · agent-performance-analyzer.md

Readiness: 🟢 READY
Variants: caveman vs verbose · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in effective_tokens. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
caveman	62	██████████ 62/20 (100%)
verbose	53	██████████ 53/20 (100%)

Experiment : prompt_compression
Workflow   : agent-performance-analyzer.md
Hypothesis : H0: no change in effective_tokens. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| caveman          |    7 |   100.0% |          955.7 |   [748.6 , 1162.9] | ██████ 62/20 (100%) |
| verbose          |    3 |   100.0% |          780.0 |    [587.5 , 972.5] | ██████ 53/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — insufficient_observations

Guardrails:
  (guardrail_metrics declared but not evaluated — insufficient_observations)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_variant · aw-failure-investigator.md

Readiness: 🟢 READY
Variants: assertive vs clinical vs narrative · Window: last 30 runs · Analysed: 3 runs with artifacts
min_samples: 20 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
assertive	144	██████████ 144/20 (100%)
clinical	138	██████████ 138/20 (100%)
narrative	133	██████████ 133/20 (100%)

Experiment : tone_variant
Workflow   : aw-failure-investigator.md
Hypothesis : H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.
Window     : last 30 runs  |  Analysed: 3 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| assertive        |    2 |   100.0% |          630.0 |    [286.9 , 973.1] | ██████ 144/20 (100%) |
| clinical         |  N/A |      N/A |            N/A |                N/A | ██████ 138/20 (100%) |
| narrative        |    1 |   100.0% |          577.0 |                N/A | ██████ 133/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — unsupported_multi_variant

Guardrails:
  (guardrail_metrics declared but not evaluated — unsupported_multi_variant)

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · ci-coach.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	37	██████████ 37/20 (100%)
detailed	49	██████████ 49/20 (100%)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    4 |   100.0% |         1290.8 |  [1045.2 , 1536.3] | ██████ 37/20 (100%) |
| detailed         |    6 |    66.7% |         1369.2 |   [789.8 , 1948.5] | ██████ 49/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — guardrail_unsupported

Guardrails:
  (guardrail_metrics declared but not evaluated — guardrail_unsupported)

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

reasoning_depth · daily-fact.md

Readiness: 🟢 READY
Variants: multi_candidate vs single_pass · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

⚠️ Data quality flag: the last 10 completed daily-fact runs are all failure (0% success rate both variants). This is a workflow-health issue independent of the tone experiment and should be triaged before trusting any metric from this experiment.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
multi_candidate	35	██████████ 35/30 (100%)
single_pass	52	██████████ 52/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-fact.md
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| multi_candidate  |    5 |     0.0% |          323.4 |    [298.9 , 347.9] | ██████ 35/30 (100%) |
| single_pass      |    5 |     0.0% |          304.4 |    [230.3 , 378.5] | ██████ 52/30 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — guardrail_unsupported

Guardrails:
  (guardrail_metrics declared but not evaluated — guardrail_unsupported)

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

---

prompt_style · daily-news.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	55	██████████ 55/20 (100%)
detailed	47	██████████ 47/20 (100%)

Experiment : prompt_style
Workflow   : daily-news.md
Hypothesis : H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    6 |    83.3% |          801.3 |    [670.1 , 932.5] | ██████ 55/20 (100%) |
| detailed         |    4 |    75.0% |          757.0 |    [679.9 , 834.1] | ██████ 47/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — insufficient_observations

Guardrails:
  (guardrail_metrics declared but not evaluated — insufficient_observations)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

reasoning_depth · daily-security-red-team.md

Readiness: 🟢 READY
Variants: iterative vs single_pass · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

⚠️ Data quality flag: the last 10 completed daily-security-red-team runs are all failure (0% success rate both variants). Investigate workflow health before drawing experiment conclusions.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
iterative	76	██████████ 76/30 (100%)
single_pass	49	██████████ 49/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| iterative        |    7 |     0.0% |         3822.1 |  [3797.3 , 3847.0] | ██████ 76/30 (100%) |
| single_pass      |    3 |     0.0% |         3821.0 |  [3758.7 , 3883.3] | ██████ 49/30 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — guardrail_unsupported

Guardrails:
  (guardrail_metrics declared but not evaluated — guardrail_unsupported)

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

prompt_style · issue-arborist.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	39	██████████ 39/20 (100%)
detailed	32	██████████ 32/20 (100%)

Experiment : prompt_style
Workflow   : issue-arborist.md
Hypothesis : H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    4 |    50.0% |          353.5 |    [221.5 , 485.5] | ██████ 39/20 (100%) |
| detailed         |    6 |    33.3% |          378.7 |    [294.6 , 462.8] | ██████ 32/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — insufficient_observations

Guardrails:
  (guardrail_metrics declared but not evaluated — insufficient_observations)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_style · typist.md

Readiness: 🟢 READY
Variants: conversational vs formal · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: none (metric pipeline not yet wired for this experiment)

H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
conversational	43	██████████ 43/20 (100%)
formal	36	██████████ 36/20 (100%)

Experiment : tone_style
Workflow   : typist.md
Hypothesis : H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| conversational   |    5 |   100.0% |         1147.0 |   [836.3 , 1457.7] | ██████ 43/20 (100%) |
| formal           |    5 |    80.0% |          915.0 |   [552.2 , 1277.8] | ██████ 36/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: no primary-metric observation pipeline wired yet — insufficient_observations

Guardrails:
  (guardrail_metrics declared but not evaluated — insufficient_observations)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

Factorial interaction: caveman × subagent_model (smoke-copilot family)

smokecopilot, smokecopilotaoaiapikey, and smokecopilotaoaientra each run two simultaneous experiments (caveman and subagent_model). Interaction cells from smoke-copilot's last 30 runs (assignment-matched):

caveman \ subagent_model	large	small
no	n=2, 50.0%	n=3, 33.3%
yes	n=4, 0.0%	n=1, 100.0%

interaction_p_value = 0.519 · all 4 cells have n < min_samples (20) → interaction_risk_status: SPARSE_CELL_RISK.

Both individual experiments (caveman, subagent_model) are currently EXTEND (not PROMOTE), so this safety hold does not change any report action today — noted here so it's tracked ahead of the day the 2×2 design reaches READY on both axes.

---

📊 Summary

View Full Experiments Table (51 experiments across 48 workflows)

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
prompt_compression	agentperformanceanalyzer	caveman	verbose	insufficient_observations	N/A	EXTEND	EXTEND
sub_agent_strategy	agentpersonaexplorer	batch	per_scenario	insufficient_observations	N/A	EXTEND	EXTEND
sub_agent_strategy	architectureguardian	single_agent	sub_agents	insufficient_samples	N/A	EXTEND	EXTEND
audit_decomposition	auditworkflows	single_agent	phased_sub_agents	insufficient_samples	FAIL	EXTEND	EXTEND
tone_variant	awfailureinvestigator	assertive	clinical	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	blogauditor	concise	detailed	insufficient_samples	N/A	EXTEND	EXTEND
tone_variant	breakingchangechecker	neutral	urgent	insufficient_samples	N/A	EXTEND	EXTEND
prompt_style	cicoach	detailed	concise	guardrail_unsupported	FAIL	EXTEND	EXTEND
output_format	copilotagentanalysis	prose	ste	unsupported_multi_variant	FAIL	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent	sub_agents	insufficient_observations	N/A	EXTEND	EXTEND
detail_level	dailyarchitecturediagram	brief	comprehensive	insufficient_samples	N/A	EXTEND	EXTEND
prompt_style	dailyastrostylelitemarkdownspellcheck	concise	detailed	insufficient_observations	N/A	EXTEND	EXTEND
model_size	dailycachestrategyanalyzer	gpt-5.3-codex	gpt-5.3-codex-spark	insufficient_samples	FAIL	EXTEND	EXTEND
model_size	dailycavemanoptimizer	agent	claude-haiku-4.5	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
output_format	dailycodemetrics	executive_summary	full_detail	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	dailycommunityattribution	concise	verbose	insufficient_observations	N/A	EXTEND	EXTEND
output_format	dailycompilerquality	concise	detailed	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	dailydochealer	agent	claude-haiku-4.5	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	dailydocupdater	agent	claude-haiku-4.5	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
reasoning_depth	dailyfact	single_pass	multi_candidate	guardrail_unsupported	FAIL	EXTEND	EXTEND
model_size	dailyfunctionnamer	n/a	n/a	insufficient_observations	N/A	EXTEND	EXTEND
output_format	dailyissuesreport	collapsible	inline	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	dailynews	concise	detailed	insufficient_observations	N/A	EXTEND	EXTEND
remove_redundant_context_v1	dailyrenderingscriptsverifier	n/a	n/a	insufficient_observations	N/A	EXTEND	EXTEND
log_fetch_strategy	dailysafeoutputoptimizer	eager	lazy	insufficient_observations	FAIL	EXTEND	EXTEND
reasoning_depth	dailysecurityredteam	single_pass	iterative	guardrail_unsupported	FAIL	EXTEND	EXTEND
semgrep_output_format	dailysemgrepscan	bullet_list	prose	unsupported_multi_variant	FAIL	INCONCLUSIVE	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer	default	relaxed	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
caveman_mode	dataflowprdiscussiondataset	no	yes	insufficient_samples	N/A	EXTEND	EXTEND
output_format	deepreport	annotated_brief	executive_brief	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
summary_detail	dependabotcampaign	brief	detailed	insufficient_samples	N/A	EXTEND	EXTEND
prompt_style	dependabotgochecker	concise	detailed	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
tool_verbosity	gpclean	full_bash	minimal_toolset	insufficient_observations	N/A	EXTEND	EXTEND
prompt_style	issuearborist	concise	detailed	insufficient_observations	N/A	EXTEND	EXTEND
reasoning_depth	plan	baseline	deep	unsupported_multi_variant	N/A	INCONCLUSIVE	INCONCLUSIVE
remove_redundant_context_v1	prsouschef	n/a	n/a	insufficient_observations	N/A	EXTEND	EXTEND
sub_agent_strategy	smokeantigravity	single_agent	sub_agents	insufficient_observations	N/A	EXTEND	EXTEND
caveman	smokecopilot	no	yes	insufficient_observations	N/A	EXTEND	EXTEND
subagent_model	smokecopilot	large	small	insufficient_observations	N/A	EXTEND	EXTEND
caveman	smokecopilotaoaiapikey	no	yes	insufficient_observations	N/A	EXTEND	EXTEND
subagent_model	smokecopilotaoaiapikey	large	small	insufficient_observations	N/A	EXTEND	EXTEND
caveman	smokecopilotaoaientra	no	yes	insufficient_observations	N/A	EXTEND	EXTEND
subagent_model	smokecopilotaoaientra	large	small	insufficient_observations	N/A	EXTEND	EXTEND
sub_agent_strategy	smokecopilotsubagents	delegated_sequential	inline_strict	unsupported_multi_variant	FAIL	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	smokegemini	single_agent	sub_agents	insufficient_observations	N/A	EXTEND	EXTEND
sub_agent_decomposition	smokepi	parallel_sub_agents	single_agent	insufficient_observations	N/A	EXTEND	EXTEND
prompt_style_test	smokeproject	n/a	n/a	insufficient_observations	N/A	EXTEND	EXTEND
sub_agent_strategy	smoketemporaryid	single_agent	sub_agents	insufficient_observations	FAIL	EXTEND	EXTEND
model_size	testqualitysentinel	n/a	n/a	insufficient_observations	N/A	EXTEND	EXTEND
tone_style	typist	conversational	formal	insufficient_observations	N/A	EXTEND	EXTEND
prefetch_strategy	weeklyblogpostwriter	eager	lazy	insufficient_samples	N/A	EXTEND	EXTEND

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy (min effect 0, confidence 0.95)
Run: 34820127819

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire primary-metric observations into the core decision pipeline. 25/51 experiments are stuck on insufficient_observations even though sample counts are healthy (e.g. awfailureinvestigator at 415 runs, smokecopilot caveman at 359 runs). Each workflow declares a metric: field, but the analyzer has no artifact path reading it back. This is the top blocker to ever reaching PROMOTE/REJECT.
2. Add native multi-variant decision support (or force 2-variant collapsing) for the 14 experiments returning unsupported_multi_variant (e.g. awfailureinvestigator tone_variant with 3 variants, dailycodemetrics, deepreport). These have large, balanced samples (is_balanced: true, p-values > 0.8) and are otherwise fully ready.
3. Wire guardrail_metrics evaluation for the 3 guardrail_unsupported experiments (cicoach, dailyfact, dailysecurityredteam) — each declares a guardrail (e.g. run_success_rate >= 0.85) that is never evaluated, so EXTEND is issued purely as a safety default even when samples are sufficient.

Top 3 eval actions

1. Triage daily-fact and daily-security-red-team workflow health before trusting any experiment metric. Both show 0% success rate on their last 10 completed runs — no eval or grader questions will produce meaningful signal while the underlying workflow is failing outright.
2. Add low-friction success/failure graders for cicoach's prompt_style experiment — detailed shows 67% success vs concise 100% (n=6 vs n=4); before any decision, an eval question isolating whether failures correlate with prompt length vs. unrelated infra flakiness would sharpen the signal.
3. Tighten issue-arborist prompt_style eval questions around link-quality, since detailed (n=6, 33% success) underperforms concise (n=4, 50% success) on raw completion rate — the declared metric (links_created) doesn't currently capture whether low-success runs are silently producing zero links.

Decision-pipeline gaps for next PR

• insufficient_observations (25 experiments) and guardrail_unsupported (3 experiments) both stem from the same root cause: no artifact writes the declared metric:/guardrail_metrics: values back to the experiment state that gh aw experiments analyze reads. This should become a normalized "observation ingestion" step.
• unsupported_multi_variant (14 experiments) — the core decision engine needs either pairwise comparison support (control vs. each candidate) or an explicit N-way ANOVA/Kruskal-Wallis path before these can leave INCONCLUSIVE.
• Factorial interaction diagnostics (smokecopilot family caveman × subagent_model) are currently computed only in this report, ad hoc. They should become a normalized core analysis signal so SPARSE_CELL_RISK can automatically gate PROMOTE decisions once any 2×2 design reaches readiness on both axes.

Generated by 🧪 daily-experiment-report · copilot · auto · 218.5 AIC · ⌖ 24.8 AIC · ⊞ 16.7K · ◷

• expires on Sep 17, 2026, 12:12 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #61070.