[experiments] Daily Experiment Report — 2026-06-27

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-27

4 experiments analysed across 4 workflows. 0 reached statistical significance (p < 0.05). Recommendations: ❌ ABANDON: 1 · 🟡 EXTEND: 3 · ✅ PROMOTE: 0.

⚡ Quick Stats

Metric	Value
Active experiments	4
Ready for analysis	0
Statistically significant (p < 0.05)	0
Recommendations	✅ PROMOTE: 0 · 🟡 EXTEND: 3 · ❌ ABANDON: 1

---

prompt_style · ci-coach · 🔴 GUARDRAIL FAILED

detailed (ctrl) vs concise · min_samples: 20 · p = 0.6048

Variant	n	Succ %	Mean dur	95% CI	p-value	Progress
detailed (ctrl)	16	62.5%	658s	[478, 838]s	(ref)	████████░░ 16/20 (80%)
concise	15	71.4%	844s	[596, 1091]s	0.6048	████████░░ 15/20 (75%)

Guardrails: run_success_rate >=0.85 → ❌ FAIL (detailed=0.625, concise=0.714)

⚠️ Most failures occurred in May 2026 (early experiment period). Recent runs since June 12 show 100% success for both variants, suggesting transient infrastructure issues.

❌ ABANDON — run_success_rate guardrail fails for both variants. Consider resetting the experiment to capture a cleaner window post-May.

---

model_size · daily-doc-healer · 🟡 COLLECTING

claude-sonnet-4.6 (ctrl) vs claude-haiku-4.5 · min_samples: 20 · p = 0.4022

Variant	n	Succ %	Mean dur	p-value	Progress
claude-sonnet-4.6 (ctrl)	8	100.0%	849s	(ref)	████░░░░░░ 8/20 (40%)
claude-haiku-4.5	12	91.7%	724s	0.4022	██████░░░░ 12/20 (60%)

Guardrails: run_success_rate >=0.90 → ✅ PASS (both variants)

🟡 EXTEND — Both variants below min_samples (sonnet 8/20, haiku 12/20). Early signal: Haiku is 15% faster (724s vs 849s) with comparable success rates.

---

output_format · daily-code-metrics · 🟡 COLLECTING (1 run away!)

full_detail (ctrl) vs executive_summary · min_samples: 20 · p = 0.0998

Variant	n	Succ %	Mean dur	95% CI	p-value	Progress
full_detail (ctrl)	22	94.4%	1016s	[944, 1088]s	(ref)	██████████ 22/20 ✅
executive_summary	19	72.7%	887s	[735, 1039]s	0.0998	██████████ 19/20 (95%)

Guardrails: report_empty_rate, quality_score_present → N/A

🟡 EXTEND — executive_summary needs 1 more run (19/20). ⚠️ Success rate gap (72.7% vs 94.4%) is notable — watch for ABANDON once min_samples is reached.

---

reasoning_depth · daily-security-red-team · 🟡 COLLECTING

single_pass (ctrl) vs iterative · min_samples: 30 · p = 0.2129

Variant	n	Succ %	Mean dur	95% CI	p-value	Progress
single_pass (ctrl)	20	91.7%	407s	[350, 465]s	(ref)	███████░░░ 20/30 (67%)
iterative	26	100.0%	445s	[409, 480]s	0.2129	█████████░ 26/30 (87%)

Guardrails: run_success_rate >=0.90 → ✅ PASS (both variants)

🟡 EXTEND — single_pass at 20/30, iterative at 26/30. iterative shows perfect success (100% vs 91.7%) and is 9% slower (+38s) — consistent with its re-evaluation design.

---

📊 Summary

Experiment	Control	Best variant	p-value	Guardrails	Rec.
prompt_style (ci-coach)	detailed 62.5%	concise 71.4%	0.6048	❌ FAIL	❌ ABANDON
model_size (daily-doc-healer)	sonnet-4.6 100%	haiku-4.5 91.7%	0.4022	✅ PASS	🟡 EXTEND
output_format (daily-code-metrics)	full_detail 94.4%	exec_summary 72.7%	0.0998	N/A	🟡 EXTEND
reasoning_depth (daily-security-red-team)	single_pass 91.7%	iterative 100.0%	0.2129	✅ PASS	🟡 EXTEND

Analysis window: last 30 completed runs per workflow · Significance threshold: p < 0.05 (two-tailed)
References: §28284290859

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 215.8 AIC · ⌖ 25.4 AIC · ⊞ 11.3K · ◷

• expires on Jun 30, 2026, 1:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #42038.