[experiments] Daily Experiment Report — 2026-10-23

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-23

53 experiments analysed across ~50 workflows (experiments/* branches). No experiment reached PROMOTE or REJECT: 44 are EXTEND, 9 are INCONCLUSIVE. Main blockers are unsupported guardrail metrics and missing observations, not sample size.

⚡ Quick Stats

Metric	Value
Active experiments	53
🟢 READY	26
🟡 COLLECTING	27
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 44 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 9
Interaction safety holds	0

📊 Core decisions by reason code

Readiness	Decision	Reason code	Count
🟡 COLLECTING	EXTEND	insufficient_samples	13
🟡 COLLECTING	EXTEND	insufficient_observations	7
🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	7
🟢 READY	EXTEND	guardrail_unsupported	15
🟢 READY	EXTEND	insufficient_observations	9
🟢 READY	INCONCLUSIVE	unsupported_multi_variant	2

🟢 READY experiments (examples)

Workflow	Experiment	Variant counts	Decision	Reason
cicoach	prompt_style	concise 49 / detailed 66	EXTEND	guardrail_unsupported
testqualitysentinel	model_size	claude-haiku-4.5 162 / claude-sonnet-5 188	EXTEND	guardrail_unsupported
smokecopilot	caveman	no 206 / yes 207	EXTEND	insufficient_observations
dailycommunityattribution	prompt_style	concise 79 / verbose 79	EXTEND	insufficient_observations
deepreport	output_format	4 variants incl. ste	INCONCLUSIVE	unsupported_multi_variant
dailycompilerquality	output_format	concise/detailed/ste	INCONCLUSIVE	unsupported_multi_variant

Per-run outcome metrics and charts were not produced this run; decisions come solely from core analysis. No tracking-issue comments/labels were emitted since no PROMOTE or guardrail_failed decision occurred.

🔧 Self-Tuning Continuation Plan

Experiment actions

1. Add native guardrail metric support for the 15 READY guardrail_unsupported experiments (cicoach, testqualitysentinel, dailynews, typist).
2. Emit observations (graders/evals) for the 9 READY insufficient_observations experiments (smokecopilot*, smokeantigravity, smokepi, dailycommunityattribution).
3. Implement K≥3 multi-variant core analysis for 9 unsupported_multi_variant experiments (deepreport, dailycodemetrics, ...).

Eval actions

1. Add verbosity evals for output_format workflows (deepreport, dailyissuesreport).
2. Add outcome evals for smoke workflows (caveman, subagent_model).
3. Add empty-output/success-rate evals for model_size experiments.

Pipeline gaps

• Resolve guardrail_unsupported / insufficient_observations metrics above.
• Normalize factorial interaction diagnostics (smokecopilot* caveman × subagent_model) into core analysis.
• gh aw experiments list --repo failed in this build; branches had to be fetched locally.

Run: 39920904764

Generated by 🧪 daily-experiment-report · copilot · auto · ~25 AIC · ⌖ ~1 AIC · ⊞ 17.1K · ◷

• expires on Oct 26, 2026, 12:00 AM UTC-08:00