[experiments] Daily Experiment Report — 2026-09-07

[github-actions[bot]]

Across all 48 workflows with active experiments: declarations (51 individual experiment analyses), zero experiments currently have decision-quality evidence. Every analysis returned either EXTEND (37) or INCONCLUSIVE (14) — no PROMOTE or REJECT decisions were possible today. The dominant blocker is a data-pipeline gap: guardrail/outcome metrics (run_success_rate, empty_output_rate, etc.) are declared in frontmatter but not yet wired into the core analysis engine as observable per-run data, and 14 experiments use 3+ variants, which the current two-variant decision engine cannot statistically resolve.

⚡ Quick Stats

Metric	Value
Active experiments (workflows)	48
Total experiment analyses	51
Ready for analysis (readiness=READY)	22
Still collecting (readiness=COLLECTING)	29
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

📋 Reason code breakdown

Reason code	Count	Meaning
insufficient_observations	26	2-variant experiments have reached min_samples but per-run outcome/guardrail metrics aren't yet captured, so the engine can't compute an effect
unsupported_multi_variant	14	Experiment declares 3+ variants; the automatic decision engine currently only supports exactly 2
insufficient_samples	8	2-variant experiment(s) below min_samples (default 20) in at least one arm
guardrail_unsupported	3	A configured guardrail metric has no native data source wired up

🟢 READY experiments (22) — sample size met, but no evidence yet

Workflow	Experiment	Reason code
agentperformanceanalyzer	prompt_compression	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	insufficient_observations
awfailureinvestigator	tone_variant (3-way)	unsupported_multi_variant
cicoach	prompt_style	guardrail_unsupported
dailyagentrxtraceoptimizer	sub_agent_strategy	insufficient_observations
dailyastrostylelitemarkdownspellcheck	prompt_style	insufficient_observations
dailycommunityattribution	prompt_style	insufficient_observations
dailyfact	reasoning_depth	guardrail_unsupported
dailynews	prompt_style	insufficient_observations
dailysecurityredteam	reasoning_depth	guardrail_unsupported
gpclean	tool_verbosity	insufficient_observations
issuearborist	prompt_style	insufficient_observations
smokeantigravity	sub_agent_strategy	insufficient_observations
smokecopilot	caveman / subagent_model	insufficient_observations
smokecopilotaoaiapikey	caveman / subagent_model	insufficient_observations
smokecopilotaoaientra	caveman / subagent_model	insufficient_observations
smokegemini	sub_agent_strategy	insufficient_observations
smokepi	sub_agent_decomposition	insufficient_observations
typist	tone_style	insufficient_observations

🟡 COLLECTING experiments (29) — still building sample size

29 experiments are below min_samples in at least one arm, or have only just started assigning variants (n=0–1). These include several fresh model_size A/B rollouts (dailycachestrategyanalyzer, dailyfunctionnamer, testqualitysentinel, smokeproject, smoketemporaryid, dailyrenderingscriptsverifier, prsouschef) that have logged only 0–1 usable observation so far, plus 8 two-variant experiments approaching but not yet at threshold (architectureguardian, auditworkflows, blogauditor, breakingchangechecker, dailyarchitecturediagram, dataflowprdiscussiondataset, dependabotcampaign, weeklyblogpostwriter), plus 13 multi-variant (3–5 arm) experiments awaiting native decision support (copilotagentanalysis, dailycavemanoptimizer, dailycodemetrics, dailycompilerquality, dailydochealer, dailydocupdater, dailyissuesreport, dailysemgrepscan, dailysubagentoptimizer, deepreport, dependabotgochecker, plan, smokecopilotsubagents).

📊 Summary

View Full Experiments Table (all 51 analyses)

Workflow	Experiment	Variants	Readiness	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	2	🟢 READY	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	2	🟢 READY	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	2	🟡 COLLECTING	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	2	🟡 COLLECTING	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	3	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	2	🟡 COLLECTING	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	2	🟡 COLLECTING	EXTEND	insufficient_samples
cicoach	prompt_style	2	🟢 READY	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailyagentrxtraceoptimizer	sub_agent_strategy	2	🟢 READY	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	2	🟡 COLLECTING	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	2	🟢 READY	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
dailycavemanoptimizer	model_size	5	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	2	🟢 READY	EXTEND	insufficient_observations
dailycompilerquality	output_format	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	5	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	5	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	2	🟢 READY	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
dailyissuesreport	output_format	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	2	🟢 READY	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	2	🟢 READY	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	2	🟡 COLLECTING	EXTEND	insufficient_samples
deepreport	output_format	4	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	2	🟡 COLLECTING	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	2	🟢 READY	EXTEND	insufficient_observations
issuearborist	prompt_style	2	🟢 READY	EXTEND	insufficient_observations
plan	reasoning_depth	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	2	🟢 READY	EXTEND	insufficient_observations
smokecopilot	caveman / subagent_model	2	🟢 READY	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman / subagent_model	2	🟢 READY	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman / subagent_model	2	🟢 READY	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	3	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	2	🟢 READY	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	2	🟢 READY	EXTEND	insufficient_observations
smokeproject	prompt_style_test	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
testqualitysentinel	model_size	0–1	🟡 COLLECTING	EXTEND	insufficient_observations
typist	tone_style	2	🟢 READY	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	2	🟡 COLLECTING	EXTEND	insufficient_samples

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire native per-run guardrail metrics (run_success_rate, empty_output_rate, ai_credits_total) into the analysis pipeline so the 26 insufficient_observations and 3 guardrail_unsupported experiments (29 of 51 analyses, all at/above sample threshold) can finally produce PROMOTE/REJECT decisions instead of perpetually EXTENDing.
2. Extend the core decision engine to support 3+ variant experiments (14 analyses currently blocked by unsupported_multi_variant, including several high-volume output_format/model_size tests with 80–260 total runs each) — this is the single largest unblocking change available.
3. For the 8 insufficient_samples experiments still below min_samples=20 per arm, no action needed yet beyond continued collection; re-check in ~1–2 weeks given current run cadence (daily/weekly triggers).

Top 3 eval actions

1. No eval/grader score data was available via experiments analyze in this run — evals are not yet correlated with experiment variant assignments. Recommend adding an eval-to-experiment join key so grader scores can serve as the "observation" layer the decision engine is missing.
2. Prioritize eval coverage for prompt_style/tone_variant/reasoning_depth experiments (10 workflows) since these are qualitative-output changes best assessed by graders rather than raw success/duration metrics.
3. For model_size experiments (6 workflows, mostly COLLECTING), pair cost (ai_credits_total) evals with quality graders so PROMOTE decisions can weigh both cost and quality guardrails once wired up.

Decision-pipeline gaps for next PR

• guardrail_unsupported/insufficient_observations: the majority failure mode (29/51 analyses) — no native metric source is feeding guardrail thresholds like run_success_rate >=0.90 even though runs clearly complete with a conclusion. This is the top-priority pipeline gap.
• unsupported_multi_variant: 14/51 analyses (27%) are blocked purely because the decision engine doesn't yet generalize beyond 2 arms — a multi-arm (e.g. ANOVA / pairwise-with-correction) path would unblock a large share of experiments immediately.
• Interaction diagnostics: no workflow in this run had 2+ concurrently active experiments assigned to the same runs, so no factorial interaction cells were computed this cycle; keep this check active as more workflows adopt multi-experiment designs.

Descriptive window: last 30 runs per workflow (not independently correlated to outcomes in this run due to missing native metrics) · Decision thresholds: from decision_policy (confidence 0.95, minimum_effect 0, regression_tolerance 0)
Run: 34097731744

Generated by 🧪 daily-experiment-report · copilot · auto · 76.2 AIC · ⌖ 8.46 AIC · ⊞ 16.5K · ◷

• expires on Sep 10, 2026, 12:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #59398.