[experiments] Daily Experiment Report — 2026-10-22

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-22

53 experiments across 50 workflows analysed. No experiment has a PROMOTE or REJECT decision: 44 are EXTEND and 9 are INCONCLUSIVE. No interaction safety holds apply because no core decision is PROMOTE.

⚡ Quick Stats

Metric	Value
Active experiments	53
🟢 READY / 🟡 COLLECTING	26 / 27
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 44 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 9
Interaction safety holds	0

Reason codes: insufficient_observations 16 · guardrail_unsupported 15 · insufficient_samples 13 · unsupported_multi_variant 9.

🔍 Why nothing is decided

• 🟢 READY but guardrail_unsupported (15): the mandatory guardrail (e.g. run_success_rate) has no supported metric observation. Affected workflows: dailyagentrxtraceoptimizer, issuearborist, gpclean, dailyastrostylelitemarkdownspellcheck, testqualitysentinel, dailydochealer, dailysecurityredteam, dailycachestrategyanalyzer, dailyfact, cicoach, agentperformanceanalyzer, typist, dailynews, smokegemini, dailydocupdater. These are the closest to a decision; fixing the data pipeline unblocks them.
• insufficient_observations (16): assignments exist but metric observations are missing. Includes smokecopilot*, smokeantigravity, smokepi, dailycommunityattribution (all READY on sample count), and awfailureinvestigator, dailysafeoutputoptimizer, agentpersonaexplorer, smoketemporaryid, smokeproject, prsouschef, dailyfunctionnamer.
• insufficient_samples (13): still collecting, e.g. auditworkflows (61/46 of 264 needed), breakingchangechecker (35/33 of 194), blogauditor (5/10 of 20), dailycavemanoptimizer (haiku 83, sonnet 18 of 20). Zero runs yet for dailyrenderingscriptsverifier, gofan, smokecopilotsubagents, copilotprnlpanalysis.
• unsupported_multi_variant → INCONCLUSIVE (9): three or more variants, which core analysis cannot compare: dailycodemetrics, copilotagentanalysis, dependabotgochecker, deepreport, dailycompilerquality, dailyissuesreport, dailysemgrepscan, plan, dailysubagentoptimizer. Note the ste variant is the common third arm in the output_format experiments.

📊 Full experiments table

Experiment	Workflow	Readiness	Samples	min_samples	Core decision	Reason code
sub_agent_strategy	dailyagentrxtraceoptimizer	READY	52/51	20	EXTEND	guardrail_unsupported
prompt_style	issuearborist	READY	59/50	30	EXTEND	guardrail_unsupported
tool_verbosity	gpclean	READY	70/80	30	EXTEND	guardrail_unsupported
prompt_style	dailyastrostylelitemarkdownspellcheck	READY	87/83	30	EXTEND	guardrail_unsupported
prompt_style	dailycommunityattribution	READY	78/79	20	EXTEND	insufficient_observations
model_size	testqualitysentinel	READY	144/178	70	EXTEND	guardrail_unsupported
model_size	dailydochealer	READY	24/70	20	EXTEND	guardrail_unsupported
reasoning_depth	dailysecurityredteam	READY	71/91	30	EXTEND	guardrail_unsupported
model_size	dailycachestrategyanalyzer	READY	24/22	20	EXTEND	guardrail_unsupported
reasoning_depth	dailyfact	READY	65/49	30	EXTEND	guardrail_unsupported
prompt_style	cicoach	READY	65/49	20	EXTEND	guardrail_unsupported
prompt_compression	agentperformanceanalyzer	READY	70/83	14	EXTEND	guardrail_unsupported
tone_style	typist	READY	48/59	10	EXTEND	guardrail_unsupported
prompt_style	dailynews	READY	61/69	30	EXTEND	guardrail_unsupported
sub_agent_strategy	smokegemini	READY	140/153	30	EXTEND	guardrail_unsupported
model_size	dailydocupdater	READY	20/56	20	EXTEND	guardrail_unsupported
caveman, subagent_model	smokecopilot	READY	206/206, 196/196	20	EXTEND	insufficient_observations
caveman, subagent_model	smokecopilotaoaiapikey	READY	104/105	20	EXTEND	insufficient_observations
caveman, subagent_model	smokecopilotaoaientra	READY	91/91	20	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	READY	99/109	20	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	READY	42/34	20	EXTEND	insufficient_observations
output_format	deepreport	READY	71/91/71/53	15	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycompilerquality	READY	56/66/22	20	INCONCLUSIVE	unsupported_multi_variant
remaining 27 experiments	see above	COLLECTING	varies	varies	EXTEND (18) / INCONCLUSIVE (7)	insufficient_samples, insufficient_observations, unsupported_multi_variant

🛠️ Self-Tuning Continuation Plan

Top experiment actions

1. Wire a supported run_success_rate observation so the 15 guardrail_unsupported experiments (all with sample counts met) can produce a decision.
2. Emit primary-metric observations for the READY insufficient_observations experiments (smokecopilot*, smokeantigravity, smokepi, dailycommunityattribution).
3. Reduce 3+ variant experiments (output_format with ste, etc.) to a control/candidate pair or add multi-variant support to core analysis.

Top eval actions

1. Tighten evals for cicoach and dailynews prompts, where prompt_style has many samples but no observations.
2. Add graders emitting output_length_chars / token counts for output_format workflows.
3. Add a success-rate grader for the model_size experiments (dailydochealer, dailydocupdater, testqualitysentinel).

Decision-pipeline gaps for next PR

• Native metric support for run_success_rate guardrails (guardrail_unsupported).
• Observation ingestion for insufficient_observations cases.
• Multi-variant comparison and factorial interaction diagnostics as normalized core signals.
• Charts and per-run outcome stats were not generated this run: the 30-run outcome correlation was skipped, and the report uses only core analyze output.

Descriptive window: core analyze output · Decision thresholds from decision_policy
Run: 39743892696

Generated by 🧪 daily-experiment-report · copilot · auto · 23.9 AIC · ⌖ 0.865 AIC · ⊞ 17.1K · ◷

• expires on Oct 25, 2026, 12:02 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #71394.