[experiments] Daily Experiment Report — 2026-10-15

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-15

49 workflows with experiment state were analysed (52 experiments). No experiment has decision-quality evidence: 38 are EXTEND and 14 are INCONCLUSIVE. There are no PROMOTE or REJECT decisions and no interaction safety holds.

⚡ Quick Stats

Metric	Value
Active experiments	52 (49 workflows)
🟢 READY / 🟡 COLLECTING	25 / 27
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (only smokecopilot* run 2 experiments per workflow; cell diagnostics not computed)

Reason codes: insufficient_observations 26 · insufficient_samples 9 · unsupported_multi_variant 14 · guardrail_unsupported 3.

⚠️ Charts and per-run outcome tables (success rate, duration CI) were not produced this run. The decisions below come entirely from gh aw experiments analyze.

🔎 Highlights

🟢 READY experiments blocked by missing observations (EXTEND)

Variant sample gates are met but there are no outcome observations for the metric:

• agentperformanceanalyzer/prompt_compression (caveman 80 / verbose 66)
• breakingchangechecker/tone_variant
• dailyagentrxtraceoptimizer/sub_agent_strategy
• dailyastrostylelitemarkdownspellcheck, dailycommunityattribution, dailynews, issuearborist (prompt_style)
• gpclean/tool_verbosity, testqualitysentinel/model_size, typist/tone_style
• smokeantigravity, smokegemini, smokepi
• smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra (caveman and subagent_model)

🛡️ Guardrail unsupported (data-pipeline gap)

• cicoach/prompt_style (detailed 62 vs concise 47): run_success_rate and empty_output_rate are unsupported
• dailyfact/reasoning_depth (single_pass 63 vs multi_candidate 46)
• dailysecurityredteam/reasoning_depth (single_pass 66 vs iterative 89)

⚪ INCONCLUSIVE (`unsupported_multi_variant`, 3+ variants)

awfailureinvestigator, copilotagentanalysis, dailycavemanoptimizer, dailycodemetrics, dailycompilerquality, dailydochealer, dailydocupdater, dailyissuesreport, dailysemgrepscan, dailysubagentoptimizer, deepreport, dependabotgochecker, plan, smokecopilotsubagents.

The core analysis cannot yet decide multi-variant experiments. It stays INCONCLUSIVE, which is not a rejection.

⚖️ Imbalanced assignment (chi-square p < 0.05)

• output_format in copilotagentanalysis: ste has only 4 runs against 38 prose and 47 structured
• output_format in dailycodemetrics: ste 16
• output_format in dailycompilerquality: ste 22
• output_format in dailyissuesreport: ste 23
• output_format in deepreport: ste 53
• model_size in dailycavemanoptimizer, dailydochealer and dailydocupdater: stray variants agent and small-agent with 1–2 runs each
• semgrep_output_format in dailysemgrepscan: balance p=0.023

The stray model_size variants suggest stale or mis-recorded variant names in state.

🟡 COLLECTING with no data yet

agentpersonaexplorer, dailyfunctionnamer, dailyrenderingscriptsverifier, dailysafeoutputoptimizer, prsouschef, smokeproject, smoketemporaryid have no usable samples.

📊 Summary

All experiments have Core decision: EXTEND or INCONCLUSIVE, and the report action equals the core decision for every one. Only dailycompilerquality declares a tracking issue (#32390). It is INCONCLUSIVE and READY, so no PROMOTE or guardrail-failure comment applies today.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire outcome observations (insufficient_observations, 26 experiments) for READY experiments, starting with smokecopilot* and dailynews.
2. Support cicoach, dailyfact and dailysecurityredteam guardrails natively (run_success_rate, empty_output_rate).
3. Investigate the ste under-assignment and the stray model_size variants.

Top eval actions

1. Add graders for token-count and output-length metrics on prompt_style workflows (dailynews, issuearborist, dailycommunityattribution).
2. Add output-format readability graders for output_format workflows (deepreport, dailycodemetrics, dailyissuesreport).
3. Add a success/empty-output grader for the reasoning_depth workflows.

Decision-pipeline gaps for the next PR

• Multi-variant (K≥3) support in core analysis, to resolve 14 unsupported_multi_variant results.
• A normalized interaction signal for the two-experiment smoke workflows.
• guardrail_unsupported metrics: run_success_rate, empty_output_rate.

Descriptive window: last 30 runs per workflow · Run: 38701818832

Generated by 🧪 daily-experiment-report · copilot · auto · 26 AIC · ⌖ 0.836 AIC · ⊞ 16.4K · ◷

• expires on Oct 18, 2026, 12:00 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #69294.