[experiments] Daily Experiment Report — 2026-10-04

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-04

49 workflows with experiment state (52 experiments) analysed via gh aw experiments analyze. No experiment has decision-quality evidence: 0 PROMOTE, 0 REJECT, 26 EXTEND, 14 INCONCLUSIVE.

⚡ Quick Stats

Metric	Value
Active experiments	52 (49 workflows)
Ready (🟢)	26
Collecting (🟡)	26
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (only smokecopilot* have 2 experiments; no PROMOTE to hold)

Reason codes: insufficient_observations 26 · insufficient_samples 9 · unsupported_multi_variant 14 · guardrail_unsupported 3.

📊 All experiments (core decisions)

Workflow	Experiment	Readiness	Control→Candidate	Decision	Reason code
agentperformanceanalyzer	prompt_compression	🟢 READY	caveman→verbose	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	🟢 READY	batch→per_scenario	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	🟡 COLLECTING	single_agent→sub_agents	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	🟡 COLLECTING	single_agent→phased_sub_agents	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	🟢 READY	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	🟡 COLLECTING	concise→detailed	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	🟢 READY	neutral→urgent	EXTEND	insufficient_observations
cicoach	prompt_style	🟢 READY	detailed→concise	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
copilotprnlpanalysis	nlp_prompt_style	🟡 COLLECTING	concise→structured	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	🟢 READY	single_agent→sub_agents	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	🟡 COLLECTING	brief→comprehensive	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	🟢 READY	concise→detailed	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	🟡 COLLECTING	gpt-5.3-codex→gpt-5.3-codex-spark	EXTEND	insufficient_samples
dailycavemanoptimizer	model_size	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	🟢 READY	concise→verbose	EXTEND	insufficient_observations
dailycompilerquality	output_format	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	🟢 READY	single_pass→multi_candidate	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	🟡 COLLECTING	(multi-variant/none)	EXTEND	insufficient_observations
dailyissuesreport	output_format	🟢 READY	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	🟢 READY	concise→detailed	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	🟡 COLLECTING	(multi-variant/none)	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	🟡 COLLECTING	eager→lazy	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	🟢 READY	single_pass→iterative	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	🟡 COLLECTING	no→yes	EXTEND	insufficient_samples
deepreport	output_format	🟢 READY	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	🟡 COLLECTING	brief→detailed	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	🟢 READY	full_bash→minimal_toolset	EXTEND	insufficient_observations
issuearborist	prompt_style	🟢 READY	concise→detailed	EXTEND	insufficient_observations
plan	reasoning_depth	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1	🟡 COLLECTING	(multi-variant/none)	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	🟢 READY	single_agent→sub_agents	EXTEND	insufficient_observations
smokecopilot	caveman	🟢 READY	no→yes	EXTEND	insufficient_observations
smokecopilot	subagent_model	🟢 READY	large→small	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	🟢 READY	no→yes	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	🟢 READY	large→small	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	🟢 READY	no→yes	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	🟢 READY	large→small	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	🟡 COLLECTING	(multi-variant/none)	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	🟢 READY	single_agent→sub_agents	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	🟢 READY	parallel_sub_agents→single_agent	EXTEND	insufficient_observations
smokeproject	prompt_style_test	🟡 COLLECTING	(multi-variant/none)	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	🟡 COLLECTING	single_agent→sub_agents	EXTEND	insufficient_observations
testqualitysentinel	model_size	🟢 READY	claude-haiku-4.5→claude-sonnet-5	EXTEND	insufficient_observations
typist	tone_style	🟢 READY	conversational→formal	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	🟡 COLLECTING	eager→lazy	EXTEND	insufficient_samples

Notes

• Readiness READY means assignment counts reached min_samples; the decision remains EXTEND because no outcome metric observations are recorded (insufficient_observations).
• cicoach, dailyfact, dailysecurityredteam: mandatory guardrails (run_success_rate, empty_output_rate) are unsupported by the metric pipeline.
• Multi-variant (K≥3) experiments are INCONCLUSIVE (unsupported_multi_variant); notably awfailureinvestigator (512 runs), deepreport, dailyissuesreport.
• auditworkflows has min_samples 264 (89 runs) and copilotprnlpanalysis has 0 recorded runs per variant.
• Per-run outcome charts and tracking-issue comments/labels were not produced: the decision layer has no outcome observations, so descriptive duration/success stats would not change any decision.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire primary-metric observations (e.g. token_count_per_run, output_length_chars) into experiment state for the 20 READY/insufficient_observations two-arm experiments, starting with the highest-volume: smokecopilot*, smokegemini, smokeantigravity, testqualitysentinel.
2. Keep EXTEND for the 9 insufficient_samples experiments; none need changes.
3. Consider collapsing K≥3 experiments (awfailureinvestigator, deepreport, dailyissuesreport) into pairwise control-vs-candidate designs, since the core rejects multi-variant.

Top eval actions

1. Add graders emitting run_success_rate and empty_output_rate for cicoach, dailyfact, dailysecurityredteam.
2. Add an output-quality grader for output_format/ste workflows (deepreport, dailycodemetrics, dailycompilerquality).
3. Add a proposal-quality grader for cicoach prompt_style.

Decision-pipeline gaps

• Unresolved: insufficient_observations (26) and guardrail_unsupported (3).
• Interaction diagnostics (smokecopilot* caveman × subagent_model) should become a normalized core signal.
• Imbalanced assignments (is_balanced=false) in several multi-variant experiments need a core signal.

Run: 37193588536

Generated by 🧪 daily-experiment-report · copilot · auto · 27.2 AIC · ⌖ 9.55 AIC · ⊞ 16.3K · ◷

• expires on Oct 7, 2026, 1:59 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #65809.