[experiments] Daily Experiment Report — 2026-10-05

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-05

52 experiments across 49 workflows were analysed. No experiment reached PROMOTE or REJECT. 38 are EXTEND and 14 are INCONCLUSIVE, all with unsupported_multi_variant. 26 are 🟢 READY on sample counts but blocked by missing outcome observations.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis (sample gate)	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no workflow has 2+ experiments except the smoke Copilot workflows, which lack outcome data)

Reason codes: insufficient_observations 26 · insufficient_samples 9 · unsupported_multi_variant 14 · guardrail_unsupported 3

📊 All experiments (core decisions)

Workflow	Experiment	Readiness	Samples	Balance p	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	🟢 READY	caveman=74, verbose=62	0.304	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	🟢 READY	batch=72, per_scenario=64	0.5	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	🟡 COLLECTING	single_agent=17, sub_agents=21	0.524	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	🟡 COLLECTING	phased_sub_agents=51, single_agent=39	0.203	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	🟢 READY	assertive=178, clinical=170, narrative=168	0.848	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	🟡 COLLECTING	concise=4, detailed=8	0.247	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	🟢 READY	neutral=30, urgent=26	0.6	EXTEND	insufficient_observations
cicoach	prompt_style	🟢 READY	concise=41, detailed=60	0	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	🟡 COLLECTING	prose=34, ste=3, structured=43	0	INCONCLUSIVE	unsupported_multi_variant
copilotprnlpanalysis	nlp_prompt_style	🟡 COLLECTING	concise=0, structured=0	0.319	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	🟢 READY	single_agent=44, sub_agents=46	0.816	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	🟡 COLLECTING	brief=12, comprehensive=7	0.25	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	🟢 READY	concise=81, detailed=73	0.527	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	🟡 COLLECTING	gpt-5.3-codex=17, gpt-5.3-codex-spark=12	0.356	EXTEND	insufficient_samples
dailycavemanoptimizer	model_size	🟡 COLLECTING	agent=1, claude-haiku-4.5=75, claude-sonnet-4.6=34, claude-sonnet-5=10, small-agent=2	0	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	🟡 COLLECTING	executive_summary=54, full_detail=58, ste=16	0	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	🟢 READY	concise=70, verbose=70	1	EXTEND	insufficient_observations
dailycompilerquality	output_format	🟡 COLLECTING	concise=56, detailed=66, ste=19	0	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	🟡 COLLECTING	agent=2, claude-haiku-4.5=64, claude-sonnet-4.6=43, claude-sonnet-5=13, small-agent=1	0	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	🟡 COLLECTING	agent=1, claude-haiku-4.5=51, claude-sonnet-4.6=47, claude-sonnet-5=15, small-agent=2	0	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	🟢 READY	multi_candidate=43, single_pass=58	0.131	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	🟡 COLLECTING		0	EXTEND	insufficient_observations
dailyissuesreport	output_format	🟢 READY	collapsible=65, inline=63, ste=23	0	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	🟢 READY	concise=63, detailed=54	0.41	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	🟡 COLLECTING		0	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	🟡 COLLECTING		0	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	🟢 READY	iterative=84, single_pass=61	0.053	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	🟡 COLLECTING	bullet_list=28, prose=27, structured_sections=22	0.023	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	🟡 COLLECTING	default=2, relaxed=1, tight=1	0.781	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	🟡 COLLECTING	no=6, yes=11	0.223	EXTEND	insufficient_samples
deepreport	output_format	🟢 READY	annotated_brief=68, executive_brief=88, full_briefing=62, ste=50	0.01	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	🟡 COLLECTING	brief=6, detailed=6	1	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	🟡 COLLECTING	concise=20, detailed=18, step_by_step=17	0.876	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	🟢 READY	full_bash=64, minimal_toolset=69	0.668	EXTEND	insufficient_observations
issuearborist	prompt_style	🟢 READY	concise=51, detailed=41	0.298	EXTEND	insufficient_observations
plan	reasoning_depth	🟡 COLLECTING	baseline=4, deep=2, shallow=1	0.369	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1	🟡 COLLECTING		0	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	🟢 READY	single_agent=99, sub_agents=109	0.495	EXTEND	insufficient_observations
smokecopilot	caveman	🟢 READY	no=190, yes=190	1	EXTEND	insufficient_observations
smokecopilot	subagent_model	🟢 READY	large=180, small=180	1	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	🟢 READY	no=98, yes=98	1	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	🟢 READY	large=98, small=98	1	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	🟢 READY	no=85, yes=84	0.897	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	🟢 READY	large=85, small=84	0.897	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	🟡 COLLECTING	delegated_sequential=17, inline_strict=16, single_agent_control=20	0.745	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	🟢 READY	single_agent=133, sub_agents=146	0.442	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	🟢 READY	parallel_sub_agents=42, single_agent=34	0.362	EXTEND	insufficient_observations
smokeproject	prompt_style_test	🟡 COLLECTING		0	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	🟡 COLLECTING		0	EXTEND	insufficient_observations
testqualitysentinel	model_size	🟢 READY	claude-haiku-4.5=100, claude-sonnet-5=112	0.415	EXTEND	insufficient_observations
typist	tone_style	🟢 READY	conversational=52, formal=42	0.303	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	🟡 COLLECTING	eager=7, lazy=10	0.474	EXTEND	insufficient_samples

Notes

• This run did not produce per-variant outcome charts. Run-level outcome records were not collected, so success-rate and duration statistics are omitted. They would not change the core decisions.
• Tracking-issue comments and labels were skipped. The workflow frontmatter checked did not declare an experiment issue: field.
• Imbalanced assignment (is_balanced=false, chi-square p<0.05): copilotagentanalysis (ste n=3 vs 34/43), dailycavemanoptimizer, dailydochealer, dailydocupdater (model_size heavily skewed toward claude-haiku-4.5), dailycodemetrics, dailycompilerquality, dailyissuesreport, deepreport (ste arms), dailysemgrepscan. Check the assignment weights.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire a native outcome metric (insufficient_observations) for the 26 READY experiments. The smoke workflows (smokecopilot*, smokegemini, smokeantigravity) are the cheapest place to start, since they already have 85–380 runs per arm.
2. Define run_success_rate as a supported observation for cicoach, dailyfact and dailysecurityredteam (guardrail_unsupported).
3. Reduce the 14 multi-variant experiments to a control plus one candidate, or add multi-variant support to core analysis. Examples: deepreport with 250 runs, awfailureinvestigator with 516 runs.

Top eval actions

1. Add graders emitting a numeric score for the prompt_style experiments (dailynews, issuearborist, dailycommunityattribution) and map them to the metric.
2. Add output-length and token-count observations for the output_format experiments (ste variants).
3. Add cost/latency observations for the model_size experiments (testqualitysentinel, dailydocupdater).

Decision-pipeline gaps

• Unresolved native metrics: insufficient_observations (26) and guardrail_unsupported (3).
• Interaction diagnostics (K1×K2 cells) for the workflows with 2 experiments (smokecopilot*) should become a normalized core signal.
• dailyfunctionnamer, dailyrenderingscriptsverifier, prsouschef, smokeproject, smoketemporaryid, dailysafeoutputoptimizer and copilotprnlpanalysis have no or near-zero assignments. Verify that state is being pushed.

Run: 37281504031

Generated by 🧪 daily-experiment-report · copilot · auto · 25.2 AIC · ⌖ 8.42 AIC · ⊞ 16.3K · ◷

• expires on Oct 8, 2026, 12:11 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #66068.