[experiments] Daily Experiment Report — 2026-10-11

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-11

52 experiments analysed across 49 workflows. None has decision-quality evidence: no PROMOTE or REJECT today. Most are blocked on missing outcome observations or unsupported multi-variant analysis.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis (🟢 READY)	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

Reason codes: insufficient_observations 26 · unsupported_multi_variant 14 · insufficient_samples 9 · guardrail_unsupported 3.

📊 Full Experiments Table

Workflow	Experiment	Readiness	Control	Candidate	Samples	min	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	🟢 READY	caveman	verbose	caveman=78, verbose=64	20	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	🟢 READY	batch	per_scenario	batch=77, per_scenario=65	20	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	🟡 COLLECTING	single_agent	sub_agents	single_agent=17, sub_agents=21	20	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	🟡 COLLECTING	single_agent	phased_sub_agents	phased_sub_agents=55, single_agent=41	264	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	🟢 READY	-	-	assertive=223, clinical=205, narrative=198	20	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	🟡 COLLECTING	concise	detailed	concise=4, detailed=9	20	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	🟢 READY	neutral	urgent	neutral=32, urgent=29	20	EXTEND	insufficient_observations
cicoach	prompt_style	🟢 READY	detailed	concise	concise=45, detailed=61	20	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	🟡 COLLECTING	-	-	prose=36, ste=4, structured=45	30	INCONCLUSIVE	unsupported_multi_variant
copilotprnlpanalysis	nlp_prompt_style	🟡 COLLECTING	concise	structured	concise=0, structured=0	30	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	🟢 READY	single_agent	sub_agents	single_agent=48, sub_agents=46	20	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	🟡 COLLECTING	brief	comprehensive	brief=12, comprehensive=8	20	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	🟢 READY	concise	detailed	concise=82, detailed=77	20	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	🟡 COLLECTING	gpt-5.3-codex	gpt-5.3-codex-spark	gpt-5.3-codex=21, gpt-5.3-codex-spark=14	20	EXTEND	insufficient_samples
dailycavemanoptimizer	model_size	🟡 COLLECTING	-	-	agent=1, claude-haiku-4.5=76, claude-sonnet-4.6=34, claude-sonnet-5=14, small-agent=2	20	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	🟡 COLLECTING	-	-	executive_summary=54, full_detail=58, ste=16	20	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	🟢 READY	concise	verbose	concise=73, verbose=73	20	EXTEND	insufficient_observations
dailycompilerquality	output_format	🟢 READY	-	-	concise=56, detailed=66, ste=22	20	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	🟡 COLLECTING	-	-	agent=2, claude-haiku-4.5=66, claude-sonnet-4.6=43, claude-sonnet-5=17, small-agent=1	20	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	🟡 COLLECTING	-	-	agent=1, claude-haiku-4.5=52, claude-sonnet-4.6=47, claude-sonnet-5=18, small-agent=2	20	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	🟢 READY	single_pass	multi_candidate	multi_candidate=45, single_pass=61	30	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	🟡 COLLECTING	-	-	none	20	EXTEND	insufficient_observations
dailyissuesreport	output_format	🟡 COLLECTING	-	-	collapsible=67, inline=64, ste=23	30	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	🟢 READY	concise	detailed	concise=65, detailed=57	20	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	🟡 COLLECTING	-	-	none	20	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	🟡 COLLECTING	eager	lazy	none	20	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	🟢 READY	single_pass	iterative	iterative=86, single_pass=65	30	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	🟡 COLLECTING	-	-	bullet_list=28, prose=27, structured_sections=22	30	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	🟡 COLLECTING	-	-	default=2, relaxed=1, tight=1	20	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	🟡 COLLECTING	no	yes	no=7, yes=11	20	EXTEND	insufficient_samples
deepreport	output_format	🟢 READY	-	-	annotated_brief=71, executive_brief=91, full_briefing=71, ste=53	20	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	🟡 COLLECTING	brief	detailed	brief=6, detailed=6	20	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	🟡 COLLECTING	-	-	concise=21, detailed=19, step_by_step=18	20	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	🟢 READY	full_bash	minimal_toolset	full_bash=66, minimal_toolset=73	20	EXTEND	insufficient_observations
issuearborist	prompt_style	🟢 READY	concise	detailed	concise=53, detailed=45	20	EXTEND	insufficient_observations
plan	reasoning_depth	🟡 COLLECTING	-	-	baseline=4, deep=2, shallow=1	20	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1	🟡 COLLECTING	-	-	none	20	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	🟢 READY	single_agent	sub_agents	single_agent=99, sub_agents=109	20	EXTEND	insufficient_observations
smokecopilot	caveman	🟢 READY	no	yes	no=193, yes=194	20	EXTEND	insufficient_observations
smokecopilot	subagent_model	🟢 READY	large	small	large=183, small=184	20	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	🟢 READY	no	yes	no=100, yes=100	20	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	🟢 READY	large	small	large=100, small=100	20	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	🟢 READY	no	yes	no=87, yes=86	20	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	🟢 READY	large	small	large=86, small=87	20	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	🟡 COLLECTING	-	-	delegated_sequential=19, inline_strict=17, single_agent_control=20	30	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	🟢 READY	single_agent	sub_agents	single_agent=136, sub_agents=148	20	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	🟢 READY	parallel_sub_agents	single_agent	parallel_sub_agents=42, single_agent=34	20	EXTEND	insufficient_observations
smokeproject	prompt_style_test	🟡 COLLECTING	-	-	none	20	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	🟡 COLLECTING	single_agent	sub_agents	none	20	EXTEND	insufficient_observations
testqualitysentinel	model_size	🟢 READY	claude-haiku-4.5	claude-sonnet-5	claude-haiku-4.5=194, claude-sonnet-5=190	20	EXTEND	insufficient_observations
typist	tone_style	🟢 READY	conversational	formal	conversational=55, formal=44	20	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	🟡 COLLECTING	eager	lazy	eager=7, lazy=10	20	EXTEND	insufficient_samples

Notes

• Charts and per-run outcome statistics (success rate, duration CIs) were not produced this run; the core decision does not depend on them. No tracking issue: fields were handled, so no issue comments or labels were posted.
• gh aw experiments list --repo was not supported by the local build; data was read from the experiments/* branches.
• Interaction diagnostics were not computed; no promotion decisions exist, so no safety hold applies.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire outcome observations for the 26 insufficient_observations experiments (READY by sample count, but no metric data), starting with high-volume ones: smokecopilot*, testqualitysentinel, awfailureinvestigator.
2. Provide a supported native metric for the run_success_rate guardrail in cicoach, dailyfact and dailysecurityredteam (guardrail_unsupported).
3. Reduce multi-variant experiments (14 unsupported_multi_variant, e.g. deepreport, dailycompilerquality, model_size experiments) to pairwise control/candidate, or add core support for K≥3 comparisons.

Top eval actions

1. Add an eval for the output_format variants (including ste), scoring verbosity and readability, for dailycodemetrics, dailyissuesreport and deepreport.
2. Add success and quality graders for prompt_style workflows (issuearborist, dailynews, dailycommunityattribution).
3. Add a grader for sub_agent_strategy outcomes across the smoke workflows.

Decision-pipeline gaps for next PR

• Experiments with null samples (dailyfunctionnamer, dailyrenderingscriptsverifier, prsouschef, smokeproject, smoketemporaryid, dailysafeoutputoptimizer) record no run data.
• copilotprnlpanalysis has 0 recorded samples per variant.
• Variant sets drifting (model_size shows stray agent / small-agent variants) and unbalanced assignment (copilotagentanalysis ste=4, chi-square p≈5e-7).
• Normalize K1×K2 interaction diagnostics into the core analysis output for workflows with two experiments (smokecopilot*).

Run: 38122947087

Generated by 🧪 daily-experiment-report · copilot · auto · 21 AIC · ⌖ 1.08 AIC · ⊞ 16.3K · ◷

• expires on Oct 14, 2026, 12:00 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #67931.