[experiments] Daily Experiment Report — 2026-10-03

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-03

52 experiments across 49 workflows were analysed from the experiments/* state branches. 26 are READY on sample counts, but none has decision-quality evidence. There are no PROMOTE or REJECT decisions, so no tracking-issue comments or labels were emitted.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (only the smokecopilot* workflows run two experiments; no PROMOTE, so no hold was needed)

🔎 Key findings

• 26 × insufficient_observations: assignment counts reached min_samples, but no outcome observations were recorded for the primary metric. This is the main pipeline gap.
• 3 × guardrail_unsupported (ci-coach, daily-fact, daily-security-red-team): the guardrails run_success_rate and empty_output_rate have status unsupported.
• 14 × unsupported_multi_variant (K ≥ 3 variants, including every ste output_format test and the model_size tests): core analysis does not yet decide multi-variant experiments.
• 9 × insufficient_samples: still collecting (for example blog-auditor at 4/20 and 8/20).
• Several model_size experiments contain stray variants (agent, small-agent, with 1–2 runs each).
• audit-workflows has min_samples 264 and is at 38/264 on its smaller variant.

📊 Full Experiments Table

Experiment	Workflow	Assignments	min_samples	Readiness	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	caveman=73, verbose=61	20	READY	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	batch=70, per_scenario=64	20	READY	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	single_agent=17, sub_agents=21	20	COLLECTING	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	phased_sub_agents=50, single_agent=38	264	COLLECTING	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	assertive=175, clinical=167, narrative=166	20	READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	concise=4, detailed=8	20	COLLECTING	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	neutral=30, urgent=26	20	READY	EXTEND	insufficient_observations
prompt_style	cicoach	concise=41, detailed=60	20	READY	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	prose=34, ste=3, structured=43	30	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	concise=0, structured=1	30	COLLECTING	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent=44, sub_agents=45	20	READY	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	brief=12, comprehensive=7	20	COLLECTING	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	concise=79, detailed=73	20	READY	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	gpt-5.3-codex=16, gpt-5.3-codex-spark=11	20	COLLECTING	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	agent=1, claude-haiku-4.5=74, claude-sonnet-4.6=34, claude-sonnet-5=9, small-agent=2	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	executive_summary=54, full_detail=58, ste=16	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	concise=70, verbose=70	20	READY	EXTEND	insufficient_observations
output_format	dailycompilerquality	concise=55, detailed=66, ste=18	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	agent=2, claude-haiku-4.5=64, claude-sonnet-4.6=43, claude-sonnet-5=11, small-agent=1	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	agent=1, claude-haiku-4.5=50, claude-sonnet-4.6=47, claude-sonnet-5=14, small-agent=2	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	multi_candidate=43, single_pass=58	30	READY	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer		20	COLLECTING	EXTEND	insufficient_observations
output_format	dailyissuesreport	collapsible=64, inline=63, ste=22	20	READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	concise=63, detailed=54	20	READY	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier		20	COLLECTING	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer		20	COLLECTING	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	iterative=83, single_pass=60	30	READY	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	bullet_list=28, prose=27, structured_sections=22	30	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	default=2, relaxed=1, tight=1	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	no=6, yes=11	20	COLLECTING	EXTEND	insufficient_samples
output_format	deepreport	annotated_brief=67, executive_brief=86, full_briefing=61, ste=47	20	READY	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	brief=6, detailed=6	20	COLLECTING	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	concise=20, detailed=18, step_by_step=17	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	full_bash=63, minimal_toolset=68	20	READY	EXTEND	insufficient_observations
prompt_style	issuearborist	concise=50, detailed=40	20	READY	EXTEND	insufficient_observations
reasoning_depth	plan	baseline=4, deep=2, shallow=1	20	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef		20	COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	single_agent=99, sub_agents=109	20	READY	EXTEND	insufficient_observations
caveman	smokecopilot	no=190, yes=189	20	READY	EXTEND	insufficient_observations
subagent_model	smokecopilot	large=179, small=180	20	READY	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	no=97, yes=98	20	READY	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	large=98, small=97	20	READY	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	no=84, yes=84	20	READY	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	large=84, small=84	20	READY	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	delegated_sequential=17, inline_strict=16, single_agent_control=19	30	COLLECTING	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	single_agent=133, sub_agents=145	20	READY	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	parallel_sub_agents=42, single_agent=34	20	READY	EXTEND	insufficient_observations
prompt_style_test	smokeproject		20	COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid		20	COLLECTING	EXTEND	insufficient_observations
model_size	testqualitysentinel	claude-haiku-4.5=73, claude-sonnet-5=90	20	READY	EXTEND	insufficient_observations
tone_style	typist	conversational=52, formal=42	20	READY	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	eager=6, lazy=10	20	COLLECTING	EXTEND	insufficient_samples

Charts and outcome-level success/duration statistics were skipped: the experiment state holds assignment counts only, and no outcome observations were available for correlation.
Run: 37107931429

🔧 Self-Tuning Continuation Plan

Top experiment actions

1. Wire primary-metric observations into experiment state (fixes 26 insufficient_observations, e.g. the smokecopilot* and smoke* sub_agent_strategy tests).
2. Implement observations for run_success_rate and empty_output_rate (unblocks ci-coach, daily-fact, daily-security-red-team).
3. Prune stray variants (agent, small-agent) from the model_size experiments and keep collecting data for blog-auditor, architecture-guardian and weekly-blog-post-writer.

Eval actions

1. Add graders emitting output_length_chars and output_token_count for the ste output_format experiments (deep-report, daily-issues-report, daily-compiler-quality, daily-code-metrics, copilot-agent-analysis).
2. Tighten the eval questions for the prompt_style workflows (ci-coach, daily-news, issue-arborist, daily-astro-style-lite-markdown-spellcheck) so they return a per-run score.
3. Add a pass/fail eval for the sub_agent_strategy smoke tests.

Decision-pipeline gaps for the next PR

• Add K ≥ 3 variant support to core analysis (Bonferroni or pairwise vs control).
• Normalise native metric support so guardrail_unsupported becomes resolvable.
• Make the factorial interaction diagnostic (smokecopilot* caveman × subagent_model) a normalized core signal.

Generated by 🧪 daily-experiment-report · copilot · auto · 24.4 AIC · ⌖ 16.7 AIC · ⊞ 16.3K · ◷

• expires on Oct 6, 2026, 12:01 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #65545.