[experiments] Daily Experiment Report — 2026-10-06

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-06

52 experiments analysed across 49 workflows. None has decision-quality evidence: 0 PROMOTE, 0 REJECT, 38 EXTEND, 14 INCONCLUSIVE. No interaction safety holds.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis	27 (🟡 COLLECTING: 25)
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

Reason codes: insufficient_observations: 26, unsupported_multi_variant: 14, insufficient_samples: 9, guardrail_unsupported: 3

⚠️ Charts and per-run outcome statistics (success rate, duration CIs) were not generated this run. The decisions below come only from core gh aw experiments analyze output.

📊 All experiments (core decisions)

Experiment	Workflow	Readiness	Control → Candidate	Samples	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	🟢 READY	caveman → verbose	caveman=75 / verbose=62 (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	🟢 READY	batch → per_scenario	batch=73 / per_scenario=64 (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	🟡 COLLECTING	single_agent → sub_agents	single_agent=17 / sub_agents=21 (min 20)	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	🟡 COLLECTING	single_agent → phased_sub_agents	phased_sub_agents=52 / single_agent=39 (min 264)	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	🟢 READY	→	assertive=182 / clinical=173 / narrative=172 (min 20)	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	🟡 COLLECTING	concise → detailed	concise=4 / detailed=8 (min 20)	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	🟢 READY	neutral → urgent	neutral=30 / urgent=27 (min 20)	EXTEND	insufficient_observations
prompt_style	cicoach	🟢 READY	detailed → concise	concise=42 / detailed=60 (min 20)	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	🟡 COLLECTING	→	prose=34 / ste=3 / structured=43 (min 30)	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	🟡 COLLECTING	concise → structured	concise=0 / structured=0 (min 30)	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	🟢 READY	single_agent → sub_agents	single_agent=45 / sub_agents=46 (min 20)	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	🟡 COLLECTING	brief → comprehensive	brief=12 / comprehensive=8 (min 20)	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	🟢 READY	concise → detailed	concise=81 / detailed=74 (min 20)	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	🟡 COLLECTING	gpt-5.3-codex → gpt-5.3-codex-spark	gpt-5.3-codex=18 / gpt-5.3-codex-spark=12 (min 20)	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	🟡 COLLECTING	→	agent=1 / claude-haiku-4.5=75 / claude-sonnet-4.6=34 / claude-sonnet-5=10 / small-agent=2 (min 20)	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	🟡 COLLECTING	→	executive_summary=54 / full_detail=58 / ste=16 (min 20)	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	🟢 READY	concise → verbose	concise=70 / verbose=71 (min 20)	EXTEND	insufficient_observations
output_format	dailycompilerquality	🟢 READY	→	concise=56 / detailed=66 / ste=20 (min 20)	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	🟡 COLLECTING	→	agent=2 / claude-haiku-4.5=65 / claude-sonnet-4.6=43 / claude-sonnet-5=13 / small-agent=1 (min 20)	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	🟡 COLLECTING	→	agent=1 / claude-haiku-4.5=51 / claude-sonnet-4.6=47 / claude-sonnet-5=15 / small-agent=2 (min 20)	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	🟢 READY	single_pass → multi_candidate	multi_candidate=43 / single_pass=59 (min 30)	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer	🟡 COLLECTING	→	(min 20)	EXTEND	insufficient_observations
output_format	dailyissuesreport	🟢 READY	→	collapsible=66 / inline=63 / ste=23 (min 20)	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	🟢 READY	concise → detailed	concise=64 / detailed=54 (min 20)	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier	🟡 COLLECTING	→	(min 20)	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	🟡 COLLECTING	eager → lazy	(min 20)	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	🟢 READY	single_pass → iterative	iterative=84 / single_pass=62 (min 30)	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	🟡 COLLECTING	→	bullet_list=28 / prose=27 / structured_sections=22 (min 30)	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	🟡 COLLECTING	→	default=2 / relaxed=1 / tight=1 (min 20)	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	🟡 COLLECTING	no → yes	no=6 / yes=11 (min 20)	EXTEND	insufficient_samples
output_format	deepreport	🟢 READY	→	annotated_brief=68 / executive_brief=89 / full_briefing=63 / ste=52 (min 20)	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	🟡 COLLECTING	brief → detailed	brief=6 / detailed=6 (min 20)	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	🟡 COLLECTING	→	concise=20 / detailed=18 / step_by_step=18 (min 20)	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	🟢 READY	full_bash → minimal_toolset	full_bash=64 / minimal_toolset=70 (min 20)	EXTEND	insufficient_observations
prompt_style	issuearborist	🟢 READY	concise → detailed	concise=51 / detailed=42 (min 20)	EXTEND	insufficient_observations
reasoning_depth	plan	🟡 COLLECTING	→	baseline=4 / deep=2 / shallow=1 (min 20)	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef	🟡 COLLECTING	→	(min 20)	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	🟢 READY	single_agent → sub_agents	single_agent=99 / sub_agents=109 (min 20)	EXTEND	insufficient_observations
caveman	smokecopilot	🟢 READY	no → yes	no=191 / yes=191 (min 20)	EXTEND	insufficient_observations
subagent_model	smokecopilot	🟢 READY	large → small	large=181 / small=181 (min 20)	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	🟢 READY	no → yes	no=98 / yes=98 (min 20)	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	🟢 READY	large → small	large=98 / small=98 (min 20)	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	🟢 READY	no → yes	no=85 / yes=84 (min 20)	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	🟢 READY	large → small	large=85 / small=84 (min 20)	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	🟡 COLLECTING	→	delegated_sequential=17 / inline_strict=16 / single_agent_control=20 (min 30)	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	🟢 READY	single_agent → sub_agents	single_agent=133 / sub_agents=146 (min 20)	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	🟢 READY	parallel_sub_agents → single_agent	parallel_sub_agents=42 / single_agent=34 (min 20)	EXTEND	insufficient_observations
prompt_style_test	smokeproject	🟡 COLLECTING	→	(min 20)	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	🟡 COLLECTING	single_agent → sub_agents	(min 20)	EXTEND	insufficient_observations
model_size	testqualitysentinel	🟢 READY	claude-haiku-4.5 → claude-sonnet-5	claude-haiku-4.5=118 / claude-sonnet-5=118 (min 20)	EXTEND	insufficient_observations
tone_style	typist	🟢 READY	conversational → formal	conversational=52 / formal=43 (min 20)	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	🟡 COLLECTING	eager → lazy	eager=7 / lazy=10 (min 20)	EXTEND	insufficient_samples

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. insufficient_observations (26): READY experiments lack metric observations. Wire declared primary metrics into the observation pipeline (evals/graders).
2. unsupported_multi_variant (14): core analysis cannot decide for K≥3 variants. Add pairwise control-vs-candidate decisions with Bonferroni correction, or reduce variants.
3. guardrail_unsupported (3: ci-coach, dailyfact, dailysecurityredteam): run_success_rate / empty_output_rate are not backed by supported metrics. Add native run-outcome observations.

Top eval actions

1. Add graders emitting token_count_per_run / output_length_chars for prompt_style and output_format experiments (ci-coach, dailycompilerquality, deepreport).
2. Add outcome graders for sub_agent_strategy experiments (agentpersonaexplorer, dailyagentrxtraceoptimizer, smoke workflows).
3. Add run-success and empty-output observations for reasoning_depth experiments.

Decision-pipeline gaps for the next PR

• Metrics/guardrails behind insufficient_observations and guardrail_unsupported: see counts above.
• Normalize the factorial interaction diagnostic (smokecopilot*: caveman × subagent_model) into a core analysis signal.
• model_size experiments in dailycavemanoptimizer, dailydochealer and dailydocupdater have stray variants (agent, small-agent) with 1–2 samples.

Run: 37432835407

Generated by 🧪 daily-experiment-report · copilot · auto · 28.9 AIC · ⌖ 8.94 AIC · ⊞ 16.3K · ◷

• expires on Oct 9, 2026, 12:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #66501.