[experiments] Daily Experiment Report — 2026-09-29

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-29

48 workflows with 51 experiments were analysed. No experiment has a PROMOTE or REJECT decision. 31 are EXTEND and 14 are INCONCLUSIVE. Most READY experiments are blocked by missing outcome observations, not by sample size.

⚡ Quick Stats

Metric	Value
Active experiments	51 (48 workflows; 3 have two experiments, 2 workflows have no runs yet)
Ready for analysis (READY)	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

Reason codes:

Readiness	Decision	Reason code	Count
🟢 READY	EXTEND	insufficient_observations	20
🟢 READY	EXTEND	guardrail_unsupported	3
🟢 READY	INCONCLUSIVE	unsupported_multi_variant	3
🟡 COLLECTING	EXTEND	insufficient_samples	8
🟡 COLLECTING	EXTEND	insufficient_observations	6
🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	11

Outcome charts (success rate and duration) were not generated. The analysis output has no per-run outcome observations, which is why most experiments report insufficient_observations. Duration and success statistics would be descriptive only and could not change any decision.

📊 All experiments (core decisions)

Experiment	Workflow	Variant assignments	min_samples	Readiness	Core decision	Reason code	Report action
prompt_compression	agentperformanceanalyzer	caveman=71, verbose=59	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
sub_agent_strategy	agentpersonaexplorer	batch=67, per_scenario=63	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
sub_agent_strategy	architectureguardian	single_agent=17, sub_agents=21	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
audit_decomposition	auditworkflows	phased_sub_agents=47, single_agent=38	264	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
tone_variant	awfailureinvestigator	assertive=163, clinical=156, narrative=159	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
prompt_style	blogauditor	concise=4, detailed=7	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
tone_variant	breakingchangechecker	neutral=27, urgent=25	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
prompt_style	cicoach	concise=39, detailed=58	20	🟢 READY	EXTEND	guardrail_unsupported	EXTEND
output_format	copilotagentanalysis	prose=34, ste=3, structured=43	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent=43, sub_agents=43	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
detail_level	dailyarchitecturediagram	brief=12, comprehensive=7	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
prompt_style	dailyastrostylelitemarkdownspellcheck	concise=76, detailed=72	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
model_size	dailycachestrategyanalyzer	gpt-5.3-codex=12, gpt-5.3-codex-spark=11	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
model_size	dailycavemanoptimizer	agent=1, claude-haiku-4.5=71, claude-sonnet-4.6=34, claude-sonnet-5=9, small-agent=2	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
output_format	dailycodemetrics	executive_summary=54, full_detail=58, ste=16	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
prompt_style	dailycommunityattribution	concise=70, verbose=70	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
output_format	dailycompilerquality	concise=55, detailed=63, ste=17	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
model_size	dailydochealer	agent=2, claude-haiku-4.5=60, claude-sonnet-4.6=43, claude-sonnet-5=11, small-agent=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
model_size	dailydocupdater	agent=1, claude-haiku-4.5=49, claude-sonnet-4.6=47, claude-sonnet-5=13, small-agent=2	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
reasoning_depth	dailyfact	multi_candidate=41, single_pass=56	30	🟢 READY	EXTEND	guardrail_unsupported	EXTEND
model_size	dailyfunctionnamer	none yet	20	🟡 COLLECTING	EXTEND	insufficient_observations	EXTEND
output_format	dailyissuesreport	collapsible=60, inline=63, ste=22	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
prompt_style	dailynews	concise=62, detailed=51	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
remove_redundant_context_v1	dailyrenderingscriptsverifier	none yet	20	🟡 COLLECTING	EXTEND	insufficient_observations	EXTEND
log_fetch_strategy	dailysafeoutputoptimizer	none yet	20	🟡 COLLECTING	EXTEND	insufficient_observations	EXTEND
reasoning_depth	dailysecurityredteam	iterative=81, single_pass=58	30	🟢 READY	EXTEND	guardrail_unsupported	EXTEND
semgrep_output_format	dailysemgrepscan	bullet_list=28, prose=27, structured_sections=22	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer	default=2, relaxed=1, tight=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
caveman_mode	dataflowprdiscussiondataset	no=6, yes=10	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
output_format	deepreport	annotated_brief=62, executive_brief=84, full_briefing=58, ste=41	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
summary_detail	dependabotcampaign	brief=6, detailed=6	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND
prompt_style	dependabotgochecker	concise=19, detailed=18, step_by_step=16	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
tool_verbosity	gpclean	full_bash=60, minimal_toolset=67	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
prompt_style	issuearborist	concise=47, detailed=39	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
reasoning_depth	plan	baseline=4, deep=2, shallow=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
remove_redundant_context_v1	prsouschef	none yet	20	🟡 COLLECTING	EXTEND	insufficient_observations	EXTEND
sub_agent_strategy	smokeantigravity	single_agent=99, sub_agents=109	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
caveman	smokecopilot	no=186, yes=186	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
subagent_model	smokecopilot	large=176, small=176	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
caveman	smokecopilotaoaiapikey	no=95, yes=95	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
subagent_model	smokecopilotaoaiapikey	large=95, small=95	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
caveman	smokecopilotaoaientra	no=81, yes=82	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
subagent_model	smokecopilotaoaientra	large=82, small=81	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
sub_agent_strategy	smokecopilotsubagents	delegated_sequential=17, inline_strict=14, single_agent_control=19	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant	INCONCLUSIVE
sub_agent_strategy	smokegemini	single_agent=129, sub_agents=144	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
sub_agent_decomposition	smokepi	parallel_sub_agents=42, single_agent=34	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
prompt_style_test	smokeproject	none yet	20	🟡 COLLECTING	EXTEND	insufficient_observations	EXTEND
sub_agent_strategy	smoketemporaryid	none yet	20	🟡 COLLECTING	EXTEND	insufficient_observations	EXTEND
model_size	testqualitysentinel	claude-haiku-4.5=47, claude-sonnet-5=44	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
tone_style	typist	conversational=48, formal=42	20	🟢 READY	EXTEND	insufficient_observations	EXTEND
prefetch_strategy	weeklyblogpostwriter	eager=6, lazy=10	20	🟡 COLLECTING	EXTEND	insufficient_samples	EXTEND

🔀 Interaction diagnostics (workflows with two experiments)

caveman × subagent_model cells (n each). All cells have n ≥ 20, so there is no SPARSE_CELL_RISK.

Workflow	no/large	no/small	yes/large	yes/small	interaction p
smokecopilot	69	76	76	68	0.444
smokecopilotaoaiapikey	30	48	48	30	0.0065
smokecopilotaoaientra	36	28	29	36	0.252

⚠️ smokecopilotaoaiapikey shows a non-independent assignment pattern. This is a possible allocation or assignment-correlation bug and should be checked.

🔁 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire outcome observations (token_count_per_run, run success, and the workflow metrics) into the state branch. This clears 26 insufficient_observations decisions; the 20 that are READY on that reason are the first ones to unblock.
2. Support multi-variant (K≥3) analysis in core. This is 14 unsupported_multi_variant results, including deepreport, awfailureinvestigator and the output_format/ste experiments.
3. Give cicoach, dailyfact and dailysecurityredteam a supported source for run_success_rate and empty_output_rate (guardrail_unsupported).

Top 3 eval actions

• No eval scores were available in this run. Map evals to the output_format workflows (deepreport, dailyissuesreport, dailycodemetrics) to get readability observations.
• Add graders for the sub_agent_strategy workflows (smokegemini, smokeantigravity, dailyagentrxtraceoptimizer), which have the highest sample counts.
• Add a token-count observation for the prompt_style workflows (issuearborist, dailynews, dailycommunityattribution).

Decision-pipeline gaps for the next PR

• Interaction diagnostics (chi-square on assignment cells) should become a normalized core signal. smokecopilotaoaiapikey p=0.0065 is not currently surfaced by core.
• The CLI experiments list --repo flag is unsupported, and the CLI needed experiment branches fetched locally.
• Five experiments have no assignments yet (prsouschef, dailyrenderingscriptsverifier, dailysafeoutputoptimizer, smokeproject, smoketemporaryid) and dailyfunctionnamer has none recorded. Check that push_experiments_state runs.

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy
Run: 36539608878

Generated by 🧪 daily-experiment-report · copilot · auto · 28.3 AIC · ⌖ 10.5 AIC · ⊞ 16.3K · ◷

• expires on Oct 2, 2026, 12:02 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #64442.