[experiments] Daily Experiment Report — 2026-10-09

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-09

49 workflows with experiment state were analysed (55 experiments). No experiment reached PROMOTE or REJECT. 38 are EXTEND and 14 are INCONCLUSIVE (the remaining ones are listed in the table). Descriptive outcome charts were skipped this run. The local CLI build lacked outcome data, and the decisions come entirely from core analysis.

⚡ Quick Stats

Metric	Value
Active experiments	55
Readiness	🟢 READY: 28 · 🟡 COLLECTING: 27
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Reason codes	insufficient_observations: 26 · unsupported_multi_variant: 14 · insufficient_samples: 9 · guardrail_unsupported: 3 · (+3 other EXTEND)
Interaction safety holds	0 (no PROMOTE decisions; smokecopilot* pairs caveman×subagent_model have balanced n but no outcome observations)

Key findings

• Observation gap dominates: most READY experiments (e.g. smokecopilot 193/193, awfailureinvestigator, deepreport) have enough assignments but no metric observations, so core cannot decide.
• Multi-variant (K≥3) experiments are unsupported by core analysis → INCONCLUSIVE (14 experiments, including all output_format+ste tests).
• Guardrails unsupported: cicoach, dailyfact, dailysecurityredteam declare run_success_rate / empty_output_rate, which have no supported metric observation.

View Full Experiments Table

Workflow	Experiment	Readiness	Samples per variant	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	READY	caveman=76, verbose=64	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy	READY	batch=75, per_scenario=65	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	COLLECTING	single_agent=17, sub_agents=21	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	COLLECTING	phased_sub_agents=54, single_agent=40	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	READY	assertive=218, clinical=204, narrative=197	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	COLLECTING	concise=4, detailed=9	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	READY	neutral=32, urgent=28	EXTEND	insufficient_observations
cicoach	prompt_style	READY	concise=44, detailed=61	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	COLLECTING	prose=35, ste=4, structured=44	INCONCLUSIVE	unsupported_multi_variant
copilotprnlpanalysis	nlp_prompt_style	COLLECTING	concise=0, structured=0	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	READY	single_agent=48, sub_agents=46	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	COLLECTING	brief=12, comprehensive=8	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	READY	concise=82, detailed=75	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	COLLECTING	gpt-5.3-codex=20, gpt-5.3-codex-spark=13	EXTEND	insufficient_samples
dailycavemanoptimizer	model_size	COLLECTING	agent=1, claude-haiku-4.5=75, claude-sonnet-4.6=34, claude-sonnet-5=13, small-agent=2	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	COLLECTING	executive_summary=54, full_detail=58, ste=16	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	READY	concise=72, verbose=72	EXTEND	insufficient_observations
dailycompilerquality	output_format	READY	concise=56, detailed=66, ste=22	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	COLLECTING	agent=2, claude-haiku-4.5=66, claude-sonnet-4.6=43, claude-sonnet-5=15, small-agent=1	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	COLLECTING	agent=1, claude-haiku-4.5=51, claude-sonnet-4.6=47, claude-sonnet-5=17, small-agent=2	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	READY	multi_candidate=44, single_pass=61	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size	COLLECTING		EXTEND	insufficient_observations
dailyissuesreport	output_format	COLLECTING	collapsible=67, inline=64, ste=23	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	READY	concise=65, detailed=56	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1	COLLECTING		EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy	COLLECTING		EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	READY	iterative=85, single_pass=64	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	COLLECTING	bullet_list=28, prose=27, structured_sections=22	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	COLLECTING	default=2, relaxed=1, tight=1	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	COLLECTING	no=7, yes=11	EXTEND	insufficient_samples
deepreport	output_format	READY	annotated_brief=71, executive_brief=90, full_briefing=70, ste=53	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	COLLECTING	brief=6, detailed=6	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	COLLECTING	concise=20, detailed=19, step_by_step=18	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	READY	full_bash=65, minimal_toolset=72	EXTEND	insufficient_observations
issuearborist	prompt_style	READY	concise=52, detailed=44	EXTEND	insufficient_observations
plan	reasoning_depth	COLLECTING	baseline=4, deep=2, shallow=1	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1	COLLECTING		EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	READY	single_agent=99, sub_agents=109	EXTEND	insufficient_observations
smokecopilot	caveman	READY	no=193, yes=193	EXTEND	insufficient_observations
smokecopilot	subagent_model	READY	large=183, small=183	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	READY	no=99, yes=100	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	READY	large=100, small=99	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	READY	no=86, yes=86	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	READY	large=86, small=86	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	COLLECTING	delegated_sequential=18, inline_strict=17, single_agent_control=20	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	READY	single_agent=135, sub_agents=148	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	READY	parallel_sub_agents=42, single_agent=34	EXTEND	insufficient_observations
smokeproject	prompt_style_test	COLLECTING		EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy	COLLECTING		EXTEND	insufficient_observations
testqualitysentinel	model_size	READY	claude-haiku-4.5=160, claude-sonnet-5=166	EXTEND	insufficient_observations
typist	tone_style	READY	conversational=55, formal=43	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	COLLECTING	eager=7, lazy=10	EXTEND	insufficient_samples

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire primary-metric observations (token_count_per_run, output_length_chars, etc.) into experiment state for READY workflows to clear 26 insufficient_observations (start with smokecopilot*, awfailureinvestigator, deepreport).
2. Add core support for K≥3 variants (Bonferroni pairwise vs control) or reduce ste/tone_variant/model_size experiments to two arms; 14 are stuck at unsupported_multi_variant.
3. Keep collecting for 9 insufficient_samples experiments (e.g. blogauditor 4/9, copilotprnlpanalysis 0/0); verify copilotprnlpanalysis is actually assigning.

Top eval actions

1. Add graders emitting run_success_rate and empty_output_rate as native observations (maps to ci-coach, daily-fact, daily-security-red-team).
2. Add an output-quality grader for output_format experiments (daily-code-metrics, daily-issues-report, deep-report).
3. Add a token/cost grader for prompt_style/prompt_compression experiments (agent-performance-analyzer, daily-news, issue-arborist).

Decision-pipeline gaps

• guardrail_unsupported: run_success_rate, empty_output_rate.
• insufficient_observations is the main blocker (26).
• Interaction diagnostics (caveman×subagent_model in 3 smoke workflows) should become a normalized core signal.
• dailysubagentoptimizer/plan have 1–4 runs per arm; model-name variants such as agent/small-agent in model_size suggest assignment-value drift.

Descriptive window: n/a · Run: 37902237486

Generated by 🧪 daily-experiment-report · copilot · auto · 23.2 AIC · ⌖ 0.859 AIC · ⊞ 16.3K · ◷

• expires on Oct 12, 2026, 12:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #67383.