[experiments] Daily Experiment Report — 2026-10-12

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-12

52 experiments analysed across 49 workflows. No experiment has a PROMOTE or REJECT decision. 38 are EXTEND and 14 are INCONCLUSIVE. Of the 52 experiments, 25 are 🟢 READY on sample counts, but none has usable outcome observations yet.

⚡ Quick Stats

Metric	Value
Active experiments	52 (49 workflows)
Ready for analysis	25
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (only smokecopilot* workflows have 2 experiments, and none has a PROMOTE decision)

Reason codes: insufficient_observations 26 · insufficient_samples 9 · unsupported_multi_variant 14 · guardrail_unsupported 3.

📈 All experiments (core decisions)

Experiment	Workflow	Readiness	Variants (n)	Decision	Reason code
prompt_compression	agentperformanceanalyzer	🟢 READY	caveman=79, verbose=64	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	🟢 READY	batch=77, per_scenario=66	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	🟡 COLLECTING	single_agent=17, sub_agents=21	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	🟡 COLLECTING	phased_sub_agents=56, single_agent=41	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	🟢 READY	assertive=226, clinical=205, narrative=199	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	🟡 COLLECTING	concise=4, detailed=9	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	🟢 READY	neutral=32, urgent=29	EXTEND	insufficient_observations
prompt_style	cicoach	🟢 READY	concise=45, detailed=61	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	🟡 COLLECTING	prose=36, ste=4, structured=46	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	🟡 COLLECTING	concise=0, structured=1	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	🟢 READY	single_agent=49, sub_agents=46	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	🟡 COLLECTING	brief=12, comprehensive=8	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	🟢 READY	concise=83, detailed=77	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	🟡 COLLECTING	gpt-5.3-codex=22, gpt-5.3-codex-spark=14	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	🟡 COLLECTING	agent=1, claude-haiku-4.5=76, claude-sonnet-4.6=34, claude-sonnet-5=15, small-agent=2	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	🟡 COLLECTING	executive_summary=54, full_detail=58, ste=16	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	🟢 READY	concise=73, verbose=74	EXTEND	insufficient_observations
output_format	dailycompilerquality	🟢 READY	concise=56, detailed=66, ste=22	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	🟡 COLLECTING	agent=2, claude-haiku-4.5=66, claude-sonnet-4.6=43, claude-sonnet-5=18, small-agent=1	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	🟡 COLLECTING	agent=1, claude-haiku-4.5=53, claude-sonnet-4.6=47, claude-sonnet-5=18, small-agent=2	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	🟢 READY	multi_candidate=45, single_pass=61	EXTEND	guardrail_unsupported
output_format	dailyissuesreport	🟡 COLLECTING	collapsible=67, inline=64, ste=23	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	🟢 READY	concise=65, detailed=57	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	🟢 READY	iterative=87, single_pass=65	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	🟡 COLLECTING	bullet_list=28, prose=27, structured_sections=22	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	🟡 COLLECTING	default=2, relaxed=1, tight=1	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	🟡 COLLECTING	no=7, yes=11	EXTEND	insufficient_samples
output_format	deepreport	🟢 READY	annotated_brief=71, executive_brief=91, full_briefing=71, ste=53	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	🟡 COLLECTING	brief=6, detailed=6	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	🟡 COLLECTING	concise=21, detailed=19, step_by_step=18	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	🟢 READY	full_bash=66, minimal_toolset=74	EXTEND	insufficient_observations
prompt_style	issuearborist	🟢 READY	concise=53, detailed=46	EXTEND	insufficient_observations
reasoning_depth	plan	🟡 COLLECTING	baseline=4, deep=2, shallow=1	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokeantigravity	🟢 READY	single_agent=99, sub_agents=109	EXTEND	insufficient_observations
caveman	smokecopilot	🟢 READY	no=193, yes=194	EXTEND	insufficient_observations
subagent_model	smokecopilot	🟢 READY	large=183, small=184	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	🟢 READY	no=100, yes=100	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	🟢 READY	large=100, small=100	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	🟢 READY	no=87, yes=86	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	🟢 READY	large=86, small=87	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	🟡 COLLECTING	delegated_sequential=19, inline_strict=17, single_agent_control=20	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	🟢 READY	single_agent=136, sub_agents=148	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	🟢 READY	parallel_sub_agents=42, single_agent=34	EXTEND	insufficient_observations
tone_style	typist	🟢 READY	conversational=55, formal=44	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	🟡 COLLECTING	eager=7, lazy=11	EXTEND	insufficient_samples

🛠️ Self-Tuning Continuation Plan

Top experiment actions

1. The 26 insufficient_observations experiments need an outcome metric emitted per run (for example, via evals or graders). They are already READY on counts, so this is the largest block.
2. The 14 unsupported_multi_variant experiments (tone_variant, output_format with ste, model_size, and others) are not covered by the core pairwise decision. Either support K≥3 comparisons in core or split them into control-vs-candidate pairs.
3. Three experiments (cicoach, dailyfact, dailysecurityredteam) return guardrail_unsupported. Implement the native guardrail metrics that they declare.

Top eval actions

1. Add an output-quality eval for the output_format experiments (copilotagentanalysis, dailycodemetrics, dailycompilerquality, dailyissuesreport, deepreport). Score readability and completeness for the ste variants.
2. Add an eval for sub-agent decomposition quality (smoke*, agentpersonaexplorer, dailyagentrxtraceoptimizer).
3. Add a task-success eval for prompt_style and tone workflows (issuearborist, typist, dailynews, breakingchangechecker).

Decision-pipeline gaps

• Native metrics are missing, as shown by 26 insufficient_observations and 3 guardrail_unsupported.
• decision_policy shows minimum_effect: 0 and regression_tolerance: 0 for awfailureinvestigator. Declare explicit thresholds.
• Factorial interaction diagnostics should become a normalized core signal. The smokecopilot* workflows have two experiments each (caveman × subagent_model).
• Several model_size experiments contain stray variants (agent, small-agent), which suggests assignment-name drift.

Charts were skipped because no per-variant outcome data is available. Descriptive window: counts from experiment branch state. Tracking-issue notifications and labels were not applied. Run: 38222105920

Generated by 🧪 daily-experiment-report · copilot · auto · 22.5 AIC · ⌖ 0.926 AIC · ⊞ 16.3K · ◷

• expires on Oct 15, 2026, 12:24 AM UTC-08:00