[experiments] Daily Experiment Report — 2026-08-25

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-08-25

47 experiments analysed across 44 workflows. 23 are ready for analysis. Core decisions: 0 PROMOTE, 32 EXTEND, 0 REJECT, 15 INCONCLUSIVE. No promotions or rejections yet — all active experiments are still accumulating evidence or lack a supported outcome metric to compute effect size.

⚡ Quick Stats

Metric	Value
Active experiments	47
Ready for analysis	23
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 32 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 15
Interaction safety holds	0

---

prompt_compression · agentperformanceanalyzer

Readiness: 🟢 READY
Variants: caveman vs verbose · min_samples: 20 per variant
Balance: balanced (p=0.6853)

📈 View Sample Progress

Variant	n	Progress
caveman	50	██████████ 50/20 (100%)
verbose	46	██████████ 46/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_strategy · agentpersonaexplorer

Readiness: 🟢 READY
Variants: batch vs per_scenario · min_samples: 20 per variant
Balance: balanced (p=0.3084)

📈 View Sample Progress

Variant	n	Progress
batch	53	██████████ 53/20 (100%)
per_scenario	43	██████████ 43/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_strategy · architectureguardian

Readiness: 🟡 COLLECTING
Variants: single_agent vs sub_agents · min_samples: 20 per variant
Balance: balanced (p=0.5239)

📈 View Sample Progress

Variant	n	Progress
single_agent	17	████████░░ 17/20 (85%)
sub_agents	21	██████████ 21/20 (100%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

audit_decomposition · auditworkflows

Readiness: 🟡 COLLECTING
Variants: phased_sub_agents vs single_agent · min_samples: 264 per variant
Balance: balanced (p=0.1191)

H0: no change in run_success_rate. H1: phased_sub_agents improves run_success_rate by at least 15% relative while keeping runtime within +10%.

📈 View Sample Progress

Variant	n	Progress
phased_sub_agents	31	█░░░░░░░░░ 31/264 (12%)
single_agent	20	█░░░░░░░░░ 20/264 (8%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

tone_variant · awfailureinvestigator

Readiness: 🟢 READY
Variants: assertive vs clinical vs narrative · min_samples: 20 per variant
Balance: balanced (p=0.3762)

📈 View Sample Progress

Variant	n	Progress
assertive	112	██████████ 112/20 (100%)
clinical	124	██████████ 124/20 (100%)
narrative	103	██████████ 103/20 (100%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · blogauditor

Readiness: 🟡 COLLECTING
Variants: concise vs detailed · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
concise	3	██░░░░░░░░ 3/20 (15%)
detailed	3	██░░░░░░░░ 3/20 (15%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

tone_variant · breakingchangechecker

Readiness: 🟡 COLLECTING
Variants: neutral vs urgent · min_samples: 20 per variant
Balance: balanced (p=0.4719)

📈 View Sample Progress

Variant	n	Progress
neutral	17	████████░░ 17/20 (85%)
urgent	13	██████░░░░ 13/20 (65%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

prompt_style · cicoach

Readiness: 🟢 READY
Variants: concise vs detailed · min_samples: 20 per variant
Balance: balanced (p=0.0000)

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Sample Progress

Variant	n	Progress
concise	31	██████████ 31/20 (100%)
detailed	41	██████████ 41/20 (100%)

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

output_format · copilotagentanalysis

Readiness: 🟡 COLLECTING
Variants: prose vs ste vs structured · min_samples: 30 per variant
Balance: imbalanced (p=0.0000)

H0: no change in ai_credits_used. H1: prose or ste format reduces ai_credits_used by >=15% while keeping empty_discussion_rate <=5%

📈 View Sample Progress

Variant	n	Progress
prose	33	██████████ 33/30 (100%)
ste	3	█░░░░░░░░░ 3/30 (10%)
structured	41	██████████ 41/30 (100%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

sub_agent_strategy · dailyagentrxtraceoptimizer

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · min_samples: 20 per variant
Balance: balanced (p=0.8667)

📈 View Sample Progress

Variant	n	Progress
single_agent	30	██████████ 30/20 (100%)
sub_agents	31	██████████ 31/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

detail_level · dailyarchitecturediagram

Readiness: 🟡 COLLECTING
Variants: brief vs comprehensive · min_samples: 20 per variant
Balance: balanced (p=0.5998)

📈 View Sample Progress

Variant	n	Progress
brief	8	████░░░░░░ 8/20 (40%)
comprehensive	6	███░░░░░░░ 6/20 (30%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

prompt_style · dailyastrostylelitemarkdownspellcheck

Readiness: 🟢 READY
Variants: concise vs detailed · min_samples: 20 per variant
Balance: balanced (p=0.4601)

📈 View Sample Progress

Variant	n	Progress
concise	61	██████████ 61/20 (100%)
detailed	53	██████████ 53/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

model_size · dailycachestrategyanalyzer

Readiness: 🟡 COLLECTING
Variants: agent vs gpt-5-codex vs gpt-5-mini vs gpt-5.4 vs gpt-5.4-mini vs small-agent · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

H0: no change in issue creation rate or run success rate. H1: gpt-5.4-mini reduces AI Credits while keeping run success rate >=0.90.

📈 View Sample Progress

Variant	n	Progress
agent	1	░░░░░░░░░░ 1/20 (5%)
gpt-5-codex	8	████░░░░░░ 8/20 (40%)
gpt-5-mini	2	█░░░░░░░░░ 2/20 (10%)
gpt-5.4	37	██████████ 37/20 (100%)
gpt-5.4-mini	31	██████████ 31/20 (100%)
small-agent	1	░░░░░░░░░░ 1/20 (5%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

model_size · dailycavemanoptimizer

Readiness: 🟡 COLLECTING
Variants: agent vs claude-haiku-4.5 vs claude-sonnet-4.6 vs small-agent · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress
agent	1	░░░░░░░░░░ 1/20 (5%)
claude-haiku-4.5	46	██████████ 46/20 (100%)
claude-sonnet-4.6	33	██████████ 33/20 (100%)
small-agent	2	█░░░░░░░░░ 2/20 (10%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

output_format · dailycodemetrics

Readiness: 🟡 COLLECTING
Variants: executive_summary vs full_detail vs ste · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress
executive_summary	42	██████████ 42/20 (100%)
full_detail	52	██████████ 52/20 (100%)
ste	6	███░░░░░░░ 6/20 (30%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · dailycommunityattribution

Readiness: 🟢 READY
Variants: concise vs verbose · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
concise	56	██████████ 56/20 (100%)
verbose	56	██████████ 56/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

output_format · dailycompilerquality

Readiness: 🟡 COLLECTING
Variants: concise vs detailed vs ste · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress
concise	46	██████████ 46/20 (100%)
detailed	50	██████████ 50/20 (100%)
ste	5	██░░░░░░░░ 5/20 (25%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

model_size · dailydochealer

Readiness: 🟡 COLLECTING
Variants: agent vs claude-haiku-4.5 vs claude-sonnet-4.6 vs small-agent · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress
agent	2	█░░░░░░░░░ 2/20 (10%)
claude-haiku-4.5	40	██████████ 40/20 (100%)
claude-sonnet-4.6	39	██████████ 39/20 (100%)
small-agent	1	░░░░░░░░░░ 1/20 (5%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

model_size · dailydocupdater

Readiness: 🟡 COLLECTING
Variants: agent vs claude-haiku-4.5 vs claude-sonnet-4.6 vs small-agent · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress
agent	1	░░░░░░░░░░ 1/20 (5%)
claude-haiku-4.5	34	██████████ 34/20 (100%)
claude-sonnet-4.6	45	██████████ 45/20 (100%)
small-agent	2	█░░░░░░░░░ 2/20 (10%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

reasoning_depth · dailyfact

Readiness: 🟡 COLLECTING
Variants: multi_candidate vs single_pass · min_samples: 30 per variant
Balance: imbalanced (p=0.0440)

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

📈 View Sample Progress

Variant	n	Progress
multi_candidate	28	█████████░ 28/30 (93%)
single_pass	45	██████████ 45/30 (100%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

model_size · dailyfunctionnamer

Readiness: 🟡 COLLECTING
Variants: · min_samples: 20 per variant
Balance: balanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress

Core decision: 🟡 EXTEND (insufficient_observations) — at least two variants are required for a decision

---

output_format · dailyissuesreport

Readiness: 🟡 COLLECTING
Variants: collapsible vs inline vs ste · min_samples: 20 per variant
Balance: imbalanced (p=0.0000)

📈 View Sample Progress

Variant	n	Progress
collapsible	47	██████████ 47/20 (100%)
inline	54	██████████ 54/20 (100%)
ste	11	██████░░░░ 11/20 (55%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · dailynews

Readiness: 🟢 READY
Variants: concise vs detailed · min_samples: 20 per variant
Balance: balanced (p=0.5300)

📈 View Sample Progress

Variant	n	Progress
concise	47	██████████ 47/20 (100%)
detailed	41	██████████ 41/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

log_fetch_strategy · dailysafeoutputoptimizer

Readiness: 🟢 READY
Variants: eager vs lazy · min_samples: 20 per variant
Balance: balanced (p=0.0000)

H0: no change in run_duration_ms. H1: eager reduces run duration by >=15% by avoiding MCP log-fetch turns

📈 View Sample Progress

Variant	n	Progress
eager	26	██████████ 26/20 (100%)
lazy	26	██████████ 26/20 (100%)

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "issue_creation_success_rate" is not backed by a supported metric observation

---

reasoning_depth · dailysecurityredteam

Readiness: 🟢 READY
Variants: iterative vs single_pass · min_samples: 30 per variant
Balance: imbalanced (p=0.0235)

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

📈 View Sample Progress

Variant	n	Progress
iterative	64	██████████ 64/30 (100%)
single_pass	41	██████████ 41/30 (100%)

Core decision: 🟡 EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

semgrep_output_format · dailysemgrepscan

Readiness: 🟡 COLLECTING
Variants: bullet_list vs prose vs structured_sections · min_samples: 30 per variant
Balance: imbalanced (p=0.0232)

H0: no change in alert creation rate across formats. H1: structured_sections produces ≥15% more alerts successfully created vs. baseline bullet_list; ste improves completeness via clearer, simpler language.

📈 View Sample Progress

Variant	n	Progress
bullet_list	28	█████████░ 28/30 (93%)
prose	27	█████████░ 27/30 (90%)
structured_sections	22	███████░░░ 22/30 (73%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

timeout_setting · dailysubagentoptimizer

Readiness: 🟡 COLLECTING
Variants: default vs relaxed vs tight · min_samples: 20 per variant
Balance: balanced (p=0.7814)

📈 View Sample Progress

Variant	n	Progress
default	2	█░░░░░░░░░ 2/20 (10%)
relaxed	1	░░░░░░░░░░ 1/20 (5%)
tight	1	░░░░░░░░░░ 1/20 (5%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

caveman_mode · dataflowprdiscussiondataset

Readiness: 🟡 COLLECTING
Variants: no vs yes · min_samples: 10 per variant
Balance: balanced (p=0.3691)

H0: no change in input_token_count, retention_rate, or run_success_rate. H1: caveman prompt cuts input tokens ≥30% with retention_rate ≥0.80 and run_success_rate ≥0.90

📈 View Sample Progress

Variant	n	Progress
no	4	████░░░░░░ 4/10 (40%)
yes	7	███████░░░ 7/10 (70%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

output_format · deepreport

Readiness: 🟡 COLLECTING
Variants: annotated_brief vs executive_brief vs full_briefing vs ste · min_samples: 20 per variant
Balance: imbalanced (p=0.0001)

📈 View Sample Progress

Variant	n	Progress
annotated_brief	28	██████████ 28/20 (100%)
executive_brief	42	██████████ 42/20 (100%)
full_briefing	31	██████████ 31/20 (100%)
ste	8	████░░░░░░ 8/20 (40%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

summary_detail · dependabotcampaign

Readiness: 🟡 COLLECTING
Variants: brief vs detailed · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
brief	6	███░░░░░░░ 6/20 (30%)
detailed	6	███░░░░░░░ 6/20 (30%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

prompt_style · dependabotgochecker

Readiness: 🟡 COLLECTING
Variants: concise vs detailed vs step_by_step · min_samples: 20 per variant
Balance: balanced (p=0.9620)

📈 View Sample Progress

Variant	n	Progress
concise	13	██████░░░░ 13/20 (65%)
detailed	13	██████░░░░ 13/20 (65%)
step_by_step	12	██████░░░░ 12/20 (60%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

tool_verbosity · gpclean

Readiness: 🟢 READY
Variants: full_bash vs minimal_toolset · min_samples: 20 per variant
Balance: balanced (p=0.2527)

📈 View Sample Progress

Variant	n	Progress
full_bash	41	██████████ 41/20 (100%)
minimal_toolset	52	██████████ 52/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · issuearborist

Readiness: 🟢 READY
Variants: concise vs detailed · min_samples: 20 per variant
Balance: balanced (p=0.1616)

📈 View Sample Progress

Variant	n	Progress
concise	31	██████████ 31/20 (100%)
detailed	21	██████████ 21/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

reasoning_depth · plan

Readiness: 🟡 COLLECTING
Variants: baseline vs deep vs shallow · min_samples: 20 per variant
Balance: balanced (p=0.3694)

📈 View Sample Progress

Variant	n	Progress
baseline	4	██░░░░░░░░ 4/20 (20%)
deep	2	█░░░░░░░░░ 2/20 (10%)
shallow	1	░░░░░░░░░░ 1/20 (5%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

sub_agent_strategy · smokeantigravity

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · min_samples: 20 per variant
Balance: balanced (p=0.4952)

📈 View Sample Progress

Variant	n	Progress
single_agent	99	██████████ 99/20 (100%)
sub_agents	109	██████████ 109/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman · smokecopilot

Readiness: 🟢 READY
Variants: no vs yes · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
no	170	██████████ 170/20 (100%)
yes	170	██████████ 170/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

subagent_model · smokecopilot

Readiness: 🟢 READY
Variants: large vs small · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
large	160	██████████ 160/20 (100%)
small	160	██████████ 160/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman · smokecopilotaoaiapikey

Readiness: 🟢 READY
Variants: no vs yes · min_samples: 20 per variant
Balance: balanced (p=0.8973)

📈 View Sample Progress

Variant	n	Progress
no	84	██████████ 84/20 (100%)
yes	85	██████████ 85/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

subagent_model · smokecopilotaoaiapikey

Readiness: 🟢 READY
Variants: large vs small · min_samples: 20 per variant
Balance: balanced (p=0.8973)

📈 View Sample Progress

Variant	n	Progress
large	85	██████████ 85/20 (100%)
small	84	██████████ 84/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman · smokecopilotaoaientra

Readiness: 🟢 READY
Variants: no vs yes · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
no	71	██████████ 71/20 (100%)
yes	71	██████████ 71/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

subagent_model · smokecopilotaoaientra

Readiness: 🟢 READY
Variants: large vs small · min_samples: 20 per variant
Balance: balanced (p=1.0000)

📈 View Sample Progress

Variant	n	Progress
large	71	██████████ 71/20 (100%)
small	71	██████████ 71/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_strategy · smokecopilotsubagents

Readiness: 🟡 COLLECTING
Variants: delegated_sequential vs inline_strict vs single_agent_control · min_samples: 30 per variant
Balance: balanced (p=0.3433)

H0: no change in pass_rate between inline_strict and delegated_sequential. H1: delegated_sequential improves pass_rate by >= 0.15 absolute versus inline_strict. Note: single_agent_control is a synthetic negative baseline (always FAIL by design) excluded from H1 comparisons.

📈 View Sample Progress

Variant	n	Progress
delegated_sequential	13	████░░░░░░ 13/30 (43%)
inline_strict	7	██░░░░░░░░ 7/30 (23%)
single_agent_control	12	████░░░░░░ 12/30 (40%)

Core decision: ⚪ INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

sub_agent_strategy · smokegemini

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · min_samples: 20 per variant
Balance: balanced (p=0.2048)

📈 View Sample Progress

Variant	n	Progress
single_agent	116	██████████ 116/20 (100%)
sub_agents	136	██████████ 136/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_decomposition · smokepi

Readiness: 🟢 READY
Variants: parallel_sub_agents vs single_agent · min_samples: 20 per variant
Balance: balanced (p=0.3619)

📈 View Sample Progress

Variant	n	Progress
parallel_sub_agents	42	██████████ 42/20 (100%)
single_agent	34	██████████ 34/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

model_size · testqualitysentinel

Readiness: 🟢 READY
Variants: claude-haiku-4.5 vs claude-sonnet-4.6 · min_samples: 20 per variant
Balance: balanced (p=0.1152)

📈 View Sample Progress

Variant	n	Progress
claude-haiku-4.5	606	██████████ 606/20 (100%)
claude-sonnet-4.6	553	██████████ 553/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_style · typist

Readiness: 🟢 READY
Variants: conversational vs formal · min_samples: 20 per variant
Balance: balanced (p=0.3895)

📈 View Sample Progress

Variant	n	Progress
conversational	36	██████████ 36/20 (100%)
formal	29	██████████ 29/20 (100%)

Core decision: 🟡 EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prefetch_strategy · weeklyblogpostwriter

Readiness: 🟡 COLLECTING
Variants: eager vs lazy · min_samples: 20 per variant
Balance: balanced (p=0.3691)

📈 View Sample Progress

Variant	n	Progress
eager	4	██░░░░░░░░ 4/20 (20%)
lazy	7	████░░░░░░ 7/20 (35%)

Core decision: 🟡 EXTEND (insufficient_samples) — one or more variants have fewer usable observations than min_samples

---

📊 Summary

View Full Experiments Table

Experiment	Workflow	Variants	Samples	Balanced	Core decision	Reason
prompt_compression	agentperformanceanalyzer	caveman, verbose	caveman=50, verbose=46	✅	🟡 EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	batch, per_scenario	batch=53, per_scenario=43	✅	🟡 EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	single_agent, sub_agents	single_agent=17, sub_agents=21	✅	🟡 EXTEND	insufficient_samples
audit_decomposition	auditworkflows	phased_sub_agents, single_agent	phased_sub_agents=31, single_agent=20	✅	🟡 EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	assertive, clinical, narrative	assertive=112, clinical=124, narrative=103	✅	⚪ INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	concise, detailed	concise=3, detailed=3	✅	🟡 EXTEND	insufficient_samples
tone_variant	breakingchangechecker	neutral, urgent	neutral=17, urgent=13	✅	🟡 EXTEND	insufficient_samples
prompt_style	cicoach	concise, detailed	concise=31, detailed=41	✅	🟡 EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	prose, ste, structured	prose=33, ste=3, structured=41	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent, sub_agents	single_agent=30, sub_agents=31	✅	🟡 EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	brief, comprehensive	brief=8, comprehensive=6	✅	🟡 EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	concise, detailed	concise=61, detailed=53	✅	🟡 EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	agent, gpt-5-codex, gpt-5-mini, gpt-5.4, gpt-5.4-mini, small-agent	agent=1, gpt-5-codex=8, gpt-5-mini=2, gpt-5.4=37, gpt-5.4-mini=31, small-agent=1	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailycavemanoptimizer	agent, claude-haiku-4.5, claude-sonnet-4.6, small-agent	agent=1, claude-haiku-4.5=46, claude-sonnet-4.6=33, small-agent=2	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	executive_summary, full_detail, ste	executive_summary=42, full_detail=52, ste=6	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	concise, verbose	concise=56, verbose=56	✅	🟡 EXTEND	insufficient_observations
output_format	dailycompilerquality	concise, detailed, ste	concise=46, detailed=50, ste=5	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	agent, claude-haiku-4.5, claude-sonnet-4.6, small-agent	agent=2, claude-haiku-4.5=40, claude-sonnet-4.6=39, small-agent=1	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	agent, claude-haiku-4.5, claude-sonnet-4.6, small-agent	agent=1, claude-haiku-4.5=34, claude-sonnet-4.6=45, small-agent=2	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	multi_candidate, single_pass	multi_candidate=28, single_pass=45	❌	🟡 EXTEND	insufficient_samples
model_size	dailyfunctionnamer			✅	🟡 EXTEND	insufficient_observations
output_format	dailyissuesreport	collapsible, inline, ste	collapsible=47, inline=54, ste=11	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	concise, detailed	concise=47, detailed=41	✅	🟡 EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	eager, lazy	eager=26, lazy=26	✅	🟡 EXTEND	guardrail_unsupported
reasoning_depth	dailysecurityredteam	iterative, single_pass	iterative=64, single_pass=41	❌	🟡 EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	bullet_list, prose, structured_sections	bullet_list=28, prose=27, structured_sections=22	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	default, relaxed, tight	default=2, relaxed=1, tight=1	✅	⚪ INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	no, yes	no=4, yes=7	✅	🟡 EXTEND	insufficient_samples
output_format	deepreport	annotated_brief, executive_brief, full_briefing, ste	annotated_brief=28, executive_brief=42, full_briefing=31, ste=8	❌	⚪ INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	brief, detailed	brief=6, detailed=6	✅	🟡 EXTEND	insufficient_samples
prompt_style	dependabotgochecker	concise, detailed, step_by_step	concise=13, detailed=13, step_by_step=12	✅	⚪ INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	full_bash, minimal_toolset	full_bash=41, minimal_toolset=52	✅	🟡 EXTEND	insufficient_observations
prompt_style	issuearborist	concise, detailed	concise=31, detailed=21	✅	🟡 EXTEND	insufficient_observations
reasoning_depth	plan	baseline, deep, shallow	baseline=4, deep=2, shallow=1	✅	⚪ INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokeantigravity	single_agent, sub_agents	single_agent=99, sub_agents=109	✅	🟡 EXTEND	insufficient_observations
caveman	smokecopilot	no, yes	no=170, yes=170	✅	🟡 EXTEND	insufficient_observations
subagent_model	smokecopilot	large, small	large=160, small=160	✅	🟡 EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	no, yes	no=84, yes=85	✅	🟡 EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	large, small	large=85, small=84	✅	🟡 EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	no, yes	no=71, yes=71	✅	🟡 EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	large, small	large=71, small=71	✅	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	delegated_sequential, inline_strict, single_agent_control	delegated_sequential=13, inline_strict=7, single_agent_control=12	✅	⚪ INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	single_agent, sub_agents	single_agent=116, sub_agents=136	✅	🟡 EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	parallel_sub_agents, single_agent	parallel_sub_agents=42, single_agent=34	✅	🟡 EXTEND	insufficient_observations
model_size	testqualitysentinel	claude-haiku-4.5, claude-sonnet-4.6	claude-haiku-4.5=606, claude-sonnet-4.6=553	✅	🟡 EXTEND	insufficient_observations
tone_style	typist	conversational, formal	conversational=36, formal=29	✅	🟡 EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	eager, lazy	eager=4, lazy=7	✅	🟡 EXTEND	insufficient_samples

🔧 Self-Tuning Continuation Plan

Top experiment actions:

1. Prioritize wiring run_success_rate / other guardrail metrics into eval graders for cicoach, daily-safe-output-optimizer, and daily-security-red-team — currently blocked on guardrail_unsupported, so no PROMOTE/REJECT decision is possible even at READY sample sizes.
2. Add a native outcome-metric observation pipeline for token_count_per_run, output_length_chars, and duration proxies so insufficient_observations experiments (e.g. agent-performance-analyzer, testqualitysentinel, smoke-*) can progress past READY once metric data lands.
3. Extend the core analysis engine to support >2 variant decisions (currently unsupported_multi_variant blocks 15 experiments: awfailureinvestigator, dailycachestrategyanalyzer, dailycavemanoptimizer, dailycodemetrics, dailycompilerquality, dailydochealer, dailydocupdater, dailyissuesreport, dailysemgrepscan, dailysubagentoptimizer, deepreport, dependabotgochecker, plan, smokecopilotsubagents, copilotagentanalysis).

Top eval actions:

1. Add/verify graders for run_success_rate and proposal_quality metrics referenced in cicoach/daily-security-red-team hypotheses so guardrails stop reporting guardrail_unsupported.
2. Backfill token/verbosity metric observations (token_count_per_run, output_length_chars) for output_format-style experiments (daily-code-metrics, daily-compiler-quality, deep-report, daily-issues-report) to unblock multi-variant comparisons once K≥3 support lands.
3. Increase sample-collection cadence or min_samples visibility for low-volume experiments still COLLECTING (blog-auditor, daily-architecture-diagram, dataflow-pr-discussion-dataset, weekly-blog-post-writer, dependabot-campaign) — several are under 10 runs/variant.

Decision-pipeline gaps for next PR:

• guardrail_unsupported appears for cicoach, daily-safe-output-optimizer, and daily-security-red-team — these mandatory guardrails have no backing metric observation; core analysis correctly holds at EXTEND rather than guessing.
• unsupported_multi_variant (15 experiments) is the single largest blocker to converting COLLECTING/READY signals into deterministic PROMOTE/REJECT decisions; this should be the top priority for the core decision engine.
• No interaction diagnostics were computed this run because no single workflow declares two or more simultaneously active experiments with sufficient assignment counts.

Descriptive window: latest branch state per workflow · Decision thresholds: from decision_policy
Run: 32824612728

Generated by 🧪 daily-experiment-report · copilot · auto · 99.8 AIC · ⌖ 23.7 AIC · ⊞ 17K · ◷

• expires on Aug 28, 2026, 12:17 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #55983.