[experiments] Daily Experiment Report — 2026-09-10

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-10

51 experiments analysed across 48 workflows. 22 are 🟢 READY (all variants past min_samples), but none has a deterministic PROMOTE or REJECT decision today — every experiment is blocked on a decision-pipeline gap (unsupported native metrics for guardrails/multi-variant tests, or outcome data not yet ingested). No interaction safety holds apply since no experiment reached PROMOTE.

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis	22
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

---

🟢 Ready for Analysis (2-variant experiments with tracking issues)

prompt_compression · agentperformanceanalyzer

Readiness: 🟢 READY
Variants: caveman vs verbose · Window: last 30 runs · Analysed: 112 runs total
min_samples: 20 per variant · Evidence: n/a, p_value=0.34724698438707013

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
caveman	61	██████████ 61/20 (100%)
verbose	51	██████████ 51/20 (100%)

Experiment : prompt_compression
Workflow   : agentperformanceanalyzer
Hypothesis : (not specified)
Window     : last 30 runs
min_samples: 20 per variant

Guardrails:
  (none declared)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_variant · awfailureinvestigator

Readiness: 🟢 READY
Variants: assertive vs clinical vs narrative · Window: last 30 runs · Analysed: 402 runs total
min_samples: 20 per variant · Evidence: n/a, p_value=0.9388512716644059

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
assertive	136	██████████ 136/20 (100%)
clinical	135	██████████ 135/20 (100%)
narrative	131	██████████ 131/20 (100%)

Experiment : tone_variant
Workflow   : awfailureinvestigator
Hypothesis : (not specified)
Window     : last 30 runs
min_samples: 20 per variant

Guardrails:
  (none declared)

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · cicoach

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 84 runs total
min_samples: 20 per variant · Evidence: mann_whitney, p_value=0

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	37	██████████ 37/20 (100%)
detailed	47	██████████ 47/20 (100%)

Experiment : prompt_style
Workflow   : cicoach
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs
min_samples: 20 per variant

Guardrails:
  run_success_rate >=0.85 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

reasoning_depth · dailyfact

Readiness: 🟢 READY
Variants: multi_candidate vs single_pass · Window: last 30 runs · Analysed: 85 runs total
min_samples: 30 per variant · Evidence: n/a, p_value=0.03709672547376961

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
multi_candidate	33	██████████ 33/30 (100%)
single_pass	52	██████████ 52/30 (100%)

Experiment : reasoning_depth
Workflow   : dailyfact
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).
Window     : last 30 runs
min_samples: 30 per variant

Guardrails:
  empty_output_rate <0.05 : UNSUPPORTED
  run_success_rate >=0.95 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

---

prompt_style · dailynews

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 100 runs total
min_samples: 20 per variant · Evidence: n/a, p_value=0.5560072787686822

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	53	██████████ 53/20 (100%)
detailed	47	██████████ 47/20 (100%)

Experiment : prompt_style
Workflow   : dailynews
Hypothesis : (not specified)
Window     : last 30 runs
min_samples: 20 per variant

Guardrails:
  (none declared)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

reasoning_depth · dailysecurityredteam

Readiness: 🟢 READY
Variants: iterative vs single_pass · Window: last 30 runs · Analysed: 121 runs total
min_samples: 30 per variant · Evidence: proportion_test, p_value=0.021846200583719545

H0: no change in finding quality. H1: iterative reduces false-positive rate by ≥20% at ≤30% token overhead

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
iterative	73	██████████ 73/30 (100%)
single_pass	48	██████████ 48/30 (100%)

Experiment : reasoning_depth
Workflow   : dailysecurityredteam
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by ≥20% at ≤30% token overhead
Window     : last 30 runs
min_samples: 30 per variant

Guardrails:
  run_success_rate >=0.9 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

prompt_style · issuearborist

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 68 runs total
min_samples: 20 per variant · Evidence: n/a, p_value=0.4736236793974227

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	37	██████████ 37/20 (100%)
detailed	31	██████████ 31/20 (100%)

Experiment : prompt_style
Workflow   : issuearborist
Hypothesis : (not specified)
Window     : last 30 runs
min_samples: 20 per variant

Guardrails:
  (none declared)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_style · typist

Readiness: 🟢 READY
Variants: conversational vs formal · Window: last 30 runs · Analysed: 77 runs total
min_samples: 20 per variant · Evidence: n/a, p_value=0.4306522830396994

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
conversational	42	██████████ 42/20 (100%)
formal	35	██████████ 35/20 (100%)

Experiment : tone_style
Workflow   : typist
Hypothesis : (not specified)
Window     : last 30 runs
min_samples: 20 per variant

Guardrails:
  (none declared)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

📊 Summary — All 51 Experiments

View Full Experiments Table

Experiment	Workflow	Runs	Readiness	Decision	Reason code
tone_variant	awfailureinvestigator	402	🟢 READY	⚪ INCONCLUSIVE	unsupported_multi_variant
caveman	smokecopilot	354	🟢 READY	🟡 EXTEND	insufficient_observations
subagent_model	smokecopilot	334	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smokegemini	262	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	208	🟢 READY	🟡 EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	179	🟢 READY	🟡 EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	179	🟢 READY	🟡 EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	152	🟢 READY	🟡 EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	152	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style	dailyastrostylelitemarkdownspellcheck	130	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style	dailycommunityattribution	128	🟢 READY	🟡 EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	121	🟢 READY	🟡 EXTEND	guardrail_unsupported
prompt_compression	agentperformanceanalyzer	112	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	112	🟢 READY	🟡 EXTEND	insufficient_observations
tool_verbosity	gpclean	109	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style	dailynews	100	🟢 READY	🟡 EXTEND	insufficient_observations
reasoning_depth	dailyfact	85	🟢 READY	🟡 EXTEND	guardrail_unsupported
prompt_style	cicoach	84	🟢 READY	🟡 EXTEND	guardrail_unsupported
tone_style	typist	77	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	76	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	dailyagentrxtraceoptimizer	73	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style	issuearborist	68	🟢 READY	🟡 EXTEND	insufficient_observations
output_format	deepreport	173	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
output_format	dailyissuesreport	128	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
output_format	dailycompilerquality	117	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	116	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailycavemanoptimizer	98	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	98	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	97	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
output_format	copilotagentanalysis	80	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
semgrep_output_format	dailysemgrepscan	77	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
audit_decomposition	auditworkflows	67	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
prompt_style	dependabotgochecker	45	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokecopilotsubagents	40	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
tone_variant	breakingchangechecker	39	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
sub_agent_strategy	architectureguardian	38	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
detail_level	dailyarchitecturediagram	16	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
caveman_mode	dataflowprdiscussiondataset	13	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
prefetch_strategy	weeklyblogpostwriter	13	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
summary_detail	dependabotcampaign	12	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
prompt_style	blogauditor	9	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
reasoning_depth	plan	7	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailycachestrategyanalyzer	4	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
timeout_setting	dailysubagentoptimizer	4	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	dailyfunctionnamer	3	🟡 COLLECTING	🟡 EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier	1	🟡 COLLECTING	🟡 EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	1	🟡 COLLECTING	🟡 EXTEND	insufficient_observations
remove_redundant_context_v1	prsouschef	1	🟡 COLLECTING	🟡 EXTEND	insufficient_observations
prompt_style_test	smokeproject	1	🟡 COLLECTING	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	1	🟡 COLLECTING	🟡 EXTEND	insufficient_observations
model_size	testqualitysentinel	1	🟡 COLLECTING	🟡 EXTEND	insufficient_observations

Descriptive window: last 30 runs per workflow (where applicable) · Decisions come from gh aw experiments analyze core logic, not recomputed here.
Reason code legend: insufficient_observations = readiness met but outcome-metric observations not yet correlated; unsupported_multi_variant = analysis type doesn't yet support 3+ variant comparison; insufficient_samples = below min_samples; guardrail_unsupported = a mandatory guardrail metric has no supported native observation source.

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Unblock guardrail_unsupported (cicoach, dailyfact, dailysecurityredteam) — wire run_success_rate and empty_output_rate guardrail metrics to a supported native observation source (e.g. workflow run conclusion + safe-output emptiness check) so these 3 READY 2-variant experiments can produce a real PROMOTE/REJECT decision instead of stalling indefinitely.
2. Extend unsupported_multi_variant support (14 experiments, e.g. awfailureinvestigator, dailycodemetrics, output_format family) — these are among the highest-run-count experiments (awfailureinvestigator has 402 runs) but the analysis layer only supports 2-variant comparisons; add a Kruskal-Wallis or omnibus + pairwise post-hoc path so 3+ variant experiments aren't permanently INCONCLUSIVE.
3. Correlate outcome metrics for insufficient_observations (25 experiments, largest bucket) — many READY experiments (smokecopilot family, dailyastrostylelitemarkdownspellcheck, gpclean, issuearborist, typist) have ample samples but the decision layer lacks per-run outcome-metric correlation; prioritize wiring token_count_per_run/duration observations for the highest-n workflows first (smokecopilot: 354 runs, smokegemini: 262 runs).

Top 3 eval actions

1. Add eval questions scoring guardrail-relevant outputs directly (success/failure classification, empty-output detection) for cicoach, dailyfact, dailysecurityredteam prompts so run_success_rate/empty_output_rate guardrails become graded rather than unsupported.
2. For output_format family experiments (deepreport, dailycodemetrics, dailycompilerquality, dailyissuesreport, copilotagentanalysis) — tighten eval rubrics to score ste variant outputs specifically, since ste consistently lags other variants in sample accumulation (e.g. copilotagentanalysis ste=3/30, dailysemgrepscan structured_sections=22/30), suggesting either lower assignment weight or higher failure/skip rate worth investigating via evals.
3. Add a grader question for sub_agent_strategy/sub_agent_decomposition experiments (dailyagentrxtraceoptimizer, agentpersonaexplorer, smokepi, smokeantigravity, smokegemini) comparing sub-agent vs single-agent output quality directly, to give the insufficient_observations reason code a path to real evidence.

Decision-pipeline gaps for next PR

• guardrail_unsupported (3 experiments) and insufficient_observations (25 experiments) together account for 28/51 (55%) of blocked decisions — both point to the same root gap: native per-run outcome-metric observation isn't yet wired into the decision core for most workflows.
• unsupported_multi_variant (14 experiments) is a normalized-analysis gap: the core decision layer needs a K≥3 variant comparison path (e.g. Kruskal-Wallis + Bonferroni-corrected pairwise) before these experiments (many with 80-400+ runs) can ever resolve.
• No interaction diagnostics were computed this run since no experiment reached PROMOTE; once the guardrail/observation gaps above are closed, factorial interaction cells (e.g. smokecopilot running both caveman and subagent_model simultaneously) should become a normalized core analysis signal rather than a reporting-layer safety hold.

Run: 34452056603

Generated by 🧪 daily-experiment-report · copilot · auto · 112.4 AIC · ⌖ 22.2 AIC · ⊞ 16.7K · ◷

• expires on Sep 13, 2026, 12:04 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #60223.