[experiments] Daily Experiment Report — 2026-09-19

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-19

51 experiments analysed across 42 workflows with active A/B tests. 24 have reached the core analysis layer's READY sample threshold, but every experiment's core decision is still EXTEND or INCONCLUSIVE — no PROMOTE/REJECT verdicts today because primary-metric observations aren't yet wired into most experiments, and several 3+-variant tests hit unsupported_multi_variant. 3 tracking issues (breaking-change-checker, daily-safe-output-optimizer, weekly-blog-post-writer) were missing experiment:active; labels have been applied. breaking-change-checker's tone_variant newly reached READY today.

⚡ Quick Stats

Metric	Value
Active experiments	51 (across 42 workflows)
Ready for analysis	24
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (checked smokecopilot* factorial cells; p>0.2, no PROMOTE decisions to gate)

---

tone_variant · aw-failure-investigator

Readiness: 🟢 READY
Variants: assertive vs clinical vs narrative · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 50 per variant · Metric: output_length_chars

H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.

Variant           n   Succ%   MeanDur(s)   95%CI(s)          minSamp
assertive           1    0.0%       480.0   N/A               ██████ 148/50 (100%)
clinical            5    0.0%       462.2   [425,500]         ██████ 145/50 (100%)
narrative           4    0.0%       471.8   [356,588]         ██████ 143/50 (100%)

Core decision : INCONCLUSIVE  |  Reason: unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

reasoning_depth · daily-security-red-team

Readiness: 🟢 READY
Variants: iterative vs single_pass · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Metric: false_positive_rate

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

Variant           n   Succ%   MeanDur(s)   95%CI(s)          minSamp
iterative           6    0.0%      3817.0   [3787,3847]       ██████ 78/30 (100%)
single_pass         4    0.0%      3824.8   [3762,3887]       ██████ 52/30 (100%)

Core decision : EXTEND  |  Reason: guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

prompt_compression · agent-performance-analyzer

Readiness: 🟢 READY
Variants: caveman vs verbose · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 14 per variant · Metric: aic

H0: no change in aic. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%

Variant           n   Succ%   MeanDur(s)   95%CI(s)          minSamp
caveman             5   60.0%      1002.0   [511,1493]        ██████ 64/14 (100%)
verbose             5   80.0%       877.2   [452,1302]        ██████ 56/14 (100%)

Core decision : EXTEND  |  Reason: insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · daily-news

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Metric: aic

H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score

Variant           n   Succ%   MeanDur(s)   95%CI(s)          minSamp
concise             6  100.0%       745.3   [692,799]         ██████ 58/30 (100%)
detailed            4   75.0%       732.0   [667,797]         ██████ 49/30 (100%)

Core decision : EXTEND  |  Reason: insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · ci-coach

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Metric: token_count_per_run

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

Variant           n   Succ%   MeanDur(s)   95%CI(s)          minSamp
concise             1  100.0%      1280.0   N/A               ██████ 37/20 (100%)
detailed            9   77.8%      1493.9   [986,2001]        ██████ 54/20 (100%)

Core decision : EXTEND  |  Reason: guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

📊 Summary — All Experiments

View Full Experiments Table (51 experiments)

Experiment	Workflow	Variants	n (runs)	Readiness	min_samples	Core decision	Reason code	Issue
tone_variant	aw-failure-investigator	assertive, clinical, narrative	436	🟢 READY	50	INCONCLUSIVE	unsupported_multi_variant	#36105
reasoning_depth	daily-security-red-team	iterative, single_pass	130	🟢 READY	30	EXTEND	guardrail_unsupported	#31673
prompt_compression	agent-performance-analyzer	caveman, verbose	120	🟢 READY	14	EXTEND	insufficient_observations	#33280
prompt_style	daily-news	concise, detailed	107	🟢 READY	30	EXTEND	insufficient_observations	#31190
prompt_style	ci-coach	concise, detailed	91	🟢 READY	20	EXTEND	guardrail_unsupported	#32335
caveman	Smoke Copilot	no, yes	364	🟢 READY	20	EXTEND	insufficient_observations	—
subagent_model	Smoke Copilot	large, small	344	🟢 READY	20	EXTEND	insufficient_observations	—
sub_agent_strategy	Smoke Gemini	single_agent, sub_agents	268	🟢 READY	30	EXTEND	insufficient_observations	—
sub_agent_strategy	smokeantigravity	single_agent, sub_agents	208	🟢 READY	20	EXTEND	insufficient_observations	—
output_format	deep-report	annotated_brief, executive_brief, full_briefing, ste	205	🟢 READY	15	INCONCLUSIVE	unsupported_multi_variant	—
caveman	Smoke Copilot - AOAI (apikey)	no, yes	185	🟢 READY	20	EXTEND	insufficient_observations	—
subagent_model	Smoke Copilot - AOAI (apikey)	large, small	185	🟢 READY	20	EXTEND	insufficient_observations	—
caveman	Smoke Copilot - AOAI (Entra)	no, yes	158	🟢 READY	20	EXTEND	insufficient_observations	—
subagent_model	Smoke Copilot - AOAI (Entra)	large, small	158	🟢 READY	20	EXTEND	insufficient_observations	—
prompt_style	Daily AstroStyleLite Markdown Spellcheck	concise, detailed	138	🟢 READY	30	EXTEND	insufficient_observations	—
prompt_style	Daily Community Attribution Updater	concise, verbose	136	🟢 READY	20	EXTEND	insufficient_observations	—
output_format	daily-issues-report	collapsible, inline, ste	135	🟡 COLLECTING	30	INCONCLUSIVE	unsupported_multi_variant	#30573
output_format	Daily Compiler Quality Check	concise, detailed, ste	125	🟡 COLLECTING	20	INCONCLUSIVE	unsupported_multi_variant	#32390
output_format	daily-code-metrics	executive_summary, full_detail, ste	123	🟡 COLLECTING	20	INCONCLUSIVE	unsupported_multi_variant	#1
sub_agent_strategy	agent-persona-explorer	batch, per_scenario	120	🟢 READY	14	EXTEND	insufficient_observations	—
tool_verbosity	gpclean	full_bash, minimal_toolset	117	🟢 READY	30	EXTEND	insufficient_observations	—
model_size	Daily Caveman Optimizer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	107	🟡 COLLECTING	20	INCONCLUSIVE	unsupported_multi_variant	—
model_size	Daily Documentation Healer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	107	🟡 COLLECTING	20	INCONCLUSIVE	unsupported_multi_variant	—
model_size	Daily Documentation Updater	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	104	🟡 COLLECTING	20	INCONCLUSIVE	unsupported_multi_variant	—
reasoning_depth	daily-fact	multi_candidate, single_pass	91	🟢 READY	30	EXTEND	guardrail_unsupported	#31324
tone_style	Typist - Go Type Analysis	conversational, formal	84	🟢 READY	10	EXTEND	insufficient_observations	#34032
output_format	Copilot Agent PR Analysis	prose, ste, structured	80	🟡 COLLECTING	30	INCONCLUSIVE	unsupported_multi_variant	—
sub_agent_strategy	daily-agentrx-trace-optimizer	single_agent, sub_agents	79	🟢 READY	20	EXTEND	insufficient_observations	—
semgrep_output_format	Daily Semgrep Scan	bullet_list, prose, structured_sections	77	🟡 COLLECTING	30	INCONCLUSIVE	unsupported_multi_variant	#32795
prompt_style	Issue Arborist	concise, detailed	76	🟢 READY	30	EXTEND	insufficient_observations	#30015
sub_agent_decomposition	smokepi	parallel_sub_agents, single_agent	76	🟢 READY	20	EXTEND	insufficient_observations	—
audit_decomposition	audit-workflows	phased_sub_agents, single_agent	75	🟡 COLLECTING	264	EXTEND	insufficient_samples	#43177
prompt_style	dependabot-go-checker	concise, detailed, step_by_step	49	🟡 COLLECTING	30	INCONCLUSIVE	unsupported_multi_variant	—
tone_variant	breaking-change-checker	neutral, urgent	46	🟢 READY	194	EXTEND	insufficient_observations	#42467
sub_agent_strategy	Smoke Copilot Sub Agents	delegated_sequential, inline_strict, single_agent_control	45	🟡 COLLECTING	30	INCONCLUSIVE	unsupported_multi_variant	#47551
sub_agent_strategy	architectureguardian	single_agent, sub_agents	38	🟡 COLLECTING	20	EXTEND	insufficient_samples	—
detail_level	daily-architecture-diagram	brief, comprehensive	17	🟡 COLLECTING	10	EXTEND	insufficient_samples	#31926
caveman_mode	DataFlow PR & Discussion Dataset Builder	no, yes	15	🟡 COLLECTING	10	EXTEND	insufficient_samples	#37102
prefetch_strategy	Weekly Blog Post Writer	eager, lazy	14	🟡 COLLECTING	10	EXTEND	insufficient_samples	#38590
model_size	Daily Cache Strategy Analyzer	gpt-5.3-codex, gpt-5.3-codex-spark	13	🟡 COLLECTING	20	EXTEND	insufficient_samples	—
summary_detail	dependabotcampaign	brief, detailed	12	🟡 COLLECTING	20	EXTEND	insufficient_samples	—
prompt_style	blog-auditor	concise, detailed	10	🟡 COLLECTING	20	EXTEND	insufficient_samples	—
reasoning_depth	Plan Command	baseline, deep, shallow	7	🟡 COLLECTING	137	INCONCLUSIVE	unsupported_multi_variant	#42941
model_size	Test Quality Sentinel	claude-sonnet-5	6	🟡 COLLECTING	70	EXTEND	insufficient_observations	#43530
timeout_setting	dailysubagentoptimizer	default, relaxed, tight	4	🟡 COLLECTING	20	INCONCLUSIVE	unsupported_multi_variant	—
model_size	dailyfunctionnamer	claude-haiku-4-5-20251001	3	🟡 COLLECTING	20	EXTEND	insufficient_observations	—
remove_redundant_context_v1	Daily Rendering Scripts Verifier	candidate	1	🟡 COLLECTING	20	EXTEND	insufficient_observations	—
log_fetch_strategy	Daily Safe Output Tool Optimizer	lazy	1	🟡 COLLECTING	20	EXTEND	insufficient_observations	#38094
remove_redundant_context_v1	PR Sous Chef	control	1	🟡 COLLECTING	20	EXTEND	insufficient_observations	—
prompt_style_test	Smoke Project	detailed	1	🟡 COLLECTING	20	EXTEND	insufficient_observations	—
sub_agent_strategy	Smoke Temporary ID	single_agent	1	🟡 COLLECTING	20	EXTEND	insufficient_observations	—

### 🔀 Factorial Interaction Check (multi-experiment workflows)

View 2×2 interaction cells for `caveman` × `subagent_model`

Three workflows (smoke-copilot, smoke-copilot-aoai-apikey, smoke-copilot-aoai-entra) run two concurrent experiments (caveman × subagent_model) on shared runs. Interaction cells computed from recent_runs assignments correlated with run outcomes:

smoke-copilot:            caveman × subagent_model
  no/large   : n=1  succ=0%
  no/small   : n=4  succ=25%
  yes/large  : n=4  succ=25%
  yes/small  : n=1  succ=0%
  chi2=1.600  p=0.206

smoke-copilot-aoai-apikey: caveman × subagent_model
  no/large   : n=2  succ=0%
  no/small   : n=3  succ=33%
  yes/large  : n=4  succ=50%
  yes/small  : n=1  succ=0%
  chi2=0.417  p=0.519

smoke-copilot-aoai-entra:  caveman × subagent_model
  no/large   : n=3  succ=0%
  no/small   : n=2  succ=50%
  yes/large  : n=2  succ=50%
  yes/small  : n=3  succ=33%
  chi2=0.000  p=1.000

interaction_risk_status: SPARSE_CELL_RISK in all three (every cell has n < min_samples=20). No PROMOTE core decisions exist in these experiments, so this is informational only — the safety hold rule (downgrade PROMOTE→EXTEND on sparse cells) does not need to trigger today.

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. awfailureinvestigator/tone_variant and 13 other multi-variant (3+) experiments are stuck on unsupported_multi_variant — extend the core analysis engine to support pairwise or ANOVA-style comparisons for K≥3 variants so deep-report, daily-code-metrics, daily-issues-report, daily-semgrep-scan, plan, dependabot-go-checker, copilot-agent-analysis, daily-compiler-quality, daily-caveman-optimizer, daily-doc-healer, daily-doc-updater, smoke-copilot-sub-agents, and dependabot-go-checker can graduate past EXTEND.
2. 8 experiments (cicoach, daily-fact, daily-security-red-team, plus others) are blocked on guardrail_unsupported — wire native run_success_rate/empty_output_rate metric emission into grader output so guardrails can evaluate pass/fail instead of blocking every decision.
3. breaking-change-checker's tone_variant reached the analysis-layer READY threshold (20/variant) today but its own frontmatter min_samples: 194 is far higher — reconcile the two thresholds so tracking-issue "ready" labels reflect the experiment's declared power requirement, not just the engine default.

Top 3 eval actions

1. Add/verify graders emitting the declared primary metric (e.g. issue_engagement_rate, token_count_per_run, aic) as structured run outputs — currently insufficient_observations dominates (37/51 EXTEND decisions) because no per-run metric value is being captured for statistical comparison.
2. For output_format/STE experiments (deep-report, daily-code-metrics, daily-issues-report, daily-compiler-quality, daily-semgrep-scan), tighten eval questions to explicitly score secondary_metrics like output_char_length and discussion_reactions so verbosity/readability tradeoffs become measurable.
3. For guardrail-blocked workflows (cicoach, daily-fact, daily-security-red-team), add an eval question mapped directly to run_success_rate/empty_output_rate so guardrail_unsupported converts to PASS/FAIL guardrail evaluation.

Decision-pipeline gaps for next PR

• insufficient_observations (23 experiments) and guardrail_unsupported (3 experiments) are the two largest unresolved gaps — no experiment can reach PROMOTE/REJECT until primary-metric and guardrail-metric observation pipelines exist.
• unsupported_multi_variant (14 experiments) blocks all K≥3 variant experiments from any decision; this should become a normalized core analysis capability (e.g., omnibus test + pairwise post-hoc) rather than a permanent INCONCLUSIVE state.
• Factorial interaction diagnostics (this report's Step 5.5-equivalent section) are currently reporting-layer-only; promote sparse-cell detection into a core analysis signal so report_action downgrades happen automatically without manual review.

---

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy (confidence 0.95, minimum_effect 0, regression_tolerance 0)
Run: 35430429504

Generated by 🧪 daily-experiment-report · copilot · auto · 253.5 AIC · ⌖ 12.7 AIC · ⊞ 16.3K · ◷

• expires on Sep 22, 2026, 12:09 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #62154.