[experiments] Daily Experiment Report — 2026-09-06

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-06

48 experiments analysed across 48 workflows in github/gh-aw. 22 have reached minimum sample size (22 🟢 READY, 29 🟡 COLLECTING). No decision reached PROMOTE or REJECT today — all analyses returned EXTEND or INCONCLUSIVE, because per-run outcome/grader metrics are not yet flowing into the core analyzer, or the experiment currently has 3+ variants (unsupported for pairwise decisioning). No interaction safety holds applied (no workflow currently runs 2+ concurrent experiments).

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis	22
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

---

📋 Per-Experiment Detail (all 51 experiment analyses)

prompt_compression · agent-performance-analyzer 🟢 — caveman(58) vs verbose(50) · metric:effective_tokens · min_samples=20 · p=0.448
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

sub_agent_strategy · agent-persona-explorer 🟢 — batch(57) vs per_scenario(51) · metric:effective_tokens · min_samples=20 · p=0.571
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

sub_agent_strategy · architectureguardian 🟡 — single_agent(17) vs sub_agents(21) · min_samples=20 · p=0.524
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

audit_decomposition · audit-workflows 🟡 — phased_sub_agents(37) vs single_agent(26) · metric:run_success_rate · min_samples=264 · p=0.162
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

tone_variant · aw-failure-investigator 🟢 — assertive(131) vs clinical(132) vs narrative(123) · metric:output_length_chars · min_samples=20 · p=0.828
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

prompt_style · blog-auditor 🟡 — concise(3) vs detailed(5) · metric:effective_token_count · min_samples=20 · p=0.487
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

tone_variant · breaking-change-checker 🟡 — neutral(21) vs urgent(15) · metric:issue_engagement_rate · min_samples=20 · p=0.319
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

prompt_style · ci-coach 🟢 — concise(36) vs detailed(45) · metric:token_count_per_run · min_samples=20 · p=0.000
→ 🟡 EXTEND (guardrail_unsupported): mandatory guardrail "run_success_rate" is not backed by a supported metric observation

output_format · copilot-agent-analysis 🟡 — prose(34) vs ste(3) vs structured(43) · metric:ai_credits_used · min_samples=30 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

sub_agent_strategy · daily-agentrx-trace-optimizer 🟢 — single_agent(35) vs sub_agents(34) · metric:issue_evidence_completeness · min_samples=20 · p=0.871
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

detail_level · daily-architecture-diagram 🟡 — brief(9) vs comprehensive(6) · metric:run_duration_ms · min_samples=20 · p=0.445
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

prompt_style · daily-astrostylelite-markdown-spellcheck 🟢 — concise(65) vs detailed(61) · metric:effective_tokens · min_samples=20 · p=0.720
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

model_size · daily-cache-strategy-analyzer 🟡 — n/a · metric:ai_credits_total · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

model_size · daily-caveman-optimizer 🟡 — agent(1) vs claude-haiku-4.5(55) vs claude-sonnet-4.6(34) vs claude-sonnet-5(2) vs small-agent(2) · metric:ai_credits_total · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

output_format · daily-code-metrics 🟡 — executive_summary(49) vs full_detail(53) vs ste(10) · metric:discussion_engagement_score · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

prompt_style · daily-community-attribution 🟢 — concise(62) vs verbose(62) · min_samples=20 · p=1.000
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

output_format · daily-compiler-quality 🟡 — concise(48) vs detailed(54) vs ste(11) · metric:discussion_engagement_score · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

model_size · daily-doc-healer 🟡 — agent(2) vs claude-haiku-4.5(47) vs claude-sonnet-4.6(43) vs claude-sonnet-5(1) vs small-agent(1) · metric:ai_credits_total · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

model_size · daily-doc-updater 🟡 — agent(1) vs claude-haiku-4.5(42) vs claude-sonnet-4.6(47) vs claude-sonnet-5(2) vs small-agent(2) · metric:ai_credits_total · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

reasoning_depth · daily-fact 🟢 — multi_candidate(32) vs single_pass(50) · metric:discussion_reaction_count · min_samples=30 · p=0.044
→ 🟡 EXTEND (guardrail_unsupported): mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

model_size · dailyfunctionnamer 🟡 — n/a · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

output_format · daily-issues-report 🟡 — collapsible(53) vs inline(57) vs ste(14) · metric:discussion_engagement_score · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

prompt_style · daily-news 🟢 — concise(52) vs detailed(45) · metric:effective_token_count · min_samples=20 · p=0.484
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

remove_redundant_context_v1 · daily-rendering-scripts-verifier 🟡 — n/a · metric:"grader:execution-duration" · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

log_fetch_strategy · daily-safe-output-optimizer 🟡 — n/a · metric:run_duration_ms · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

reasoning_depth · daily-security-red-team 🟢 — iterative(70) vs single_pass(47) · metric:false_positive_rate · min_samples=30 · p=0.032
→ 🟡 EXTEND (guardrail_unsupported): mandatory guardrail "run_success_rate" is not backed by a supported metric observation

semgrep_output_format · daily-semgrep-scan 🟡 — bullet_list(28) vs prose(27) vs structured_sections(22) · metric:alert_creation_rate · min_samples=30 · p=0.023
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

timeout_setting · dailysubagentoptimizer 🟡 — default(2) vs relaxed(1) vs tight(1) · min_samples=20 · p=0.781
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

caveman_mode · dataflow-pr-discussion-dataset 🟡 — no(5) vs yes(8) · metric:input_token_count · min_samples=20 · p=0.410
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

output_format · deep-report 🟡 — annotated_brief(38) vs executive_brief(58) vs full_briefing(46) vs ste(15) · metric:token_count · min_samples=20 · p=0.000
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

summary_detail · dependabotcampaign 🟡 — brief(6) vs detailed(6) · min_samples=20 · p=1.000
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

prompt_style · dependabot-go-checker 🟡 — concise(15) vs detailed(15) vs step_by_step(13) · metric:issue_categorization_accuracy · min_samples=20 · p=0.904
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

tool_verbosity · gpclean 🟢 — full_bash(49) vs minimal_toolset(56) · metric:effective_token_count · min_samples=20 · p=0.502
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

prompt_style · issue-arborist 🟢 — concise(37) vs detailed(27) · metric:links_created · min_samples=20 · p=0.208
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

reasoning_depth · plan 🟡 — baseline(4) vs deep(2) vs shallow(1) · metric:plan_quality_score · min_samples=20 · p=0.369
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

remove_redundant_context_v1 · pr-sous-chef 🟡 — n/a · metric:"grader:execution-duration" · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

sub_agent_strategy · smokeantigravity 🟢 — single_agent(99) vs sub_agents(109) · min_samples=20 · p=0.495
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

caveman · smoke-copilot 🟢 — no(176) vs yes(176) · min_samples=20 · p=1.000
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

subagent_model · smoke-copilot 🟢 — large(166) vs small(166) · min_samples=20 · p=1.000
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

caveman · smoke-copilot-aoai-apikey 🟢 — no(88) vs yes(89) · min_samples=20 · p=0.898
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

subagent_model · smoke-copilot-aoai-apikey 🟢 — large(89) vs small(88) · min_samples=20 · p=0.898
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

caveman · smoke-copilot-aoai-entra 🟢 — no(75) vs yes(75) · min_samples=20 · p=1.000
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

subagent_model · smoke-copilot-aoai-entra 🟢 — large(75) vs small(75) · min_samples=20 · p=1.000
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

sub_agent_strategy · smoke-copilot-sub-agents 🟡 — delegated_sequential(14) vs inline_strict(9) vs single_agent_control(15) · metric:pass_rate · min_samples=30 · p=0.400
→ ⚪ INCONCLUSIVE (unsupported_multi_variant): automatic decisions currently require exactly two variants

sub_agent_strategy · smoke-gemini 🟢 — single_agent(121) vs sub_agents(139) · metric:effective_tokens · min_samples=20 · p=0.263
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

sub_agent_decomposition · smokepi 🟢 — parallel_sub_agents(42) vs single_agent(34) · min_samples=20 · p=0.362
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

prompt_style_test · smoke-project 🟡 — n/a · metric:run_success_rate · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

sub_agent_strategy · smoke-temporary-id 🟡 — n/a · metric:effective_token_count · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

model_size · test-quality-sentinel 🟡 — n/a · metric:review_usefulness_acceptance_rate · min_samples=20 · p=0.000
→ 🟡 EXTEND (insufficient_observations): at least two variants are required for a decision

tone_style · typist 🟢 — conversational(40) vs formal(34) · metric:discussion_engagement_score · min_samples=20 · p=0.493
→ 🟡 EXTEND (insufficient_observations): primary metric observations are not available for statistical comparison

prefetch_strategy · weekly-blog-post-writer 🟡 — eager(5) vs lazy(7) · metric:agent_turn_count · min_samples=20 · p=0.571
→ 🟡 EXTEND (insufficient_samples): one or more variants have fewer usable observations than min_samples

---

📊 Summary

View Full Experiments Table

Experiment	Workflow	Variants	Reason code	Core decision
prompt_compression	agent-performance-analyzer	caveman, verbose	insufficient_observations	EXTEND
sub_agent_strategy	agent-persona-explorer	batch, per_scenario	insufficient_observations	EXTEND
sub_agent_strategy	architectureguardian	single_agent, sub_agents	insufficient_samples	EXTEND
audit_decomposition	audit-workflows	phased_sub_agents, single_agent	insufficient_samples	EXTEND
tone_variant	aw-failure-investigator	assertive, clinical, narrative	unsupported_multi_variant	INCONCLUSIVE
prompt_style	blog-auditor	concise, detailed	insufficient_samples	EXTEND
tone_variant	breaking-change-checker	neutral, urgent	insufficient_samples	EXTEND
prompt_style	ci-coach	concise, detailed	guardrail_unsupported	EXTEND
output_format	copilot-agent-analysis	prose, ste, structured	unsupported_multi_variant	INCONCLUSIVE
sub_agent_strategy	daily-agentrx-trace-optimizer	single_agent, sub_agents	insufficient_observations	EXTEND
detail_level	daily-architecture-diagram	brief, comprehensive	insufficient_samples	EXTEND
prompt_style	daily-astrostylelite-markdown-spellcheck	concise, detailed	insufficient_observations	EXTEND
model_size	daily-cache-strategy-analyzer	n/a	insufficient_observations	EXTEND
model_size	daily-caveman-optimizer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	unsupported_multi_variant	INCONCLUSIVE
output_format	daily-code-metrics	executive_summary, full_detail, ste	unsupported_multi_variant	INCONCLUSIVE
prompt_style	daily-community-attribution	concise, verbose	insufficient_observations	EXTEND
output_format	daily-compiler-quality	concise, detailed, ste	unsupported_multi_variant	INCONCLUSIVE
model_size	daily-doc-healer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	unsupported_multi_variant	INCONCLUSIVE
model_size	daily-doc-updater	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	unsupported_multi_variant	INCONCLUSIVE
reasoning_depth	daily-fact	multi_candidate, single_pass	guardrail_unsupported	EXTEND
model_size	dailyfunctionnamer	n/a	insufficient_observations	EXTEND
output_format	daily-issues-report	collapsible, inline, ste	unsupported_multi_variant	INCONCLUSIVE
prompt_style	daily-news	concise, detailed	insufficient_observations	EXTEND
remove_redundant_context_v1	daily-rendering-scripts-verifier	n/a	insufficient_observations	EXTEND
log_fetch_strategy	daily-safe-output-optimizer	n/a	insufficient_observations	EXTEND
reasoning_depth	daily-security-red-team	iterative, single_pass	guardrail_unsupported	EXTEND
semgrep_output_format	daily-semgrep-scan	bullet_list, prose, structured_sections	unsupported_multi_variant	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer	default, relaxed, tight	unsupported_multi_variant	INCONCLUSIVE
caveman_mode	dataflow-pr-discussion-dataset	no, yes	insufficient_samples	EXTEND
output_format	deep-report	annotated_brief, executive_brief, full_briefing, ste	unsupported_multi_variant	INCONCLUSIVE
summary_detail	dependabotcampaign	brief, detailed	insufficient_samples	EXTEND
prompt_style	dependabot-go-checker	concise, detailed, step_by_step	unsupported_multi_variant	INCONCLUSIVE
tool_verbosity	gpclean	full_bash, minimal_toolset	insufficient_observations	EXTEND
prompt_style	issue-arborist	concise, detailed	insufficient_observations	EXTEND
reasoning_depth	plan	baseline, deep, shallow	unsupported_multi_variant	INCONCLUSIVE
remove_redundant_context_v1	pr-sous-chef	n/a	insufficient_observations	EXTEND
sub_agent_strategy	smokeantigravity	single_agent, sub_agents	insufficient_observations	EXTEND
caveman	smoke-copilot	no, yes	insufficient_observations	EXTEND
subagent_model	smoke-copilot	large, small	insufficient_observations	EXTEND
caveman	smoke-copilot-aoai-apikey	no, yes	insufficient_observations	EXTEND
subagent_model	smoke-copilot-aoai-apikey	large, small	insufficient_observations	EXTEND
caveman	smoke-copilot-aoai-entra	no, yes	insufficient_observations	EXTEND
subagent_model	smoke-copilot-aoai-entra	large, small	insufficient_observations	EXTEND
sub_agent_strategy	smoke-copilot-sub-agents	delegated_sequential, inline_strict, single_agent_control	unsupported_multi_variant	INCONCLUSIVE
sub_agent_strategy	smoke-gemini	single_agent, sub_agents	insufficient_observations	EXTEND
sub_agent_decomposition	smokepi	parallel_sub_agents, single_agent	insufficient_observations	EXTEND
prompt_style_test	smoke-project	n/a	insufficient_observations	EXTEND
sub_agent_strategy	smoke-temporary-id	n/a	insufficient_observations	EXTEND
model_size	test-quality-sentinel	n/a	insufficient_observations	EXTEND
tone_style	typist	conversational, formal	insufficient_observations	EXTEND
prefetch_strategy	weekly-blog-post-writer	eager, lazy	insufficient_samples	EXTEND

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. unsupported_multi_variant blocks 14 experiments (awfailureinvestigator, copilotagentanalysis, dailycavemanoptimizer, dailycodemetrics, dailycompilerquality, dailydochealer, dailydocupdater, dailyissuesreport, dailysemgrepscan, dailysubagentoptimizer, dependabotgochecker, deepreport, plan, smokecopilotsubagents) — reduce to 2-variant A/B or extend the core analyzer to support >2-variant decisioning.
2. The 8 READY experiments (agentperformanceanalyzer, awfailureinvestigator, cicoach, dailyfact, dailynews, dailysecurityredteam, issuearborist, typist) have sufficient samples but decisions are blocked on insufficient_observations/guardrail_unsupported — wire up native metric/grader observation ingestion for their primary metrics so the next run can reach PROMOTE/REJECT.
3. Remaining COLLECTING + insufficient_samples experiments (architectureguardian, auditworkflows, blogauditor, breakingchangechecker, dailyarchitecturediagram, dataflowprdiscussiondataset, dependabotcampaign, weeklyblogpostwriter) need no action — continue running until min_samples reached.

Top 3 eval actions

1. Add grader/eval coverage to populate primary metrics for the 8 READY workflows (esp. cicoach's token_count_per_run and dailysecurityredteam's false_positive_rate) — closest to an actionable decision.
2. Add eval coverage for output-quality metrics used by multi-variant output_format experiments (dailycodemetrics, dailycompilerquality, dailyissuesreport, deepreport) ahead of multi-variant core-analysis support.
3. Verify discussion_engagement_score graders (dailycodemetrics, dailycompilerquality, dailyissuesreport, typist) are actually emitting per-run values — several show guardrail_unsupported/insufficient_observations despite READY sample sizes.

Decision-pipeline gaps for next PR

• insufficient_observations (26) + guardrail_unsupported (3) experiments show the core analyzer isn't receiving per-run primary-metric observations for most workflows — largest blocker to PROMOTE/REJECT.
• unsupported_multi_variant (14) shows the core analyzer only supports 2-variant pairwise decisions; multi-variant (ANOVA/chi-square) support is needed as a normalized core signal.
• No interaction diagnostics were computed this run — no workflow currently declares 2+ concurrent experiments; exercise the factorial-interaction helper once one exists.

Descriptive window: cumulative branch state per workflow (git-based state) · Decision thresholds: from decision_policy in gh aw experiments analyze
Run: 34020232283

Generated by 🧪 daily-experiment-report · copilot · auto · 151.6 AIC · ⌖ 17.2 AIC · ⊞ 16.5K · ◷

• expires on Sep 9, 2026, 12:07 AM UTC-08:00