[experiments] Daily Experiment Report — 2026-09-08

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-08

48 workflows in github/gh-aw declare an experiments: A/B test, totalling 51 individual experiment analyses. No experiment currently has decision-quality evidence for a deterministic PROMOTE or REJECT — 22 have reached READY sample thresholds but are blocked from a core decision by unsupported outcome-metric pipelines (guardrail/native-metric gaps) or by having more than two variants, and the remaining 29 are still COLLECTING runs.

⚡ Quick Stats

Metric	Value
Active experiments	51 (across 48 workflows)
Ready for analysis (readiness=READY)	22
Still collecting samples	29
Decisions with sufficient evidence (PROMOTE/REJECT)	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

📋 Why nothing is promoted or rejected yet

🟡 EXTEND — 37 experiments (all READY-but-blocked or still collecting)

• insufficient_observations (26) — variant assignment counts have reached min_samples, but the declared metric/secondary_metrics have no corresponding per-run observation data yet (metric collection pipeline gap, not a sample-size problem). Examples: agent-performance-analyzer/prompt_compression, daily-news/prompt_style, typist/tone_style, smoke-copilot/caveman, smoke-copilot/subagent_model, smoke-gemini/sub_agent_strategy.
• insufficient_samples (8) — still below min_samples per variant. Examples: architectureguardian/sub_agent_strategy, audit-workflows/audit_decomposition, blog-auditor/prompt_style, breaking-change-checker/tone_variant, daily-architecture-diagram/detail_level, dataflow-pr-discussion-dataset/caveman_mode, dependabotcampaign/summary_detail, weekly-blog-post-writer/prefetch_strategy.
• guardrail_unsupported (3) — samples are sufficient, but mandatory guardrails (run_success_rate, empty_output_rate) have status unsupported — no backing metric observation exists to evaluate the guardrail. Affects ci-coach/prompt_style, daily-fact/reasoning_depth, daily-security-red-team/reasoning_depth.

⚪ INCONCLUSIVE — 14 experiments (`unsupported_multi_variant`)

All 14 INCONCLUSIVE results share the same reason: automatic decisions currently require exactly two variants, and these experiments declare 3+ variants (e.g. output_format: structured/prose/table/ste). Affected workflows: aw-failure-investigator, copilot-agent-analysis, daily-caveman-optimizer, daily-code-metrics, daily-compiler-quality, daily-doc-healer, daily-doc-updater, daily-issues-report, daily-semgrep-scan, dailysubagentoptimizer, deep-report, dependabot-go-checker, plan, smoke-copilot-sub-agents.

This is a core-analysis capability gap, not evidence of no effect — treat these as unresolved, not rejected.

📈 Full Experiment Table (all 51 analyses)

Workflow                               Experiment             Ready       Decision      Reason                  
----------------------------------------------------------------------------------------------------------------
agent-performance-analyzer             prompt_compression     READY       EXTEND        insufficient_observations
agent-persona-explorer                 sub_agent_strategy     READY       EXTEND        insufficient_observations
architectureguardian                   sub_agent_strategy     COLLECTING  EXTEND        insufficient_samples    
audit-workflows                        audit_decomposition    COLLECTING  EXTEND        insufficient_samples    
blog-auditor                           prompt_style           COLLECTING  EXTEND        insufficient_samples    
breaking-change-checker                tone_variant           COLLECTING  EXTEND        insufficient_samples    
ci-coach                               prompt_style           READY       EXTEND        guardrail_unsupported   
daily-agentrx-trace-optimizer          sub_agent_strategy     READY       EXTEND        insufficient_observations
daily-architecture-diagram             detail_level           COLLECTING  EXTEND        insufficient_samples    
daily-astrostylelite-markdown-spellch  prompt_style           READY       EXTEND        insufficient_observations
daily-cache-strategy-analyzer          model_size             COLLECTING  EXTEND        insufficient_observations
daily-community-attribution            prompt_style           READY       EXTEND        insufficient_observations
daily-fact                             reasoning_depth        READY       EXTEND        guardrail_unsupported   
daily-news                             prompt_style           READY       EXTEND        insufficient_observations
daily-rendering-scripts-verifier       remove_redundant_cont  COLLECTING  EXTEND        insufficient_observations
daily-safe-output-optimizer            log_fetch_strategy     COLLECTING  EXTEND        insufficient_observations
daily-security-red-team                reasoning_depth        READY       EXTEND        guardrail_unsupported   
dailyfunctionnamer                     model_size             COLLECTING  EXTEND        insufficient_observations
dataflow-pr-discussion-dataset         caveman_mode           COLLECTING  EXTEND        insufficient_samples    
dependabotcampaign                     summary_detail         COLLECTING  EXTEND        insufficient_samples    
gpclean                                tool_verbosity         READY       EXTEND        insufficient_observations
issue-arborist                         prompt_style           READY       EXTEND        insufficient_observations
pr-sous-chef                           remove_redundant_cont  COLLECTING  EXTEND        insufficient_observations
smoke-copilot                          caveman                READY       EXTEND        insufficient_observations
smoke-copilot                          subagent_model         READY       EXTEND        insufficient_observations
smoke-copilot-aoai-apikey              caveman                READY       EXTEND        insufficient_observations
smoke-copilot-aoai-apikey              subagent_model         READY       EXTEND        insufficient_observations
smoke-copilot-aoai-entra               caveman                READY       EXTEND        insufficient_observations
smoke-copilot-aoai-entra               subagent_model         READY       EXTEND        insufficient_observations
smoke-gemini                           sub_agent_strategy     READY       EXTEND        insufficient_observations
smoke-project                          prompt_style_test      COLLECTING  EXTEND        insufficient_observations
smoke-temporary-id                     sub_agent_strategy     COLLECTING  EXTEND        insufficient_observations
smokeantigravity                       sub_agent_strategy     READY       EXTEND        insufficient_observations
smokepi                                sub_agent_decompositi  READY       EXTEND        insufficient_observations
test-quality-sentinel                  model_size             COLLECTING  EXTEND        insufficient_observations
typist                                 tone_style             READY       EXTEND        insufficient_observations
weekly-blog-post-writer                prefetch_strategy      COLLECTING  EXTEND        insufficient_samples    
aw-failure-investigator                tone_variant           READY       INCONCLUSIVE  unsupported_multi_variant
copilot-agent-analysis                 output_format          COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-caveman-optimizer                model_size             COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-code-metrics                     output_format          COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-compiler-quality                 output_format          COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-doc-healer                       model_size             COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-doc-updater                      model_size             COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-issues-report                    output_format          COLLECTING  INCONCLUSIVE  unsupported_multi_variant
daily-semgrep-scan                     semgrep_output_format  COLLECTING  INCONCLUSIVE  unsupported_multi_variant
dailysubagentoptimizer                 timeout_setting        COLLECTING  INCONCLUSIVE  unsupported_multi_variant
deep-report                            output_format          COLLECTING  INCONCLUSIVE  unsupported_multi_variant
dependabot-go-checker                  prompt_style           COLLECTING  INCONCLUSIVE  unsupported_multi_variant
plan                                   reasoning_depth        COLLECTING  INCONCLUSIVE  unsupported_multi_variant
smoke-copilot-sub-agents               sub_agent_strategy     COLLECTING  INCONCLUSIVE  unsupported_multi_variant

📊 Sample-Size Progress by Variant

agent-performance-analyzer / prompt_compression: caveman: ██████████ 59/20 (100%) | verbose: ██████████ 51/20 (100%)
agent-persona-explorer / sub_agent_strategy: batch: ██████████ 58/20 (100%) | per_scenario: ██████████ 52/20 (100%)
architectureguardian / sub_agent_strategy: single_agent: ████████░░ 17/20 (85%) | sub_agents: ██████████ 21/20 (100%)
audit-workflows / audit_decomposition: phased_sub_agents: █░░░░░░░░░ 38/264 (14%) | single_agent: █░░░░░░░░░ 27/264 (10%)
aw-failure-investigator / tone_variant: assertive: ██████████ 133/20 (100%) | clinical: ██████████ 134/20 (100%) | narrative: ██████████ 127/20 (100%)
blog-auditor / prompt_style: concise: ██░░░░░░░░ 3/20 (15%) | detailed: ██░░░░░░░░ 5/20 (25%)
breaking-change-checker / tone_variant: neutral: ██████████ 21/20 (100%) | urgent: ████████░░ 16/20 (80%)
ci-coach / prompt_style: concise: ██████████ 36/20 (100%) | detailed: ██████████ 46/20 (100%)
copilot-agent-analysis / output_format: prose: ██████████ 34/30 (100%) | ste: █░░░░░░░░░ 3/30 (10%) | structured: ██████████ 43/30 (100%)
daily-agentrx-trace-optimizer / sub_agent_strategy: single_agent: ██████████ 36/20 (100%) | sub_agents: ██████████ 35/20 (100%)
daily-architecture-diagram / detail_level: brief: █████░░░░░ 10/20 (50%) | comprehensive: ███░░░░░░░ 6/20 (30%)
daily-astrostylelite-markdown-spellcheck / prompt_style: concise: ██████████ 65/20 (100%) | detailed: ██████████ 63/20 (100%)
daily-cache-strategy-analyzer / model_size: 
daily-caveman-optimizer / model_size: agent: ░░░░░░░░░░ 1/20 (5%) | claude-haiku-4.5: ██████████ 56/20 (100%) | claude-sonnet-4.6: ██████████ 34/20 (100%) | claude-sonnet-5: ██░░░░░░░░ 3/20 (15%) | small-agent: █░░░░░░░░░ 2/20 (10%)
daily-code-metrics / output_format: executive_summary: ██████████ 49/20 (100%) | full_detail: ██████████ 55/20 (100%) | ste: █████░░░░░ 10/20 (50%)
daily-community-attribution / prompt_style: concise: ██████████ 63/20 (100%) | verbose: ██████████ 63/20 (100%)
daily-compiler-quality / output_format: concise: ██████████ 48/20 (100%) | detailed: ██████████ 55/20 (100%) | ste: ██████░░░░ 12/20 (60%)
daily-doc-healer / model_size: agent: █░░░░░░░░░ 2/20 (10%) | claude-haiku-4.5: ██████████ 48/20 (100%) | claude-sonnet-4.6: ██████████ 43/20 (100%) | claude-sonnet-5: █░░░░░░░░░ 2/20 (10%) | small-agent: ░░░░░░░░░░ 1/20 (5%)
daily-doc-updater / model_size: agent: ░░░░░░░░░░ 1/20 (5%) | claude-haiku-4.5: ██████████ 42/20 (100%) | claude-sonnet-4.6: ██████████ 47/20 (100%) | claude-sonnet-5: ██░░░░░░░░ 4/20 (20%) | small-agent: █░░░░░░░░░ 2/20 (10%)
daily-fact / reasoning_depth: multi_candidate: ██████████ 33/30 (100%) | single_pass: ██████████ 50/30 (100%)
daily-issues-report / output_format: collapsible: ██████████ 54/20 (100%) | inline: ██████████ 57/20 (100%) | ste: ████████░░ 15/20 (75%)
daily-news / prompt_style: concise: ██████████ 52/20 (100%) | detailed: ██████████ 46/20 (100%)
daily-rendering-scripts-verifier / remove_redundant_context_v1: 
daily-safe-output-optimizer / log_fetch_strategy: 
daily-security-red-team / reasoning_depth: iterative: ██████████ 71/30 (100%) | single_pass: ██████████ 48/30 (100%)
daily-semgrep-scan / semgrep_output_format: bullet_list: █████████░ 28/30 (93%) | prose: █████████░ 27/30 (90%) | structured_sections: ███████░░░ 22/30 (73%)
dailyfunctionnamer / model_size: 
dailysubagentoptimizer / timeout_setting: default: █░░░░░░░░░ 2/20 (10%) | relaxed: ░░░░░░░░░░ 1/20 (5%) | tight: ░░░░░░░░░░ 1/20 (5%)
dataflow-pr-discussion-dataset / caveman_mode: no: ██░░░░░░░░ 5/20 (25%) | yes: ████░░░░░░ 8/20 (40%)
deep-report / output_format: annotated_brief: ██████████ 42/20 (100%) | executive_brief: ██████████ 58/20 (100%) | full_briefing: ██████████ 50/20 (100%) | ste: ████████░░ 15/20 (75%)
dependabot-go-checker / prompt_style: concise: ████████░░ 15/20 (75%) | detailed: ████████░░ 16/20 (80%) | step_by_step: ██████░░░░ 13/20 (65%)
dependabotcampaign / summary_detail: brief: ███░░░░░░░ 6/20 (30%) | detailed: ███░░░░░░░ 6/20 (30%)
gpclean / tool_verbosity: full_bash: ██████████ 50/20 (100%) | minimal_toolset: ██████████ 57/20 (100%)
issue-arborist / prompt_style: concise: ██████████ 37/20 (100%) | detailed: ██████████ 29/20 (100%)
plan / reasoning_depth: baseline: ██░░░░░░░░ 4/20 (20%) | deep: █░░░░░░░░░ 2/20 (10%) | shallow: ░░░░░░░░░░ 1/20 (5%)
pr-sous-chef / remove_redundant_context_v1: 
smoke-copilot / caveman: no: ██████████ 176/20 (100%) | yes: ██████████ 177/20 (100%)
smoke-copilot / subagent_model: large: ██████████ 166/20 (100%) | small: ██████████ 167/20 (100%)
smoke-copilot-aoai-apikey / caveman: no: ██████████ 89/20 (100%) | yes: ██████████ 89/20 (100%)
smoke-copilot-aoai-apikey / subagent_model: large: ██████████ 89/20 (100%) | small: ██████████ 89/20 (100%)
smoke-copilot-aoai-entra / caveman: no: ██████████ 76/20 (100%) | yes: ██████████ 75/20 (100%)
smoke-copilot-aoai-entra / subagent_model: large: ██████████ 76/20 (100%) | small: ██████████ 75/20 (100%)
smoke-copilot-sub-agents / sub_agent_strategy: delegated_sequential: █████░░░░░ 14/30 (47%) | inline_strict: ███░░░░░░░ 10/30 (33%) | single_agent_control: █████░░░░░ 15/30 (50%)
smoke-gemini / sub_agent_strategy: single_agent: ██████████ 121/20 (100%) | sub_agents: ██████████ 140/20 (100%)
smoke-project / prompt_style_test: 
smoke-temporary-id / sub_agent_strategy: 
smokeantigravity / sub_agent_strategy: single_agent: ██████████ 99/20 (100%) | sub_agents: ██████████ 109/20 (100%)
smokepi / sub_agent_decomposition: parallel_sub_agents: ██████████ 42/20 (100%) | single_agent: ██████████ 34/20 (100%)
test-quality-sentinel / model_size: 
typist / tone_style: conversational: ██████████ 41/20 (100%) | formal: ██████████ 34/20 (100%)
weekly-blog-post-writer / prefetch_strategy: eager: ██░░░░░░░░ 5/20 (25%) | lazy: ████░░░░░░ 8/20 (40%)

🧭 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Fix the outcome-metric pipeline for insufficient_observations (26 experiments) — the largest single blocker. min_samples is satisfied but declared metrics (e.g. effective_tokens, discussion_engagement_score, token_count_per_run) aren't being recorded per-run. Prioritize daily-news, typist, smoke-copilot family, and smoke-gemini — all READY and high-traffic.
2. Add native run_success_rate / empty_output_rate guardrail observation support to unblock ci-coach, daily-fact, daily-security-red-team (3 guardrail_unsupported experiments, all READY).
3. Extend core decision logic to K≥3 variants (Bonferroni-style multi-comparison) to resolve the 14 unsupported_multi_variant INCONCLUSIVE results, most of which are testing output_format (structured/prose/table/ste) across content-heavy workflows.

Top 3 eval actions

1. Add/verify graders that emit empty_output_rate and run_success_rate as first-class observations so guardrail evaluation stops returning unsupported.
2. For insufficient_observations workflows using token/verbosity metrics (effective_tokens, token_count_per_run, ai_credits_total), confirm the grader step actually persists a per-run metric value rather than only a step summary.
3. Tighten output_format eval criteria (readability/verbosity scoring) ahead of enabling multi-variant decisions, since those experiments will become analyzable once (3) above lands.

Decision-pipeline gaps for next PR

• insufficient_observations and guardrail_unsupported together represent 29 of 37 EXTEND decisions — the dominant unresolved gap is metric/guardrail observation plumbing, not sample collection.
• Multi-variant (K≥3) decision support is the single highest-leverage core-analysis feature gap, affecting 14 experiments across content-format and reasoning-depth tests.
• No interaction diagnostics were computed this run since no workflow has two concurrent active experiments; revisit once experiments graduate from EXTEND/INCONCLUSIVE and combinatorial testing resumes.

Descriptive window: gh aw experiments analyze per-workflow variant/sample state · Decision thresholds: from each experiment's decision_policy
Run: 34201423180

Generated by 🧪 daily-experiment-report · copilot · auto · 68.3 AIC · ⌖ 16.4 AIC · ⊞ 16.5K · ◷

• expires on Sep 11, 2026, 12:01 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #59668.