[experiments] Daily Experiment Report — 2026-08-23

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-08-23

44 workflows in github/gh-aw currently run 47 active A/B experiments via experiments: frontmatter. 23 experiments have all variants at/above min_samples (ready for outcome analysis); 24 are still collecting data. No workflow declares a tracking issue: field, so this run posts findings only to this discussion (no issue comments/labels were applicable — see note in Step 8/9 below).

Important caveat: the p_value/chi_square figures below are the variant-assignment balance test (is traffic split as configured?), not a primary-metric significance test. None of these 47 experiments currently expose per-run outcome metrics (token_count, success_rate, etc.) in the git-branch state.json/state.jsonl used by gh aw experiments analyze — all declared guardrail_metrics report status: "unsupported". Full PROMOTE/ABANDON decisions require per-run outcome extraction (see Step 10 below) that is not yet wired up.

⚡ Quick Stats

Metric	Value
Active workflows with experiments	44
Active experiments	47
Ready for outcome analysis (min_samples reached, all variants)	23
Still collecting (EXTEND)	24
Assignment-balance test flagged imbalanced (p < 0.05)	14 (mostly multi-variant model/format experiments with skewed rollout, e.g. legacy variant labels retained from earlier experiment iterations)
Tracking issues configured	0

---

🟢 Ready for Analysis (23 experiments, all variants ≥ min_samples)

📈 View Full Table

Workflow                              Experiment              n     Variants  Balance p   Status
testqualitysentinel                   model_size              1092  2         0.1599      balanced
smokecopilot                          caveman                 332   2         1.0000      balanced
awfailureinvestigator                 tone_variant            331   3         0.4444      balanced
smokecopilot                          subagent_model          312   2         1.0000      balanced
smokegemini                           sub_agent_strategy      249   2         0.1407      balanced
smokeantigravity                      sub_agent_strategy      208   2         0.4952      balanced
smokecopilotaoaiapikey                caveman                 166   2         1.0000      balanced
smokecopilotaoaiapikey                subagent_model          166   2         1.0000      balanced
smokecopilotaoaientra                 caveman                 139   2         0.8926      balanced
smokecopilotaoaientra                 subagent_model          139   2         0.8926      balanced
dailyastrostylelitemarkdownspellcheck prompt_style            112   2         0.5780      balanced
dailycommunityattribution             prompt_style            110   2         1.0000      balanced
dailysecurityredteam                  reasoning_depth         103   2         0.0222 *    imbalanced
agentperformanceanalyzer              prompt_compression      94    2         0.6824      balanced
agentpersonaexplorer                  sub_agent_strategy      94    2         0.3031      balanced
gpclean                                tool_verbosity          91    2         0.1691      balanced
dailynews                             prompt_style            87    2         0.4594      balanced
smokepi                               sub_agent_decomposition 76    2         0.3619      balanced
cicoach                               prompt_style            71    2        <0.0001 *   imbalanced
typist                                tone_style              64    2         0.3187      balanced
dailyagentrxtraceoptimizer            sub_agent_strategy      59    2         0.6972      balanced
dailysafeoutputoptimizer              log_fetch_strategy      50    2        <0.0001 *   imbalanced
issuearborist                         prompt_style            48    2         0.3907      balanced

Significance: * p<0.05 (assignment balance test, not outcome significance)

Recommendation for this group: EXTEND (instrument outcome metrics before PROMOTE/ABANDON). Sample sizes are healthy, but without per-run outcome data (token cost, success rate, reviewer/reaction signals) none of these can be safely promoted or abandoned yet — recommend adding outcome capture to pick_experiment/finalize steps as the next iteration (see Step 10).

---

🟡 Still Collecting (24 experiments, at least one variant below min_samples)

📈 View Full Table

Workflow                          Experiment              n     Variants  Balance p    Status
dailyissuesreport                 output_format           110   3        <0.0001 *    imbalanced (ste variant under-sampled: n=10)
deepreport                        output_format           101   4        <0.0001 *    imbalanced (ste variant under-sampled: n=5)
dailycompilerquality              output_format           99    3        <0.0001 *    imbalanced (ste variant under-sampled: n=4)
dailycodemetrics                  output_format           98    3        <0.0001 *    imbalanced (ste variant under-sampled: n=5)
dailycavemanoptimizer             model_size              80    4        <0.0001 *    imbalanced (legacy agent/small-agent labels, n=1–2)
dailydochealer                    model_size              80    4        <0.0001 *    imbalanced (legacy agent/small-agent labels, n=1–2)
dailydocupdater                   model_size              80    4        <0.0001 *    imbalanced (legacy agent/small-agent labels, n=1–2)
dailycachestrategyanalyzer        model_size              78    6        <0.0001 *    imbalanced (5 low-n legacy variant labels)
dailysemgrepscan                  semgrep_output_format   77    3         0.0232 *    imbalanced
copilotagentanalysis              output_format           75    3        <0.0001 *    imbalanced (ste variant under-sampled: n=1)
dailyfact                         reasoning_depth         72    2         0.0562      near-balanced
auditworkflows                    audit_decomposition     49    2         0.1117      balanced
architectureguardian              sub_agent_strategy      38    2         0.5239      balanced
dependabotgochecker                prompt_style           37    3         0.8921      balanced
breakingchangechecker             tone_variant            30    2         0.4719      balanced
smokecopilotsubagents             sub_agent_strategy      30    3         0.4595      balanced
dailyarchitecturediagram          detail_level            13    2         0.7727      balanced
dependabotcampaign                summary_detail          12    2         1.0000      balanced
dataflowprdiscussiondataset       caveman_mode            11    2         0.3691      balanced
weeklyblogpostwriter              prefetch_strategy       10    2         0.5346      balanced
plan                               reasoning_depth        7     3         0.3694      balanced
blogauditor                       prompt_style            6     2         1.0000      balanced
dailysubagentoptimizer            timeout_setting         4     3         0.7814      balanced
dailyfunctionnamer                model_size              3     0        <0.0001      insufficient data

Recommendation: EXTEND for all 24. The 8 output_format/model_size experiments flagged imbalanced above mostly carry legacy variant labels (agent, small-agent, or early ste runs) from earlier iterations of the same experiment key — these should be pruned from the variant set or the branch state reset so future balance tests reflect only the current variant lineup.

---

📊 Summary Table (all 47 experiments)

View Full Experiments Table (sorted by sample size)

See the two tables above — 23 rows in "Ready for Analysis", 24 rows in "Still Collecting".

Analysis window: full experiments/* branch state (git for-each-ref) · Significance threshold: p < 0.05 (two-tailed, assignment-balance test only — see caveat above)
Run: 32626952068

---

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Instrument outcome-metric capture (token_count, success_rate, run_duration_ms) into each experiment's finalize step so gh aw experiments analyze guardrails stop reporting status: "unsupported" — currently blocking all 47 experiments from PROMOTE/ABANDON decisions.
2. Prune legacy/renamed variant labels from model_size and output_format experiments (dailycavemanoptimizer, dailydochealer, dailydocupdater, dailycachestrategyanalyzer, copilotagentanalysis, dailycodemetrics, dailycompilerquality, dailyissuesreport, deepreport) — stale agent/small-agent/early ste counts are skewing the balance test and diluting min_samples progress for the live variants.
3. Add issue: tracking fields to the highest-value, near-ready experiments (dailysecurityredteam, cicoach, dailysafeoutputoptimizer, testqualitysentinel) so future runs get automatic ready/significance/guardrail notifications per Step 8.

Top 3 eval actions

1. For output_format experiments comparing ste variants, tighten eval questions to explicitly score "clarity vs. information loss" — ste is consistently the lowest-n variant across 5 workflows, suggesting either low rollout weight or eval friction discouraging its selection.
2. For model_size/sub_agent_strategy experiments, add an eval question scoring "task completion equivalence" between small and large models to give a quality floor before promoting cost-saving variants.
3. Map low-scoring eval questions for prompt_style (concise vs. detailed) workflows to the token_count_per_run metric already declared in ci-coach.md, so eval quality and cost savings can be jointly reported.

Grader-ready hooks

• Decision-path hook: extend the per-run outcome record (Step 2 schema: run_id, experiment, variant, conclusion, duration_ms) with a grader_score field once graders are available, so recommendation gating can weight quality alongside sample size.
• Missing fields to capture now: per-run token_count/ai_credits_used, output_char_length, and a boolean guardrail_pass per declared guardrail — all currently absent from state.json/state.jsonl, which is why every guardrail in this run shows unsupported.

Generated by 🧪 daily-experiment-report · auto · 96.2 AIC · ⌖ 10.3 AIC · ⊞ 17.4K · ◷

• expires on Aug 26, 2026, 12:10 AM UTC-08:00