[experiments] Daily Experiment Report — 2026-09-25

[github-actions[bot]]

Daily Experiment Report — 2026-09-25

Automated statistical summary of all active agentic workflow A/B experiments in github/gh-aw. Decisions below are the canonical output of gh aw experiments analyze (deterministic core logic) — descriptive success-rate/duration stats are supplementary context only and never override the core decision.

Quick Stats

Metric	Value
Active experiments	51 (across 48 workflows)
Readiness: READY	26
Readiness: COLLECTING	25
Decision: PROMOTE	0
Decision: EXTEND	37
Decision: REJECT	0
Decision: INCONCLUSIVE	14
Interaction safety holds (PROMOTE + sparse cells)	0
Experiments with tracking issues	21
READY experiments w/ tracking issues (already notified previously)	11

Decision-Pipeline Gaps (surfaced this run)

• unsupported_multi_variant (14 experiments): every K≥3-variant experiment returns INCONCLUSIVE because the core analysis engine does not yet support statistical comparison beyond pairwise (2-variant) tests. These cannot progress to PROMOTE/REJECT until multi-variant support (e.g. ANOVA / Kruskal-Wallis with Bonferroni-corrected pairwise follow-up) is added.
• guardrail_unsupported (3 experiments): guardrail metrics are configured but no per-run observation pipeline currently feeds them, so guardrail pass/fail cannot be evaluated end-to-end. All guardrails repo-wide currently report status: unsupported.
• insufficient_observations (26 experiments): primary metric values are not yet being recorded per run for statistical comparison (assignment counts exist, but the metric itself is missing), blocking readiness advancement.
• CLI bug: gh aw experiments list/analyze --repo <owner/repo> fails because pkg/cli/experiments_fetch.go calls gh api ... --repo <repo>, and gh api has no --repo flag. Today's report used a local git-branch fallback (git fetch origin 'refs/heads/experiments/*:...') instead of the documented remote path. Filing as a follow-up dev task.

Per-Workflow Experiment Detail

Each workflow's active experiment(s) are shown below (compact ASCII comparison table, evidence, guardrails, and core decision).

agentperformanceanalyzer — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#33280``

prompt_compression

Hypothesis : H0: no change in aic. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| caveman          |   5 |  40.0% |      592.8  | [67.4,1118.2]    | ██████ 68/20 |
| verbose          |   5 |  60.0% |      782.4  | [294.3,1270.5]   | ██████ 58/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3767
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

agentpersonaexplorer — 1 experiment(s) — decision(s): EXTEND — readiness: READY

sub_agent_strategy

Hypothesis : H0: no change in aic or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| batch            |   4 |  50.0% |      563.5  | [219.0,908.0]    | ██████ 65/20 |
| per_scenario     |   6 |  33.3% |      491.3  | [274.4,708.2]    | ██████ 61/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.7203
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

architectureguardian — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

sub_agent_strategy

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| single_agent     |   4 | 100.0% |      623.5  | [392.5,854.5]    | █████░ 17/20 |
| sub_agents       |   6 | 100.0% |      662.0  | [491.8,832.2]    | ██████ 21/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.5239
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

auditworkflows — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING · tracking: ``#43177``

audit_decomposition

Hypothesis : H0: no change in run_success_rate. H1: phased_sub_agents improves run_success_rate by at least 15% relative while keeping runtime within ...
min_samples: 264/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| phased_sub_agents |   6 |  66.7% |      560.0  | [396.3,723.7]    | █░░░░░ 45/264 |
| single_agent     |   4 |  50.0% |      536.5  | [218.5,854.5]    | █░░░░░ 36/264 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3187
Guardrails: timeout_rate 0.05:UNSUPPORTED
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

awfailureinvestigator — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: READY · tracking: ``#36105``

tone_variant

Hypothesis : H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or ...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| assertive        |   5 | 100.0% |      541.8  | [493.2,590.4]    | ██████ 157/20 |
| clinical         |   2 | 100.0% |      672.5  | [208.7,1136.3]   | ██████ 154/20 |
| narrative        |   3 | 100.0% |      531.0  | [380.0,682.0]    | ██████ 151/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.9335
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

blogauditor — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

prompt_style

Hypothesis : H0: no change in audit correctness or discussion quality. H1: concise variant reduces token cost ≥20% with no degradation in validation a...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   4 |  75.0% |      629.0  | [399.5,858.5]    | █░░░░░ 4/20 |
| detailed         |   6 |  83.3% |      508.5  | [415.6,601.4]    | ██░░░░ 7/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3691
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

breakingchangechecker — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#42467``

tone_variant

Hypothesis : H0: no change in issue_engagement_rate. H1: urgent increases issue_engagement_rate by >=7 percentage points without degrading run_success...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| neutral          |   4 |   0.0% |      406.5  | [360.0,453.0]    | ██████ 26/20 |
| urgent           |   6 |  16.7% |      752.8  | [453.9,1051.8]   | ██████ 24/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.7691
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

cicoach — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#32335``

prompt_style

Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   1 | 100.0% |     1110.0  | N/A              | ██████ 38/20 |
| detailed         |   9 |  66.7% |     1365.8  | [880.0,1851.6]   | ██████ 57/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: mann_whitney, p=0.0000
Guardrails: run_success_rate >=0.85:UNSUPPORTED | empty_output_rate <=0.05:UNSUPPORTED
Decision: EXTEND  |  Reason: guardrail_unsupported  |  Report action: EXTEND
Rationale: mandatory guardrail "run_success_rate" is not backed by a supported metric observation

copilotagentanalysis — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING

output_format

Hypothesis : H0: no change in ai_credits_used. H1: prose or ste format reduces ai_credits_used by >=15% while keeping empty_discussion_rate <=5%
min_samples: 30/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| prose            |   2 |  50.0% |    11515.0  | [-120184.8,143214.8] | ██████ 34/30 |
| ste              |   3 | 100.0% |      658.3  | [598.3,718.4]    | █░░░░░ 3/30 |
| structured       |   5 |  80.0% |      664.6  | [219.4,1109.8]   | ██████ 43/30 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: t_test, p=0.0000
Guardrails: empty_discussion_rate 0.05:UNSUPPORTED
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailyagentrxtraceoptimizer — 1 experiment(s) — decision(s): EXTEND — readiness: READY

sub_agent_strategy

Hypothesis : H0: no change in issue quality or run success rate. H1: sub_agents variant yields higher evidence completeness score with equal or lower ...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| single_agent     |   5 |  20.0% |      660.0  | [120.6,1199.4]   | ██████ 42/20 |
| sub_agents       |   5 |  20.0% |      760.6  | [-116.2,1637.4]  | ██████ 42/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

dailyarchitecturediagram — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING · tracking: ``#31926``

detail_level

Hypothesis : H0: no change in run success rate or issue engagement. H1: brief variant reduces token usage ≥20% and run duration ≥15% with no drop in s...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| brief            |   5 |  80.0% |      432.4  | [305.7,559.1]    | ███░░░ 11/20 |
| comprehensive    |   5 |  80.0% |      457.4  | [125.8,789.0]    | ██░░░░ 7/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3484
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

dailyastrostylelitemarkdownspellcheck — 1 experiment(s) — decision(s): EXTEND — readiness: READY

prompt_style

Hypothesis : Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   5 |  60.0% |      464.0  | [326.4,601.6]    | ██████ 73/20 |
| detailed         |   5 |  20.0% |      376.0  | [299.0,453.0]    | ██████ 71/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.8431
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

dailycachestrategyanalyzer — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

model_size

Hypothesis : H0: no change in issue creation rate or run success rate. H1: gpt-5.3-codex-spark reduces AI Credits while keeping run success rate >=0.90.
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| gpt-5.3-codex    |   4 |  25.0% |      601.2  | [125.0,1077.5]   | ███░░░ 9/20 |
| gpt-5.3-codex-spark |   6 |   0.0% |      496.8  | [258.4,735.2]    | ███░░░ 10/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.8037
Guardrails: run_success_rate >=0.90:UNSUPPORTED | empty_output_rate <=0.10:UNSUPPORTED
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

dailycavemanoptimizer — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING

model_size

Hypothesis : H0: no change in PR creation rate or run success rate. H1: Claude Haiku reduces AI credit usage >=30% with equivalent run success rate (>...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| agent            |   0 |   N/A% |        N/A  | N/A              | ░░░░░░ 1/20 |
| claude-haiku-4.5 |   7 |  28.6% |      380.7  | [233.2,528.2]    | ██████ 67/20 |
| claude-sonnet-4.6 |   0 |   N/A% |        N/A  | N/A              | ██████ 34/20 |
| claude-sonnet-5  |   3 |  66.7% |      569.3  | [315.8,822.9]    | ███░░░ 9/20 |
| small-agent      |   0 |   N/A% |        N/A  | N/A              | █░░░░░ 2/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailycodemetrics — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING · tracking: ``#1``

output_format

Hypothesis : H0: no change in discussion engagement rate. H1: executive_summary variant increases discussion reactions+comments by ≥20% due to improve...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| executive_summary |   4 |   0.0% |      332.2  | [298.3,366.2]    | ██████ 54/20 |
| full_detail      |   2 |   0.0% |      376.0  | [-843.8,1595.8]  | ██████ 58/20 |
| ste              |   4 |   0.0% |      328.8  | [258.0,399.5]    | █████░ 16/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailycommunityattribution — 1 experiment(s) — decision(s): EXTEND — readiness: READY

prompt_style

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   5 |  60.0% |      510.8  | [256.5,765.1]    | ██████ 70/20 |
| verbose          |   5 |  40.0% |      433.2  | [179.6,686.8]    | ██████ 70/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

dailycompilerquality — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING · tracking: ``#32390``

output_format

Hypothesis : H0: no change in discussion engagement. H1: concise variant achieves equal engagement with ≥30% fewer output tokens; ste variant achieves...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   3 |   0.0% |      351.0  | [114.1,587.9]    | ██████ 54/20 |
| detailed         |   3 |  33.3% |      505.0  | [-252.3,1262.3]  | ██████ 61/20 |
| ste              |   4 |   0.0% |      383.0  | [239.3,526.7]    | █████░ 16/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailydochealer — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING

model_size

Hypothesis : H0: no change in issue/PR creation rate or run success rate. H1: Claude Haiku reduces AI credit usage >=30% with equivalent run success r...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| agent            |   0 |   N/A% |        N/A  | N/A              | █░░░░░ 2/20 |
| claude-haiku-4.5 |   7 | 100.0% |      685.3  | [476.4,894.2]    | ██████ 60/20 |
| claude-sonnet-4.6 |   0 |   N/A% |        N/A  | N/A              | ██████ 43/20 |
| claude-sonnet-5  |   3 |  66.7% |      800.3  | [-321.3,1922.0]  | ██░░░░ 7/20 |
| small-agent      |   0 |   N/A% |        N/A  | N/A              | ░░░░░░ 1/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailydocupdater — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING

model_size

Hypothesis : H0: no change in PR creation rate or run success rate. H1: Claude Haiku reduces AI credit usage >=30% with equivalent run success rate (>...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| agent            |   0 |   N/A% |        N/A  | N/A              | ░░░░░░ 1/20 |
| claude-haiku-4.5 |   4 |  75.0% |      380.0  | [321.9,438.1]    | ██████ 46/20 |
| claude-sonnet-4.6 |   0 |   N/A% |        N/A  | N/A              | ██████ 47/20 |
| claude-sonnet-5  |   6 |  33.3% |      309.5  | [257.7,361.3]    | ████░░ 12/20 |
| small-agent      |   0 |   N/A% |        N/A  | N/A              | █░░░░░ 2/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailyfact — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#31324``

reasoning_depth

Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% re...
min_samples: 30/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| multi_candidate  |   7 |   0.0% |      342.4  | [283.4,401.4]    | ██████ 40/30 |
| single_pass      |   3 |   0.0% |      461.7  | [-0.3,923.7]     | ██████ 55/30 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.1195
Guardrails: empty_output_rate <0.05:UNSUPPORTED | run_success_rate >=0.95:UNSUPPORTED
Decision: EXTEND  |  Reason: guardrail_unsupported  |  Report action: EXTEND
Rationale: mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

dailyfunctionnamer — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

model_size

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: at least two variants are required for a decision

dailyissuesreport — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: READY · tracking: ``#30573``

output_format

Hypothesis : H0: no change in discussion engagement score. H1: inline format produces ≥20% higher reactions+replies by making charts and recommendatio...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| collapsible      |   4 |  75.0% |      746.0  | [683.7,808.3]    | ██████ 59/20 |
| inline           |   2 | 100.0% |      717.5  | [75.8,1359.2]    | ██████ 61/20 |
| ste              |   4 |  75.0% |      793.8  | [354.8,1232.7]   | ██████ 21/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailynews — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#31190``

prompt_style

Hypothesis : H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   7 | 100.0% |      729.4  | [679.2,779.7]    | ██████ 61/20 |
| detailed         |   3 | 100.0% |      695.0  | [547.0,843.0]    | ██████ 50/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.2969
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

dailyrenderingscriptsverifier — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

remove_redundant_context_v1

Hypothesis : H0: No meaningful difference in execution-duration between control and candidate. H1: Removing the four redundant Guidelines bullets decr...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: at least two variants are required for a decision

dailysafeoutputoptimizer — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING · tracking: ``#38094``

log_fetch_strategy

Hypothesis : H0: no change in run_duration_ms. H1: eager reduces run duration by >=15% by avoiding MCP log-fetch turns
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
+------------------+-----+--------+-------------+------------------+------------+
Evidence: mann_whitney, p=0.0000
Guardrails: issue_creation_success_rate 0.8:UNSUPPORTED
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: at least two variants are required for a decision

dailysecurityredteam — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#31673``

reasoning_depth

Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead
min_samples: 30/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| iterative        |   4 |   0.0% |     3822.2  | [3774.8,3869.7]  | ██████ 80/30 |
| single_pass      |   6 |   0.0% |     3812.5  | [3771.2,3853.8]  | ██████ 55/30 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: proportion_test, p=0.0297
Guardrails: run_success_rate >=0.9:UNSUPPORTED | empty_output_rate <=0.05:UNSUPPORTED
Decision: EXTEND  |  Reason: guardrail_unsupported  |  Report action: EXTEND
Rationale: mandatory guardrail "run_success_rate" is not backed by a supported metric observation

dailysemgrepscan — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING · tracking: ``#32795``

semgrep_output_format

Hypothesis : H0: no change in alert creation rate across formats. H1: structured_sections produces ≥15% more alerts successfully created vs. baseline ...
min_samples: 30/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| bullet_list      |   3 | 100.0% |      537.7  | [284.8,790.5]    | ██████ 28/30 |
| prose            |   3 | 100.0% |      527.3  | [317.0,737.7]    | █████░ 27/30 |
| structured_sections |   4 | 100.0% |      440.2  | [306.0,574.5]    | ████░░ 22/30 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: proportion_test, p=0.0232
Guardrails: run_success_rate >=0.85:UNSUPPORTED
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dailysubagentoptimizer — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING

timeout_setting

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| default          |   2 | 100.0% |      799.0  | [-382.7,1980.7]  | █░░░░░ 2/20 |
| relaxed          |   1 | 100.0% |      659.0  | N/A              | ░░░░░░ 1/20 |
| tight            |   1 | 100.0% |      706.0  | N/A              | ░░░░░░ 1/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.7814
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dataflowprdiscussiondataset — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING · tracking: ``#37102``

caveman_mode

Hypothesis : H0: no change in input_token_count, retention_rate, or run_success_rate. H1: caveman prompt cuts input tokens ≥30% with retention_rate ≥0...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| no               |   4 |  75.0% |     5895.0  | [-11107.8,22897.8] | ██░░░░ 6/20 |
| yes              |   6 |  66.7% |     4109.0  | [-4998.6,13216.6] | ███░░░ 10/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3187
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

deepreport — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: READY

output_format

Hypothesis : H0: no change in discussion engagement or token cost. H1: executive_brief reduces token usage by ≥20% without reducing engagement; annota...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| annotated_brief  |   4 |  50.0% |      953.5  | [645.2,1261.8]   | ██████ 56/20 |
| executive_brief  |   3 | 100.0% |      998.3  | [736.8,1259.8]   | ██████ 82/20 |
| full_briefing    |   1 |   0.0% |      910.0  | N/A              | ██████ 58/20 |
| ste              |   2 |   0.0% |      774.5  | [56.6,1492.4]    | ██████ 33/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0001
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

dependabotcampaign — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

summary_detail

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| brief            |   4 | 100.0% |      302.0  | [259.6,344.4]    | ██░░░░ 6/20 |
| detailed         |   6 | 100.0% |      294.3  | [255.0,333.6]    | ██░░░░ 6/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

dependabotgochecker — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING

prompt_style

Hypothesis : H0: no change in issue categorization accuracy across prompt styles. H1: concise style reduces token cost by ≥15% with <5% drop in accuracy.
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   3 |  66.7% |      380.0  | [211.5,548.5]    | █████░ 18/20 |
| detailed         |   4 |  25.0% |      403.0  | [283.5,522.5]    | █████░ 18/20 |
| step_by_step     |   3 |  33.3% |      324.0  | [199.0,449.0]    | ████░░ 15/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.8374
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

gpclean — 1 experiment(s) — decision(s): EXTEND — readiness: READY

tool_verbosity

Hypothesis : H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + al...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| full_bash        |   6 |   0.0% |      296.2  | [251.0,341.3]    | ██████ 58/20 |
| minimal_toolset  |   4 |   0.0% |      281.0  | [256.0,306.0]    | ██████ 65/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.5355
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

issuearborist — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#30015``

prompt_style

Hypothesis : H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| concise          |   4 |   0.0% |      345.8  | [225.8,465.7]    | ██████ 44/20 |
| detailed         |   6 |   0.0% |      327.5  | [285.2,369.8]    | ██████ 38/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.5150
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

plan — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING · tracking: ``#42941``

reasoning_depth

Hypothesis : H0: no change in plan quality score or success rate. H1: deep improves quality score by >=8%, or shallow reduces run cost/latency by >=15...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| baseline         |   4 | 100.0% |      447.5  | [332.8,562.2]    | █░░░░░ 4/20 |
| deep             |   2 | 100.0% |      545.0  | [-916.2,2006.2]  | █░░░░░ 2/20 |
| shallow          |   1 | 100.0% |      470.0  | N/A              | ░░░░░░ 1/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3694
Guardrails: none configured
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

prsouschef — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

remove_redundant_context_v1

Hypothesis : H0: No meaningful difference in execution-duration between control and candidate. H1: Removing the redundant backup skip-rules table decr...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: at least two variants are required for a decision

smokeantigravity — 1 experiment(s) — decision(s): EXTEND — readiness: READY

sub_agent_strategy

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| single_agent     |   5 | 100.0% |      616.8  | [234.7,998.9]    | ██████ 99/20 |
| sub_agents       |   5 | 100.0% |      720.2  | [214.8,1225.6]   | ██████ 109/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.4952
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

smokecopilot — 2 experiment(s) — decision(s): EXTEND — readiness: READY

caveman

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| no               |   5 |  40.0% |      746.4  | [582.4,910.4]    | ██████ 185/20 |
| yes              |   5 |  40.0% |      821.4  | [716.1,926.7]    | ██████ 185/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

subagent_model

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| large            |   5 |  40.0% |      743.8  | [583.7,903.9]    | ██████ 175/20 |
| small            |   5 |  40.0% |      824.0  | [716.3,931.7]    | ██████ 175/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

smokecopilotaoaiapikey — 2 experiment(s) — decision(s): EXTEND — readiness: READY

caveman

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| no               |   5 |   0.0% |      457.2  | [412.0,502.4]    | ██████ 94/20 |
| yes              |   5 |  20.0% |      530.4  | [373.4,687.4]    | ██████ 94/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

subagent_model

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| large            |   5 |  20.0% |      517.6  | [358.7,676.5]    | ██████ 94/20 |
| small            |   5 |   0.0% |      470.0  | [403.4,536.6]    | ██████ 94/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=1.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

smokecopilotaoaientra — 2 experiment(s) — decision(s): EXTEND — readiness: READY

caveman

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| no               |   5 |   0.0% |      493.2  | [365.1,621.3]    | ██████ 81/20 |
| yes              |   5 |  20.0% |      600.8  | [390.2,811.4]    | ██████ 80/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.8962
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

subagent_model

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| large            |   4 |   0.0% |      537.0  | [356.7,717.3]    | ██████ 80/20 |
| small            |   6 |  16.7% |      553.7  | [373.5,733.8]    | ██████ 81/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.8962
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

smokecopilotsubagents — 1 experiment(s) — decision(s): INCONCLUSIVE — readiness: COLLECTING · tracking: ``#47551``

sub_agent_strategy

Hypothesis : H0: no change in pass_rate between inline_strict and delegated_sequential. H1: delegated_sequential improves pass_rate by >= 0.15 absolut...
min_samples: 30/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| delegated_sequential |   1 | 100.0% |      381.0  | N/A              | ███░░░ 15/30 |
| inline_strict    |   5 |  80.0% |      417.6  | [313.2,522.0]    | ███░░░ 14/30 |
| single_agent_control |   4 |  50.0% |      366.0  | [200.4,531.6]    | ████░░ 19/30 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.6110
Guardrails: empty_output_rate 0.01:UNSUPPORTED | false_pass_rate 0.05:UNSUPPORTED
Decision: INCONCLUSIVE  |  Reason: unsupported_multi_variant  |  Report action: INCONCLUSIVE
Rationale: automatic decisions currently require exactly two variants

smokegemini — 1 experiment(s) — decision(s): EXTEND — readiness: READY

sub_agent_strategy

Hypothesis : H0: no change in aic. H1: sub_agents reduces tokens by >=20%
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| single_agent     |   7 |  14.3% |      518.3  | [196.0,840.6]    | ██████ 128/20 |
| sub_agents       |   3 |   0.0% |      372.7  | [323.6,421.7]    | ██████ 143/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3655
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

smokepi — 1 experiment(s) — decision(s): EXTEND — readiness: READY

sub_agent_decomposition

Hypothesis : (not specified)
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| parallel_sub_agents |   5 | 100.0% |      259.2  | [144.0,374.4]    | ██████ 42/20 |
| single_agent     |   5 | 100.0% |      198.8  | [173.4,224.2]    | ██████ 34/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.3619
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

smokeproject — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

prompt_style_test

Hypothesis : H0: no change in run_success_rate. H1: concise/step_by_step achieve ≥90% success rate with ≥30% fewer prompt tokens
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.0000
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: at least two variants are required for a decision

smoketemporaryid — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING

sub_agent_strategy

Hypothesis : H0: no change in AI credit count. H1: sub-agents reduce AI credit count by 15-25% and improve success rate.
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
+------------------+-----+--------+-------------+------------------+------------+
Evidence: t_test, p=0.0000
Guardrails: all_issues_created ==3:UNSUPPORTED | temporary_id_resolution_rate >=0.95:UNSUPPORTED
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: at least two variants are required for a decision

testqualitysentinel — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#43530``

model_size

Hypothesis : H0: model-size variant does not improve review usefulness acceptance rate. H1: a larger reasoning-capable model improves review usefulnes...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| claude-haiku-4.5 |   5 | 100.0% |     1026.0  | [140.5,1911.5]   | ██████ 32/20 |
| claude-sonnet-5  |   5 | 100.0% |      805.4  | [-82.2,1693.0]   | ██████ 30/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.7879
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

typist — 1 experiment(s) — decision(s): EXTEND — readiness: READY · tracking: ``#34032``

tone_style

Hypothesis : H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis qua...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| conversational   |   4 |  75.0% |     1013.2  | [399.6,1626.9]   | ██████ 47/20 |
| formal           |   6 |  50.0% |      732.5  | [272.1,1192.9]   | ██████ 41/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.5300
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_observations  |  Report action: EXTEND
Rationale: primary metric observations are not available for statistical comparison

weeklyblogpostwriter — 1 experiment(s) — decision(s): EXTEND — readiness: COLLECTING · tracking: ``#38590``

prefetch_strategy

Hypothesis : H0: no change in agent_turn_count. H1: eager pre-fetching reduces agent turns by ≥20% and duration by ≥15% by eliminating tool round-trip...
min_samples: 20/variant  |  Window: last 30 runs
+------------------+-----+--------+-------------+------------------+------------+
| Variant          |  n  | Succ % | Mean dur(s) | 95% CI (s)       | min_samp   |
+------------------+-----+--------+-------------+------------------+------------+
| eager            |   4 | 100.0% |      620.8  | [598.5,643.0]    | ██░░░░ 5/20 |
| lazy             |   6 | 100.0% |      741.2  | [637.7,844.6]    | ███░░░ 10/20 |
+------------------+-----+--------+-------------+------------------+------------+
Evidence: none, p=0.1935
Guardrails: none configured
Decision: EXTEND  |  Reason: insufficient_samples  |  Report action: EXTEND
Rationale: one or more variants have fewer usable observations than min_samples

Factorial Interaction Diagnostics

Three workflows (smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra) run two concurrent experiments (caveman × subagent_model). Pairwise interaction cells were computed for each; all cells are sparse (n < min_samples=20 per cell), yielding interaction_risk_status: SPARSE_CELL_RISK. However, none of these experiments' core decisions are PROMOTE, so the interaction safety hold (which only applies to a PROMOTE core decision) does not trigger for any of them today — report_action equals core_decision (EXTEND) in all three cases.

Self-Tuning Continuation Plan

Top experiment actions:

1. Extend the 14 unsupported_multi_variant (K≥3) experiments until multi-variant statistical comparison lands in core; consider splitting long-running 3+-variant experiments into pairwise A/B pairs to unblock decisions sooner.
2. Wire up primary-metric observation collection for the 26 insufficient_observations experiments — assignment counts are recorded but the metric values needed for comparison are not, which is the single largest blocker to readiness across the fleet.
3. For the 11 READY experiments with tracking issues (already notified on a prior day), no new tracking-issue comment is warranted today since no decision changed to PROMOTE/REJECT and none newly reached READY today — continue monitoring.

Top eval/pipeline actions:

1. Implement guardrail metric observation wiring so the 3 guardrail_unsupported experiments (and all guardrail-configured experiments generally) can produce real pass/fail guardrail results instead of status: unsupported.
2. Fix the gh aw experiments list/analyze --repo CLI bug (gh api ... --repo <repo> is an invalid flag combination in pkg/cli/experiments_fetch.go) so future runs of this report don't need the local git-branch fallback.
3. Backfill/verify hypothesis/metric/secondary_metrics frontmatter for the ~27 experiments currently missing an explicit hypothesis, to make the ASCII tables above more informative.

Decision-pipeline gaps: unsupported_multi_variant (14), guardrail_unsupported (3, and universally unsupported across all guardrail-configured experiments), insufficient_observations (26), and the --repo/gh api CLI flag bug noted above.

Generated 2026-09-25T08:12:56.389082Z by the daily-experiment-report workflow.

Generated by 🧪 daily-experiment-report · copilot · auto · 255.5 AIC · ⌖ 33 AIC · ⊞ 16.4K · ◷

• expires on Sep 28, 2026, 12:17 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #63590.