[experiments] Daily Experiment Report — 2026-05-05

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-05-05

4 experiments analysed across 4 workflows. No experiments have reached statistical readiness — all are in the data-collection phase. Recommendation for all active experiments: EXTEND.

⚠️ Infrastructure note: Experiment state is not being persisted to git branches due to a bug in push_signed_commits.cjs (orphan branch first-commit: empty baseRef returns 0 commits to GraphQL, push silently succeeds without writing remotely). Variant counts below are estimated from the balanced least-count algorithm.

---

📊 Overview

---

Experiment 1 — daily-community-attribution · prompt_style

Experiment : prompt_style
Workflow   : daily-community-attribution.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 4 runs (all with experiment artifact)
min_samples: 20 per variant

+------------------+------+----------+--------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s) | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+--------------+--------------------+-----------+---------------+
| concise (ctrl)   |  2*  | 100.0%   |     N/A      | N/A                |  (ref)    |  2/20  (10%)  |
| verbose          |  2*  | 100.0%   |     N/A      | N/A                |   N/A     |  2/20  (10%)  |
+------------------+------+----------+--------------+--------------------+-----------+---------------+
* Variant counts estimated — state.json not persisted to git (see infrastructure note)
  Duration per variant unknown — cannot separate without state.json
  Overall durations (all 4 runs): 834s, 327s, 399s, 554s  (mean=529s)

Recommendation: EXTEND
Rationale     : Only 4/20 minimum samples collected; no statistical inference possible yet.

---

Experiment 2 — daily-astrostylelite-markdown-spellcheck · prompt_style

Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck.md
Hypothesis : "Testing if detailed prompts improve spellcheck quality and catch more issues"
Window     : last 30 runs  |  Analysed: 2 runs with experiment artifact
min_samples: 30 per variant

Guardrails declared:
  empty_output_rate <0.10
  run_success_rate  >=0.90

+------------------+------+----------+--------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s) | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+--------------+--------------------+-----------+---------------+
| concise (ctrl)   |  1*  | 100.0%   |    615.0     | N/A (n<2)          |  (ref)    |  1/30  ( 3%)  |
| detailed         |  1*  | 100.0%   |    532.0     | N/A (n<2)          |   N/A     |  1/30  ( 3%)  |
+------------------+------+----------+--------------+--------------------+-----------+---------------+
* Variant counts estimated  |  Guardrails: insufficient data to evaluate

Recommendation: EXTEND
Rationale     : Only 2/30 minimum samples collected; far too early for any inference.

---

Experiment 3 — smoke-copilot · caveman

Experiment : caveman
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 14 non-skipped runs since May 2
min_samples: 20 per variant

Note: smoke-copilot runs across PR branches + main. State starts fresh each run
      (state.json not persisted — see infrastructure note). Variant assignment is
      random per run (first run always has balanced 0/0 counts → random pick).

+------------------+------+----------+--------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s) | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+--------------+--------------------+-----------+---------------+
| yes (ctrl)       |  7*  |  57.1%*  |     N/A      | N/A                |  (ref)    |  7/20  (35%)  |
| no               |  7*  |  57.1%*  |     N/A      | N/A                |   N/A     |  7/20  (35%)  |
+------------------+------+----------+--------------+--------------------+-----------+---------------+
* Variant counts estimated 50/50 (random assignment with fresh state each run)
  Success rate estimated from overall: 8 success / 14 runs = 57%
  Duration per variant unknown without state.json

Recommendation: EXTEND
Rationale     : 7/20 estimated samples per variant; without persisted state, 
                per-variant statistics cannot be computed reliably.

---

Experiment 4 — issue-arborist · prompt_style

Experiment : prompt_style
Workflow   : issue-arborist.md
Hypothesis : "Testing if detailed prompts improve triage accuracy"
start_date : 2026-05-05 (TODAY — experiment just activated)
Tracking   : issue `#30015`
min_samples: 30 per variant

Guardrail  : empty_output_rate ==0

Status: NOT STARTED — no runs recorded with experiment yet.

Recommendation: EXTEND (0/30 samples)
Rationale     : Experiment activates today; await first runs before analysis.

---

🔍 Infrastructure Finding

A bug was identified in experiment state persistence. In push_experiment_state.cjs, when creating a new orphan branch for the first time, checkoutOrCreateBranch() returns an empty string "" as baseRef. This causes pushSignedCommits to compute git rev-list ""..HEAD which yields zero commits, logging "no new commits to push via GraphQL" and returning without error — while the state was never actually pushed to the remote branch. Subsequent runs all start from empty state rather than accumulating history.

Impact: Per-variant cumulative counts and run histories are not being persisted. Each run picks variants randomly instead of using the balanced least-count strategy over the full history.

Tracking: See push_signed_commits.cjs line 138 and push_experiment_state.cjs for the orphan branch path.

---

📋 Summary Table

Workflow	Experiment	Runs	min_samples	Progress	Recommendation
community-attribution	prompt_style	4	20	10%	⏳ EXTEND
spellcheck	prompt_style	2	30	3%	⏳ EXTEND
smoke-copilot	caveman	14	20	35%*	⏳ EXTEND
issue-arborist	prompt_style	0	30	0%	⏳ EXTEND

*smoke-copilot count includes all non-skipped runs; per-variant split is random (state not persisted)

References: §25366113777

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

• productionresultssa13.blob.core.windows.net
• productionresultssa14.blob.core.windows.net
• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "productionresultssa13.blob.core.windows.net"
    - "productionresultssa14.blob.core.windows.net"
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by daily-experiment-report · ● 11.6M · ◷

• expires on May 8, 2026, 8:56 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #30799.