[experiments] Daily Experiment Report — 2026-05-27

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-05-27

22 active experiments tracked across 21 workflows. 8 experiments have outcome data available for statistical analysis — all are still collecting data (no experiment reached statistical significance today). No promotions or guardrail failures detected.

⚡ Quick Stats

Metric	Value
Active experiments	22
Experiments with outcome data	8
Statistically significant (p < 0.05)	0
Recommendations	✅ PROMOTE: 0 · 🟡 EXTEND: 7 · ❌ ABANDON: 1

---

caveman · smoke-copilot.md

Status: 🟢 READY
Variants: no vs yes · Window: last 30 runs
min_samples: 30 per variant

Does caveman mode affect smoke test success rate?

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
no	37	██████████ 37/30 (100%)
yes	36	██████████ 36/30 (100%)

Experiment : caveman
Workflow   : smoke-copilot.md
Hypothesis : Does caveman mode affect smoke test success rate?
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| no               |   37 |    57.1% |          758.9 | [660,858]          |     (ref) | ████████ 37/30 (100%) |
| yes              |   36 |    53.8% |         1425.0 | [147,2703]         |    0.8632 | ████████ 36/30 (100%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: ABANDON
Rationale     : No significant difference detected with all variants at 30+ runs

Recommendation: ABANDON — No significant difference detected with all variants at 30+ runs

---

subagent_model · smoke-copilot.md

Status: 🟡 COLLECTING
Variants: large vs small · Window: last 30 runs
min_samples: 30 per variant

Does sub-agent model size affect test success and duration?

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
large	26	█████████░ 26/30 (87%)
small	27	█████████░ 27/30 (90%)

Chart not available

Experiment : subagent_model
Workflow   : smoke-copilot.md
Hypothesis : Does sub-agent model size affect test success and duration?
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| large            |   26 |    46.2% |         1399.5 | [118,2681]         |     (ref) | ███████░ 26/30 (87%) |
| small            |   27 |    64.3% |          782.6 | [668,898]          |    0.3434 | ███████░ 27/30 (90%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 30 runs

Recommendation: EXTEND — Collecting data: not all variants reached 30 runs

---

sub_agent_strategy · smoke-gemini.md

Status: 🟡 COLLECTING
Variants: single_agent vs sub_agents · Window: last 30 runs
min_samples: 30 per variant

Does using sub-agents improve Gemini smoke test success?

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_agent	15	█████░░░░░ 15/30 (50%)
sub_agents	28	█████████░ 28/30 (93%)

Experiment : sub_agent_strategy
Workflow   : smoke-gemini.md
Hypothesis : Does using sub-agents improve Gemini smoke test success?
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| single_agent     |   15 |    88.9% |          378.1 | [234,522]          |     (ref) | ████░░░░ 15/30 (50%) |
| sub_agents       |   28 |    94.1% |          814.5 | [-112,1741]        |    0.6341 | ███████░ 28/30 (93%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 30 runs

Recommendation: EXTEND — Collecting data: not all variants reached 30 runs

---

sub_agent_decomposition · smoke-pi.md

Status: 🟡 COLLECTING
Variants: single_agent vs parallel_sub_agents · Window: last 30 runs
min_samples: 20 per variant

Does parallel sub-agent decomposition improve Pi calculation quality?

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_agent	20	██████████ 20/20 (100%)
parallel_sub_agents	13	██████░░░░ 13/20 (65%)

Experiment : sub_agent_decomposition
Workflow   : smoke-pi.md
Hypothesis : Does parallel sub-agent decomposition improve Pi calculation quality?
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| single_agent     |   20 |    88.9% |          634.0 | [-239,1507]        |     (ref) | ████████ 20/20 (100%) |
| parallel_sub_agents |   13 |   100.0% |          227.2 | [149,305]          |    0.3264 | █████░░░ 13/20 (65%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 20 runs

Recommendation: EXTEND — Collecting data: not all variants reached 20 runs

---

prompt_style · ci-coach.md

Status: 🟡 COLLECTING
Variants: detailed vs concise · Window: last 30 runs
min_samples: 20 per variant

H0: no change in PR rate. H1: concise prompt reduces tokens ≥25% without quality loss

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
detailed	5	██░░░░░░░░ 5/20 (25%)
concise	3	██░░░░░░░░ 3/20 (15%)

Chart not available

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR rate. H1: concise prompt reduces tokens ≥25% without quality loss
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| detailed         |    5 |    40.0% |          449.6 | [97,802]           |     (ref) | ██░░░░░░ 5/20 (25%) |
| concise          |    3 |    33.3% |          499.7 | [-715,1715]        |    0.8504 | █░░░░░░░ 3/20 (15%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Guardrails:
  run_success_rate >=0.85: detailed=FAIL(0.40), concise=FAIL(0.33)
  empty_output_rate <=0.05: (insufficient data)

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 20 runs

Recommendation: EXTEND — Collecting data: not all variants reached 20 runs

---

output_format · daily-issues-report.md

Status: 🟡 COLLECTING
Variants: collapsible vs inline · Window: last 30 runs
min_samples: 30 per variant

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
collapsible	7	██░░░░░░░░ 7/30 (23%)
inline	13	████░░░░░░ 13/30 (43%)

Chart not available

Experiment : output_format
Workflow   : daily-issues-report.md
Hypothesis : (not specified)
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| collapsible      |    7 |     0.0% |          330.9 | [304,358]          |     (ref) | ██░░░░░░ 7/30 (23%) |
| inline           |   13 |     0.0% |          357.8 | [298,418]          |       N/A | ███░░░░░ 13/30 (43%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 30 runs

Recommendation: EXTEND — Collecting data: not all variants reached 30 runs

---

prompt_style · daily-community-attribution.md

Status: 🟡 COLLECTING
Variants: concise vs verbose · Window: last 30 runs
min_samples: 20 per variant

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	11	██████░░░░ 11/20 (55%)
verbose	11	██████░░░░ 11/20 (55%)

Experiment : prompt_style
Workflow   : daily-community-attribution.md
Hypothesis : (not specified)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| concise          |   11 |    90.9% |          548.2 | [422,675]          |     (ref) | ████░░░░ 11/20 (55%) |
| verbose          |   11 |    81.8% |          413.7 | [321,507]          |    0.5344 | ████░░░░ 11/20 (55%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 20 runs

Recommendation: EXTEND — Collecting data: not all variants reached 20 runs

---

prompt_style · daily-astrostylelite-markdown-spellcheck.md

Status: 🟡 COLLECTING
Variants: concise vs detailed · Window: last 30 runs
min_samples: 30 per variant

(not specified)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	9	███░░░░░░░ 9/30 (30%)
detailed	13	████░░░░░░ 13/30 (43%)

Chart not available

Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck.md
Hypothesis : (not specified)
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples       |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
| concise          |    9 |    88.9% |          427.0 | [366,488]          |     (ref) | ██░░░░░░ 9/30 (30%) |
| detailed         |   13 |    92.3% |          410.7 | [356,466]          |    0.7839 | ███░░░░░ 13/30 (43%) |
+------------------+------+----------+----------------+--------------------+-----------+-------------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001

Recommendation: EXTEND
Rationale     : Collecting data: not all variants reached 30 runs

Recommendation: EXTEND — Collecting data: not all variants reached 30 runs

---

📊 Summary — All Experiments

View Full Experiments Table

Experiment	Workflow	Total runs	Min n/target	Control SR	Rec
prompt_compression	agentperformanceanalyzer	7	3/20	N/A	🟡 EXTEND
sub_agent_strategy	agentpersonaexplorer	5	1/20	N/A	🟡 EXTEND
prompt_style	cicoach	8	3/20	33.3%	🟡 EXTEND
detail_level	dailyarchitecturediagram	1	1/20	N/A	🟡 EXTEND
prompt_style	dailyastrostylelitemarkdownspellcheck	22	9/30	92.3%	🟡 EXTEND
output_format	dailycodemetrics	11	4/20	N/A	🟡 EXTEND
prompt_style	dailycommunityattribution	22	11/20	90.9%	🟡 EXTEND
output_format	dailycompilerquality	12	4/20	N/A	🟡 EXTEND
reasoning_depth	dailyfact	10	3/20	N/A	🟡 EXTEND
output_format	dailyissuesreport	20	7/30	0.0%	🟡 EXTEND
prompt_style	dailynews	16	6/20	N/A	🟡 EXTEND
reasoning_depth	dailysecurityredteam	15	7/20	N/A	🟡 EXTEND
semgrep_output_format	dailysemgrepscan	10	2/20	N/A	🟡 EXTEND
output_format	deepreport	16	5/20	N/A	🟡 EXTEND
tool_verbosity	gpclean	3	1/20	N/A	🟡 EXTEND
prompt_style	issuearborist	4	2/20	N/A	🟡 EXTEND
sub_agent_strategy	smokeantigravity	14	6/20	N/A	🟡 EXTEND
caveman	smokecopilot	73	36/30	57.1%	❌ ABANDON
subagent_model	smokecopilot	53	26/30	46.2%	🟡 EXTEND
sub_agent_strategy	smokegemini	43	15/30	94.1%	🟡 EXTEND
sub_agent_decomposition	smokepi	33	13/20	88.9%	🟡 EXTEND
tone_style	typist	1	1/20	N/A	🟡 EXTEND

Analysis window: last 30 runs per workflow · Significance threshold: p < 0.05 (two-tailed)
Run: §26501239502

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · sonnet46 6.5M · ◷

• expires on May 30, 2026, 9:11 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-30T09:11:35.780Z.

Closed by Workflow