[experiments] Daily Experiment Report — 2026-06-26

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-26

37 experiments across 38 workflows analysed. 2 reached a final decision today.

⚠️ Health alert: daily-issues-report shows 0% success rate for both variants across 30 runs — investigate root cause independently of the experiment.

⚡ Quick Stats

Metric	Value
Active experiments	37
At min_samples (READY)	9
Statistically significant	0
Recommendations	✅ PROMOTE: 0 · 🟡 EXTEND: 26 · ❌ ABANDON: 2 · 🔵 READY: 9

---

❌ caveman · smoke-copilot — ABANDON

no=85.7% (n=14) vs yes=73.3% (n=15) · p=0.411 · State: 107/108 runs · min_samples=20 ✅

📈 Statistics & Factorial

| Variant       |  n | Succ%  | Dur(s) | 95% CI        | p-value | Progress          |
|---------------|----|--------|--------|---------------|---------|-------------------|
| no (ctrl)     | 14 |  85.7% |  802.6 | [722 , 883]   |  (ref)  | ██████████ 107/20 |
| yes           | 15 |  73.3% |  820.5 | [748 , 893]   |  0.411  | ██████████ 108/20 |

Factorial (caveman × subagent_model): chi2=2.09, p=0.148 — ⚠️ SPARSE_CELL_RISK (cells n<20)

❌ ABANDON — No significant effect (p=0.411). Caveman compression neither helps nor hurts.

---

❌ subagent_model · smoke-copilot — ABANDON

large=86.7% (n=15) vs small=71.4% (n=14) · p=0.311 · State: 98/97 runs · min_samples=20 ✅

| Variant       |  n | Succ%  | Dur(s) | 95% CI        | p-value | Progress         |
|---------------|----|--------|--------|---------------|---------|------------------|
| large (ctrl)  | 15 |  86.7% |  823.1 | [738 , 908]   |  (ref)  | ██████████ 98/20 |
| small         | 14 |  71.4% |  799.8 | [735 , 864]   |  0.311  | ██████████ 97/20 |

❌ ABANDON — No significant effect (p=0.311). Retain large sub-agent model as default.

---

🔵 Ready for Analysis (9 experiments)

All variants at min_samples — awaiting outcome correlation from workflow API data.

Experiment	Workflow	Counts	min_samples
sub_agent_strategy	smoke-gemini	101/70 ⚠️ imbalanced	30
sub_agent_strategy	smoke-antigravity	67/75	30
caveman	smoke-copilot-aoai-apikey	45/45	20
subagent_model	smoke-copilot-aoai-apikey	45/45	20
caveman	smoke-copilot-aoai-entra	31/31	20
subagent_model	smoke-copilot-aoai-entra	31/31	20
prompt_compression	agent-performance-analyzer	22/14	14
prompt_style	daily-community-attribution	26/26	20
tone_style	typist	12/12	10

smoke-gemini has imbalanced assignment (101 vs 70, p=0.011). Account for this in analysis.

---

🟡 Notable EXTEND Experiments with Outcome Data

Experiment	Workflow	Ctrl%	Best%	p-value	Status
prompt_style	daily-news	7.1% (detailed)	31.2% (concise)	0.100	Trending ↑
output_format	daily-code-metrics	75.0% (exec_summary)	94.1% (full_detail)	0.141	Trending ↑
output_format	daily-issues-report	0.0% (collapsible)	0.0% (inline)	N/A	⚠️ 0% success
output_format	deep-report	100% (exec_brief)	100% (full_briefing)	N/A	Balanced
reasoning_depth	daily-fact	64.7% (single_pass)	66.7% (multi_cand.)	0.913	No effect

View All 26 EXTEND Experiments

Workflow	Experiment	Counts	min_samples	Note
aw-failure-investigator	tone_variant	40/30/31	50
daily-astrostylelite-markdown-spellcheck	prompt_style	28/26	30
daily-security-red-team	reasoning_depth	25/20	30
daily-compiler-quality	output_format	18/23	20
daily-semgrep-scan	semgrep_output_format	11/16/13	30
agent-persona-explorer	sub_agent_strategy	24/12	14
gpclean	tool_verbosity	14/19	30
ci-coach	prompt_style	14/16	20
copilot-agent-analysis	output_format	8/10	30
daily-agentrx-trace-optimizer	sub_agent_strategy	7/12	20
daily-doc-updater	model_size (4 variants)	2/1/10/9	20	⚠️ n<5
daily-doc-healer	model_size (4 variants)	1/2/12/7	20	⚠️ n<5
daily-caveman-optimizer	model_size (4 variants)	2/1/7/12	20	⚠️ n<5
daily-cache-strategy-analyzer	model_size (6 variants)	varies	20	⚠️ n<5
daily-safe-output-optimizer	log_fetch_strategy	6/5	20
dependabot-go-checker	prompt_style	5/4/3	30	⚠️ n<5
architecture-guardian	sub_agent_strategy	3/5	30	⚠️ n<5
issue-arborist	prompt_style	2/2	30	⚠️ n<5
daily-architecture-diagram	detail_level	4	10	⚠️ n<5
dataflow-pr-discussion-dataset	caveman_mode	1/2	10	⚠️ n<5
weekly-blog-post-writer	prefetch_strategy	2	10	⚠️ n<5

Analysis window: last 30 runs · Significance threshold: p < 0.05 (two-tailed)
References: §28227665765

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 229.9 AIC · ⌖ 31.6 AIC · ⊞ 11.3K · ◷

• expires on Jun 29, 2026, 1:14 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #41869.