[experiments] Daily Experiment Report — 2026-06-16

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-16

28 experiments across 25 workflows: 6 reached min_samples, 1 pre-significant (p<0.05 but needs more data). Recommendations — ✅ PROMOTE: 0 · 🟡 EXTEND: 23 · ❌ ABANDON: 5

⚡ Quick Stats

Metric	Value
Active experiments	28
All variants at min_samples	6
Pre-significant (p<0.05, not yet at min_samples)	1 ⚠️
PROMOTE / EXTEND / ABANDON	0 / 23 / 5

---

⚠️ Notable Signal: subagent_model · smoke-copilot-aoai-apikey

Status: 🟡 COLLECTING · p = 0.0034 ** · runs: small=20/30, large=19/30

small sub-agent: 80.0% success vs large: 26.7% success — 53 pp gap. Highly significant but min_samples=30 not yet reached.

Variant	n	Succ%	Progress
small	20	80.0%	█████░░░ 20/30 (67%)
large	19	26.7%	█████░░░ 19/30 (63%)

Recommendation: 🟡 EXTEND — Strong signal; collect 30 runs per variant before concluding.

---

❌ Concluded Experiments (ABANDON — no detectable effect)

caveman · smoke-copilot (n=157, p=0.8928)

yes (36%) vs no (33%)

Recommendation: ❌ ABANDON — No statistically significant difference detected (p=0.8928) with all variants at min_samples — no detectable effect.

---

subagent_model · smoke-copilot (n=137, p=0.5176)

small (40%) vs large (29%)

Recommendation: ❌ ABANDON — No statistically significant difference detected (p=0.5176) with all variants at min_samples — no detectable effect.

---

sub_agent_strategy · smoke-gemini (n=120, p=0.2000)

single_agent (0%) vs sub_agents (0%)

Recommendation: ❌ ABANDON — No statistically significant difference detected (p=0.2000) with all variants at min_samples — no detectable effect.

---

sub_agent_strategy · smoke-antigravity (n=91, p=0.3825)

single_agent (93%) vs sub_agents (100%)

Recommendation: ❌ ABANDON — No statistically significant difference detected (p=0.3825) with all variants at min_samples — no detectable effect.

---

prompt_style · daily-astrostylelite-markdown-spellcheck (n=44, p=0.3414)

detailed (100%) vs concise (94%)

Recommendation: ❌ ABANDON — No statistically significant difference detected (p=0.3414) with all variants at min_samples — no detectable effect.

---

⚠️ Alert: smoke-gemini shows 0% success on both variants (120 runs). The workflow itself is broken regardless of variant — investigate the workflow independently.

---

🟡 Active Experiments (EXTEND — Collecting Data)

View all 23 active experiments

Experiment	Workflow	Runs	Min	p-value	Lead variant
tone_variant	aw-failure-investigator	62	50	0.2000	clinical (100%)
prompt_style	daily-community-attribution	42	20	N/A	concise (93%)
output_format	daily-issues-report	41	30	N/A	collapsible (0%)
subagent_model	smoke-copilot-aoai-apikey	39	30	0.0034 **	small (80%)
caveman	smoke-copilot-aoai-apikey	39	30	0.0631	no (71%)
reasoning_depth	daily-security-red-team	35	30	0.0941	iterative (100%)
prompt_style	daily-news	33	20	0.2000	detailed (0%)
output_format	daily-compiler-quality	31	20	0.3757	detailed (82%)
output_format	deep-report	30	15	0.1360	full_briefing (100%)
semgrep_output_format	daily-semgrep-scan	30	30	0.2825	structured_sections (100%)
output_format	daily-code-metrics	30	20	0.8227	full_detail (88%)
sub_agent_strategy	agent-persona-explorer	27	14	0.2899	per_scenario (100%)
prompt_compression	agent-performance-analyzer	27	14	0.0965	verbose (100%)
tool_verbosity	gpclean	23	30	0.2000	full_bash (100%)
reasoning_depth	daily-fact	23	20	0.4085	single_pass (53%)
prompt_style	ci-coach	22	20	0.7462	detailed (57%)
tone_style	typist	15	10	0.2049	conversational (100%)
subagent_model	smoke-copilot-aoai-entra	14	30	0.5930	small (57%)
caveman	smoke-copilot-aoai-entra	14	30	0.5930	no (57%)
summary_detail	dependabot-campaign	9	30	0.2000	detailed (100%)
prompt_style	issue-arborist	4	30	0.2000	concise (0%)
prefetch_strategy	weekly-blog-post-writer	1	10	N/A	lazy (100%)
sub_agent_strategy	architecture-guardian	1	30	N/A	sub_agents (100%)

---

Analysis window: last 30 runs per workflow · Significance threshold: p < 0.05 (two-tailed z-test)
References: §27608458671

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 2.1K AIC · ⌖ 35 AIC · ⊞ 22.8K · ◷

• expires on Jun 19, 2026, 2:07 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #39756.