[experiments] Daily Experiment Report — 2026-06-21

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-21

38 workflows · 41 A/B experiments · 11 READY (🟢 min_samples reached) · 30 EXTEND (🟡 collecting) · 4 imbalanced (p < 0.05).

⚠️ Per-run outcome metrics (success rate, duration, tokens) not available — GitHub API auth required. PROMOTE/ABANDON deferred; balance and readiness reported from state.json counts only.

---

⚡ Quick Stats

Metric	Value
Active experiment workflows	38
Total A/B experiments	41
🟢 READY_FOR_ANALYSIS	11
🟡 EXTEND (collecting)	30
⚠️ Imbalanced (balance p < 0.05)	4
Tracking issues configured	13

---

🟢 READY FOR ANALYSIS (11 experiments)

smoke-copilot — caveman × subagent_model (factorial)

Both experiments perfectly balanced (p=1.000). 192 runs (caveman) / 172 runs (subagent_model).

Details

Exp	Variant	n	Progress
caveman	no (ctrl)	96	██████████ 480% ✓
caveman	yes	96	██████████ 480% ✓
subagent_model	large (ctrl)	86	██████████ 430% ✓
subagent_model	small	86	██████████ 430% ✓

Recommendation: 🟢 READY — Outcome metrics needed for PROMOTE/ABANDON.

---

smoke-gemini — sub_agent_strategy ⚠️ IMBALANCED

Hypothesis: sub_agents reduces effective_tokens by ≥20%

Details

Variant	n	Obs%	Balance
sub_agents (ctrl)	92	61.7%	p=0.004 ⚠️
single_agent	57	38.3%	—

chi2=8.22, df=1, p=0.0041 (IMBALANCED) — expected 50/50, got 62/38. Check weight: config.

Recommendation: ⚠️ INVESTIGATE BALANCE before outcome analysis.

---

smoke-antigravity — sub_agent_strategy

Hypothesis: sub_agents reduces effective_tokens by ≥20%

Variant	n	Progress
single_agent (ctrl)	57	██████████ 190% ✓
sub_agents	63	██████████ 210% ✓

Balance: p=0.584 ✅ · Recommendation: 🟢 READY

---

Additional READY experiments

Experiment	Workflow	Variants (n min)	Balance p	Rec
prompt_style	daily-astrostylelite-spellcheck	detailed=25, concise=24	0.886 ✅	🟢
prompt_style	daily-community-attribution	concise=24, verbose=23	0.884 ✅	🟢
sub_agent_decomposition	smoke-pi	single=34, parallel=42	0.359 ✅	🟢
caveman	smoke-copilot-aoai-apikey	yes=34, no=34	1.000 ✅	🟢
subagent_model	smoke-copilot-aoai-apikey	small=34, large=34	1.000 ✅	🟢
caveman	smoke-copilot-aoai-entra	no=20, yes=20	1.000 ✅	🟢
subagent_model	smoke-copilot-aoai-entra	large=20, small=20	1.000 ✅	🟢

---

🟡 EXTEND (30 experiments)

View all 30 EXTEND experiments

Experiment	Workflow	Min-variant count	min_samples	Progress
tone_variant	aw-failure-investigator	narrative=24	50	████░░░░░░ 48%
prompt_compression	agent-performance-analyzer	verbose=12	14	████████░░ 85%
sub_agent_strategy	agent-persona-explorer ⚠️	per_scenario=10	14	███████░░░ 71%
prompt_style	ci-coach	concise=11	20	█████░░░░░ 55%
output_format	daily-code-metrics	exec_summary=16	20	████████░░ 80%
output_format	daily-compiler-quality	detailed=15	20	███████░░░ 75%
reasoning_depth	daily-fact	multi_candidate=9	30	███░░░░░░░ 30%
output_format	daily-issues-report	collapsible=22	30	███████░░░ 73%
prompt_style	daily-news	concise=18	20	█████████░ 90%
reasoning_depth	daily-security-red-team	single_pass=18	30	██████░░░░ 60%
semgrep_output_format	daily-semgrep-scan	bullet_list=10	30	███░░░░░░░ 33%
output_format	deep-report	full_briefing=11	15	███████░░░ 73%
tool_verbosity	gpclean	full_bash=12	30	████░░░░░░ 40%
tone_style	typist	formal=8	20	████░░░░░░ 40%
+ 16 more	various	< 10 runs	20	early stage

⚠️ agent-persona-explorer/sub_agent_strategy also imbalanced (p=0.034).

---

Analysis: state.json from 38 experiments/* branches · Balance test: Pearson chi-square · p < 0.05 threshold · Outcome metrics deferred (API auth required)
Run: 27899547499

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 250.4 AIC · ⌖ 12.8 AIC · ⊞ 11.2K · ◷

• expires on Jun 24, 2026, 1:32 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #40763.