[experiments] Daily Experiment Report — 2026-06-17

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-17

41 experiments · 38 workflows · 1,396 runs · 🟢 READY: 8 · 🟡 EXTEND: 33 · ❌ ABANDON: 0

⚠️ Outcome metrics (success rate, significance) require per-run data. Balance tests confirm allocation uniformity only.

Metric	Value
Workflows	38
🟢 READY	8

---

🟢 Ready for Analysis

caveman · smoke-copilot · 164 runs · balance: ✅ (χ2=0.000, p=1.0000)

Variants: no n=82, yes n=82

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

subagent_model · smoke-copilot · 144 runs · balance: ✅ (χ2=0.000, p=1.0000)

Variants: large n=72, small n=72

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

sub_agent_strategy · smoke-gemini · 127 runs · balance: ⚠️ IMBALANCED (χ2=5.740, p=0.0166)

Variants: single_agent n=50, sub_agents n=77
⚠️ sub_agents overrepresented at 60.6% (expected ~50%)

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples; allocation appears imbalanced but analysis can proceed with caution

sub_agent_strategy · smoke-antigravity · 98 runs · balance: ✅ (χ2=0.041, p=0.8399)

Variants: single_agent n=50, sub_agents n=48

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

caveman · smoke-copilot-aoai-apikey · 46 runs · balance: ✅ (χ2=0.000, p=1.0000)

Variants: yes n=23, no n=23

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

subagent_model · smoke-copilot-aoai-apikey · 46 runs · balance: ✅ (χ2=0.000, p=1.0000)

Variants: small n=23, large n=23

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

sub_agent_decomposition · smoke-pi · 76 runs · balance: ✅ (χ2=0.842, p=0.3588)

Variants: single_agent n=34, parallel_sub_agents n=42

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

prompt_style · daily-community-attribution · 43 runs · balance: ✅ (χ2=0.023, p=0.8788)

Variants: concise n=22, verbose n=21

Recommendation: READY_FOR_ANALYSIS — All variants reached minimum samples and allocation appears balanced

---

🟡 EXTEND — Experiments with Tracking Issues

• tone_variant · aw-failure-investigator · #36105 · 66 runs
  Slowest: assertive ██░░░░ 17/50 (34%)
  
• output_format · daily-issues-report · #30573 · 42 runs
  Slowest: inline ████░░ 20/30 (66%)
• reasoning_depth · daily-security-red-team · #31673 · 36 runs
  Slowest: single_pass ███░░░ 17/30 (56%)
• prompt_style · daily-news · #31190 · 34 runs
  Slowest: concise ███░░░ 13/30 (43%)
• output_format · daily-compiler-quality · #32390 · 32 runs
  Slowest: detailed ████░░ 13/20 (65%)
• semgrep_output_format · daily-semgrep-scan · #32795 · 31 runs
  Slowest: bullet_list ██░░░░ 10/30 (33%)
• output_format · daily-code-metrics · #1 · 31 runs
  Slowest: executive_summary ████░░ 15/20 (75%)
• prompt_compression · agent-performance-analyzer · #33280 · 28 runs
  Slowest: verbose █████░ 11/14 (78%)
• reasoning_depth · daily-fact · #31324 · 24 runs
  Slowest: multi_candidate █░░░░░ 7/30 (23%)
• prompt_style · ci-coach · #32335 · 23 runs
  Slowest: concise ███░░░ 9/20 (45%)
• tone_style · typist · #34032 · 16 runs
  Slowest: formal ████░░ 7/10 (70%)
• summary_detail · dependabot-campaign · #37533 · 10 runs
  Slowest: brief █░░░░░ 5/30 (16%)
• prompt_style · issue-arborist · #30015 · 4 runs
  Slowest: concise ░░░░░░ 2/30 (6%)
• detail_level · daily-architecture-diagram · #31926 · 3 runs
  Slowest: comprehensive ░░░░░░ 0/10 (0%)
• sub_agent_strategy · architecture-guardian · #39062 · 2 runs
  Slowest: sub_agents ░░░░░░ 1/30 (3%)
• prefetch_strategy · weekly-blog-post-writer · #38590 · 1 runs
  Slowest: eager ░░░░░░ 0/10 (0%)
• caveman_mode · dataflow-pr-discussion-dataset · #37102 · 1 runs
  Slowest: no ░░░░░░ 0/10 (0%)

All 33 EXTEND experiments (compact)

Experiment	Workflow	Runs	Slowest	Issue
tone_variant	aw-failure-investiga	66	17/50	#36105
prompt_style	daily-astrostylelite	45	22/30	—
output_format	daily-issues-report	42	20/30	#30573
caveman	smoke-copilot-aoai-e	20	10/20	—
subagent_model	smoke-copilot-aoai-e	20	10/20	—
reasoning_depth	daily-security-red-t	36	17/30	#31673
prompt_style	daily-news	34	13/30	#31190
output_format	daily-compiler-quali	32	13/20	#32390
semgrep_output_format	daily-semgrep-scan	31	10/30	#32795
output_format	daily-code-metrics	31	15/20	#1
output_format	deep-report	31	10/15	—
sub_agent_strategy	agent-persona-explor	28	6/14	—
prompt_compression	agent-performance-an	28	11/14	#33280
tool_verbosity	gpclean	24	10/30	—
reasoning_depth	daily-fact	24	7/30	#31324
prompt_style	ci-coach	23	9/20	#32335
tone_style	typist	16	7/10	#34032
model_size	daily-doc-healer	13	1/20	—
model_size	daily-caveman-optimi	13	1/20	—
sub_agent_strategy	daily-agentrx-trace-	12	3/20	—
model_size	daily-cache-strategy	12	1/20	—
model_size	daily-function-namer	12	2/20	—
model_size	daily-doc-updater	12	1/20	—
summary_detail	dependabot-campaign	10	5/30	#37533
output_format	copilot-agent-analys	9	4/30	—
prompt_style	dependabot-go-checke	8	2/30	—
log_fetch_strategy	daily-safe-output-op	5	2/20	—
timeout_setting	dailysubagentoptimiz	4	1/20	—
prompt_style	issue-arborist	4	2/30	#30015
detail_level	daily-architecture-d	3	0/10	#31926
sub_agent_strategy	architecture-guardia	2	1/30	#39062
prefetch_strategy	weekly-blog-post-wri	1	0/10	#38590
caveman_mode	dataflow-pr-discussi	1	0/10	#37102

---

Balance: chi-square goodness-of-fit (α=0.05)
Run: §27679037491

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · ◷

• expires on Jun 20, 2026, 1:52 AM UTC-08:00