[experiments] Daily Experiment Report — 2026-06-19

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-19

38 experiments across 35 workflows · 6 READY · 32 collecting data

⚠️ Outcome metrics unavailable — GitHub Actions API rate limit exhausted. Stats from assignment counts only.

⚡ Quick Stats

Metric	Value
Total experiments	38
Workflows	35
🟢 Ready for analysis	6
🟡 Collecting data	32
Outcome significance	N/A
Recommendations	🟡 EXTEND all (pending outcome data)

---

🟢 Ready for Analysis

Experiment	Workflow	Variants	Balance	Status
sub_agent_strategy	smoke-antigravity	single_agent n=55, sub_agents n=60	✅ p=0.641	🟢 READY
caveman	smoke-copilot-aoai-apikey	yes n=32, no n=31	✅ p=0.900	🟢 READY
subagent_model	smoke-copilot-aoai-apikey	small n=31, large n=32	✅ p=0.900	🟢 READY
caveman	smoke-copilot	yes n=93, no n=93	✅ p=1.000	🟢 READY
subagent_model	smoke-copilot	small n=83, large n=83	✅ p=1.000	🟢 READY
sub_agent_strategy	smoke-gemini	single_agent n=54, sub_agents n=90	⚠️ p=0.003	🟢 READY

📊 sub_agent_strategy · smoke-antigravity

Hypothesis: H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%
min_samples: 30 · Total runs: 115

Variant	n	Progress
single_agent	55	██████████ 55/30 (100%)
sub_agents	60	██████████ 60/30 (100%)

Recommendation: EXTEND — Outcome metrics needed for PROMOTE/ABANDON.

📊 sub_agent_strategy · smoke-gemini

Hypothesis: H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%
min_samples: 30 · Total runs: 144

Variant	n	Progress
single_agent	54	██████████ 54/30 (100%)
sub_agents	90	██████████ 90/30 (100%)

Recommendation: EXTEND — Outcome metrics needed for PROMOTE/ABANDON.

📊 caveman · smoke-copilot

Hypothesis:
min_samples: 30 · Total runs: 186

Variant	n	Progress
yes	93	██████████ 93/30 (100%)
no	93	██████████ 93/30 (100%)

Recommendation: EXTEND — Outcome metrics needed for PROMOTE/ABANDON.

📊 subagent_model · smoke-copilot

Hypothesis:
min_samples: 30 · Total runs: 166

Variant	n	Progress
small	83	██████████ 83/30 (100%)
large	83	██████████ 83/30 (100%)

Recommendation: EXTEND — Outcome metrics needed for PROMOTE/ABANDON.

---

🟡 Collecting Data (EXTEND)

Near-ready (≥ 70% progress):

Experiment	Workflow	Best variant	Min progress
prompt_compression	agent-performance-an	caveman n=18	█████████░ 12/14 (86%)
output_format	daily-code-metrics	full_detail n=17	████████░░ 16/20 (80%)
tone_style	typist	conversational n=10	████████░░ 8/10 (80%)
output_format	daily-compiler-quali	concise n=19	████████░░ 15/20 (75%)
prompt_style	daily-community-attr	concise n=23	███████░░░ 22/30 (73%)
output_format	daily-issues-report	collapsible n=22	███████░░░ 22/30 (73%)
output_format	deep-report	full_briefing n=11	███████░░░ 11/15 (73%)

View all EXTEND experiments

Experiment	Workflow	Total	min_samples	Progress
tone_variant	aw-failure-investiga	74	50	████░░░░░░ 20/50 (40%)
prompt_style	daily-community-attr	45	30	███████░░░ 22/30 (73%)
output_format	daily-issues-report	44	30	███████░░░ 22/30 (73%)
prompt_style	daily-news	41	30	██████░░░░ 17/30 (57%)
reasoning_depth	daily-security-red-t	38	30	██████░░░░ 17/30 (57%)
caveman	smoke-copilot-aoai-e	36	30	██████░░░░ 18/30 (60%)
subagent_model	smoke-copilot-aoai-e	36	30	██████░░░░ 18/30 (60%)
output_format	daily-compiler-quali	34	20	████████░░ 15/20 (75%)
output_format	daily-code-metrics	33	20	████████░░ 16/20 (80%)
semgrep_output_forma	daily-semgrep-scan	33	30	███░░░░░░░ 10/30 (33%)
output_format	deep-report	33	15	███████░░░ 11/15 (73%)
prompt_compression	agent-performance-an	30	14	█████████░ 12/14 (86%)
sub_agent_strategy	agent-persona-explor	30	14	██████░░░░ 8/14 (57%)
reasoning_depth	daily-fact	26	30	███░░░░░░░ 9/30 (30%)
tool_verbosity	gpclean	26	30	███░░░░░░░ 10/30 (33%)
prompt_style	ci-coach	25	20	█████░░░░░ 10/20 (50%)
tone_style	typist	18	10	████████░░ 8/10 (80%)
model_size	daily-caveman-optimi	15	20	░░░░░░░░░░ 1/20 (5%)
model_size	daily-doc-healer	15	20	░░░░░░░░░░ 1/20 (5%)
model_size	daily-doc-updater	15	20	░░░░░░░░░░ 1/20 (5%)
sub_agent_strategy	daily-agentrx-trace-	14	20	██░░░░░░░░ 4/20 (20%)
model_size	daily-cache-strategy	14	20	░░░░░░░░░░ 1/20 (5%)
summary_detail	dependabot-campaign	12	30	██░░░░░░░░ 6/30 (20%)
output_format	copilot-agent-analys	11	30	█░░░░░░░░░ 4/30 (13%)
prompt_style	dependabot-go-checke	9	30	█░░░░░░░░░ 2/30 (7%)
log_fetch_strategy	daily-safe-output-op	6	20	██░░░░░░░░ 3/20 (15%)
sub_agent_strategy	architecture-guardia	4	30	░░░░░░░░░░ 1/30 (3%)
prompt_style	issue-arborist	4	30	█░░░░░░░░░ 2/30 (7%)
detail_level	daily-architecture-d	3	10	░░░░░░░░░░ 0/10 (0%)
model_size	daily-function-namer	2	20	░░░░░░░░░░ 0/20 (0%)
caveman_mode	dataflow-pr-discussi	2	10	█░░░░░░░░░ 1/10 (10%)
prefetch_strategy	weekly-blog-post-wri	1	10	░░░░░░░░░░ 0/10 (0%)

---

Analysis: experiment state branches · Significance threshold: p < 0.05
Run: 27817198841

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 318.3 AIC · ⌖ 9.7 AIC · ⊞ 11.2K · ◷

• expires on Jun 22, 2026, 1:47 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #40603.