[experiments] Daily Experiment Report — 2026-06-12

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-06-12

23 experiments analysed across 21 workflows (5 ABANDON · 3 need outcome data · 15 EXTEND). No experiments reached statistical significance (p < 0.05). 2 workflows have critical failures and require immediate attention.

⚡ Quick Stats

Metric	Value
Active experiments	23
Ready for analysis	3
Statistically significant (p < 0.05)	0
Recommendations	❌ ABANDON: 5 · 🟡 EXTEND: 15 · 🔬 READY: 3

🚨 Critical Alerts

• daily-issues-report / output_format — 0% success rate across all variants (all 30 measured runs failed — workflow appears broken)
• daily-compiler-quality / output_format — detailed sr=77.8% < run_success_rate threshold >=0.85
• daily-news / prompt_style — 0% success rate across all variants (all 30 measured runs failed — workflow appears broken)

---

❌ ABANDON Recommendations

❌ smoke-copilot / caveman

Variant	n	Success%	Outcome data
no	72	43.8%	16 runs
yes	72	58.8%	17 runs
Rationale: No significant difference (best p=0.3865); no evidence to promote any variant — p = 0.3865

❌ smoke-copilot / subagent_model

Variant	n	Success%	Outcome data
large	62	33.3%	6 runs
small	62	28.6%	7 runs
Rationale: No significant difference (best p=0.8529); no evidence to promote any variant — p = 0.8529

🚨 daily-issues-report / output_format

H0: no change in discussion engagement score. H1: inline format produces ≥20% higher reactions+replies by making charts and recommendations immediately visible
| Variant | n | Success% | Outcome data |
|---------|---|---------|-------------|
| collapsible | 19 | 0.0% | 17 runs |
| inline | 17 | 0.0% | 13 runs |
Rationale: ⚠️ 0% success rate across all variants (all 30 measured runs failed — workflow appears broken)

🚨 daily-compiler-quality / output_format

H0: no change in discussion engagement. H1: concise variant achieves equal engagement with ≥30% fewer output tokens
| Variant | n | Success% | Outcome data |
|---------|---|---------|-------------|
| detailed | 9 | 77.8% | 9 runs |
| concise | 18 | 61.1% | 18 runs |
Rationale: ⚠️ detailed sr=77.8% < run_success_rate threshold >=0.85

🚨 daily-news / prompt_style

H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score
| Variant | n | Success% | Outcome data |
|---------|---|---------|-------------|
| detailed | 20 | 0.0% | 19 runs |
| concise | 11 | 0.0% | 11 runs |
Rationale: ⚠️ 0% success rate across all variants (all 30 measured runs failed — workflow appears broken)

📊 smoke-copilot Factorial Interaction (caveman × subagent_model)

Chi-square independence test p = 0.157 — no significant interaction detected.
However, SPARSE_CELL_RISK (cells < 20 runs): do not promote based on main effects alone.

---

🔬 Ready for Analysis (Needs Outcome Data)

These experiments have enough variant runs but require more outcome metrics to compute significance.

🔬 smoke-gemini / sub_agent_strategy

H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%
| Variant | n | Success% (limited data) |
|---------|---|------------------------|
| sub_agents | 68 | 100.0% (5 runs) |
| single_agent | 39 | 100.0% (4 runs) |
Balance: ⚠️ imbalanced (χ2=7.86, p=0.005)

🔬 smoke-antigravity / sub_agent_strategy

H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%
| Variant | n | Success% (limited data) |
|---------|---|------------------------|
| single_agent | 38 | 100.0% (3 runs) |
| sub_agents | 40 | 100.0% (6 runs) |
Balance: ✅ balanced

🔬 smoke-pi / sub_agent_decomposition

Variant	n	Success% (limited data)
single_agent	34	100.0% (4 runs)
parallel_sub_agents	42	100.0% (5 runs)
Balance: ✅ balanced

📈 View ready-experiment charts

---

🟡 Collecting Data (EXTEND)

Notable experiments near significance:

• daily-community-attribution / prompt_style — concise 94.7% (n=19) vs verbose 84.2% (n=19), p = 0.2904
• daily-security-red-team / reasoning_depth — iterative 100.0% (n=17) vs single_pass 78.6% (n=14), p = 0.0510
• daily-code-metrics / output_format — executive_summary 84.6% (n=13) vs full_detail 84.6% (n=13), p = 1.0000

View all 15 EXTEND experiments

Workflow	Experiment	Variants	Progress
ci-coach #32335	prompt_style	concise vs detailed	██░░░░ 7/20 · ████░░ 13/20
agent-performance-analyzer #33280	prompt_compression	caveman vs verbose	██████ 14/14 · ████░░ 9/14
agent-persona-explorer	sub_agent_strategy	batch vs per_scenario	██████ 17/14 · ███░░░ 6/14
aw-failure-investigator #36105	tone_variant	clinical vs narrative vs assertive	██░░░░ 19/50 · ██░░░░ 14/50 · ██░░░░ 13/50
daily-astrostylelite-markdown-spellcheck	prompt_style	detailed vs concise	████░░ 20/30 · ████░░ 20/30
daily-community-attribution	prompt_style	concise vs verbose	██████ 19/20 · ██████ 19/20
daily-security-red-team #31673	reasoning_depth	iterative vs single_pass	███░░░ 17/30 · ███░░░ 14/30
daily-code-metrics #1	output_format	executive_summary vs full_detail	████░░ 13/20 · ████░░ 13/20
typist #34032	tone_style	conversational vs formal	████░░ 7/10 · ████░░ 6/10
deep-report	output_format	executive_brief vs full_briefing vs annotated_brief	████░░ 9/15 · ████░░ 10/15 · ████░░ 9/15
daily-semgrep-scan #32795	semgrep_output_format	structured_sections vs prose vs bullet_list	██░░░░ 9/30 · ██░░░░ 8/30 · ██░░░░ 9/30
gpclean	tool_verbosity	full_bash vs minimal_toolset	██░░░░ 8/30 · ██░░░░ 11/30
smoke-copilot-aoai-apikey	caveman	yes vs no	████░░ 13/20 · ████░░ 13/20
smoke-copilot-aoai-apikey	subagent_model	small vs large	████░░ 13/20 · ████░░ 13/20
daily-fact #31324	reasoning_depth	single_pass vs multi_candidate	███░░░ 15/30 · █░░░░░ 6/30

---

References: §27406359516

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• proxy.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "proxy.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 3K AIC · ⌖ 32.7 AIC · ⊞ 23.2K · ◷

• expires on Jun 15, 2026, 1:47 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #39042.