[experiments] Daily Experiment Report — 2026-07-01

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-07-01

37 experiments across 34 workflows. 4 reached significance (p<0.05).

⚡ Quick Stats

Metric	Value
Active experiments	37
Workflows	34
Significant (p<0.05)	4
PROMOTE / ABANDON / EXTEND	0 / 14 / 23

---

📊 Featured Experiments

tone_variant · aw-failure-investigator

🟡 EXTEND | total_runs=121 | min_samples=50 | p=N/A

📋 Statistics Table

Exp: tone_variant  |  Workflow: aw-failure-investigator
min_samples: 50/variant  |  Window: last 30 runs  |  Total: 121

+----------------+----+--------+----------+--------------+----------+--------------+
| Variant        | n  | Succ%  | Dur(s)   | 95% CI(s)    | p-value  | Progress     |
+----------------+----+--------+----------+--------------+----------+--------------+
| clinical       | 12 | 100.0% |    761.7 | [639.5,883.9] | (ref)    | 48/50 (96%)  |
| narrative      | 10 | 100.0% |    718.4 | [611.5,825.3] | N/A      | 36/50 (72%)  |
| assertive      | 8  | 100.0% |    785.2 | [684.1,886.4] | N/A      | 37/50 (74%)  |
+----------------+----+--------+----------+--------------+----------+--------------+
Recommendation: EXTEND  |  Collecting data — 29 more runs needed to reach min_samples=50 for all variants.

---

caveman · smoke-copilot

❌ ABANDON | total_runs=224 | min_samples=20 | p=1.0000

📋 Statistics Table

Exp: caveman  |  Workflow: smoke-copilot
min_samples: 20/variant  |  Window: last 30 runs  |  Total: 224

+----------------+----+--------+----------+--------------+----------+--------------+
| Variant        | n  | Succ%  | Dur(s)   | 95% CI(s)    | p-value  | Progress     |
+----------------+----+--------+----------+--------------+----------+--------------+
| no             | 15 |  80.0% |    813.9 | [746.6,881.2] | (ref)    | 112/20 (100%) |
| yes            | 15 |  80.0% |    823.4 | [749.7,897.1] | 1.0000   | 112/20 (100%) |
+----------------+----+--------+----------+--------------+----------+--------------+
Recommendation: ABANDON  |  No statistically significant effect detected after reaching min_samples (p=1.0000).

---

model_size · daily-cache-strategy-analyzer

🟡 EXTEND | total_runs=26 | min_samples=20 | p=0.0001

📋 Statistics Table

Exp: model_size  |  Workflow: daily-cache-strategy-analyzer
min_samples: 20/variant  |  Window: last 30 runs  |  Total: 26

+----------------+----+--------+----------+--------------+----------+--------------+
| Variant        | n  | Succ%  | Dur(s)   | 95% CI(s)    | p-value  | Progress     |
+----------------+----+--------+----------+--------------+----------+--------------+
| gpt-5.4        | 8  | 100.0% |    753.0 | [671.1,834.9] | (ref)    | 8/20 (40%)   |
| small-agent    | 1  |   0.0% |    336.0 | N/A          | 0.0027 ** | 1/20 (5%)    |
| agent          | 1  |   0.0% |    316.0 | N/A          | 0.0027 ** | 1/20 (5%)    |
| gpt-5.4-mini   | 6  |   0.0% |    422.2 | [351.3,493.0] | 0.0002 *** | 6/20 (30%)   |
| gpt-5-mini     | 2  |   0.0% |    378.0 | [-371.7,1127.7] | 0.0016 ** | 2/20 (10%)   |
| gpt-5-codex    | 8  |   0.0% |    342.8 | [300.8,384.7] | 0.0001 *** | 8/20 (40%)   |
+----------------+----+--------+----------+--------------+----------+--------------+
Recommendation: EXTEND  |  Collecting data — 94 more runs needed to reach min_samples=20 for all variants.

---

tone_style · typist

❌ ABANDON | total_runs=27 | min_samples=10 | p=0.0767

📋 Statistics Table

Exp: tone_style  |  Workflow: typist
min_samples: 10/variant  |  Window: last 30 runs  |  Total: 27

+------------------+----+--------+----------+--------------+----------+--------------+
| Variant          | n  | Succ%  | Dur(s)   | 95% CI(s)    | p-value  | Progress     |
+------------------+----+--------+----------+--------------+----------+--------------+
| formal           | 14 |  78.6% |    885.6 | [733.3,1038.0] | (ref)    | 14/10 (100%) |
| conversational   | 13 | 100.0% |    943.4 | [779.7,1107.1] | 0.0767   | 13/10 (100%) |
+------------------+----+--------+----------+--------------+----------+--------------+
Recommendation: ABANDON  |  No statistically significant effect detected after reaching min_samples (p=0.0767).

---

🔴 ABANDON Experiments

Experiment	Workflow	Total Runs	p-value	Reason
caveman	smoke-copilot	224	1.0000	No effect
subagent_model	smoke-copilot	204	1.0000	No effect
sub_agent_strategy	smoke-gemini	180	N/A	No effect
sub_agent_strategy	smoke-antigravity	151	0.1709	No effect
caveman	smoke-copilot-aoai-apikey	99	0.3613	No effect
subagent_model	smoke-copilot-aoai-apikey	99	0.4642	No effect
caveman	smoke-copilot-aoai-entra	71	1.0000	No effect
subagent_model	smoke-copilot-aoai-entra	71	0.2827	No effect
prompt_style	daily-community-attribution	57	0.1709	No effect
output_format	daily-compiler-quality	46	0.6660	No effect
output_format	daily-code-metrics	45	0.3203	No effect
sub_agent_strategy	agent-persona-explorer	42	N/A	No effect
prompt_compression	agent-performance-analyzer	41	0.4720	No effect
tone_style	typist	27	0.0767	No effect

🟡 EXTEND Experiments — Still Collecting

Experiment	Workflow	Total Runs	min_samples	Worst %
tone_variant	aw-failure-investigator	121	50	72%
prompt_style	daily-astrostylelite-markdown-spellcheck	59	30	93%
output_format	daily-issues-report	56	30	90%
reasoning_depth	daily-security-red-team	50	30	70%
prompt_style	daily-news	49	30	80%
semgrep_output_format	daily-semgrep-scan	45	30	40%
output_format	deep-report	41	15	80%
tool_verbosity	gpclean	38	30	56%
reasoning_depth	daily-fact	34	30	43%
prompt_style	ci-coach	33	20	80%
model_size	daily-caveman-optimizer	27	20	5%
model_size	daily-doc-healer	27	20	5%
model_size	daily-doc-updater	27	20	5%
model_size	daily-cache-strategy-analyzer	26	20	5%
output_format	copilot-agent-analysis	23	30	36%
sub_agent_strategy	daily-agentrx-trace-optimizer	22	20	50%
log_fetch_strategy	daily-safe-output-optimizer	16	20	40%
prompt_style	dependabot-go-checker	14	30	10%
sub_agent_strategy	architecture-guardian	10	30	10%
detail_level	daily-architecture-diagram	5	10	10%
prompt_style	issue-arborist	4	30	6%
caveman_mode	dataflow-pr-discussion-dataset	3	10	10%
prefetch_strategy	weekly-blog-post-writer	3	10	30%

Analysis window: last 30 runs · p<0.05 threshold · Run: §28506199806

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

• awmgmcpg
• proxy.golang.org
• releaseassets.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"
    - "proxy.golang.org"
    - "releaseassets.githubusercontent.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · 268.2 AIC · ⌖ 27.4 AIC · ⊞ 2.2K · ◷

• expires on Jul 4, 2026, 1:32 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #43154.