[experiments] Daily Experiment Report — 2026-09-09

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-09

51 experiments analysed across 48 workflows — 22 have reached min_samples and are ready-for-analysis; all currently resolve to EXTEND or INCONCLUSIVE because outcome-metric observations (token_count_per_run, run_success_rate, etc.) are not yet wired into the decision pipeline. No PROMOTE/REJECT decisions today.

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis	22
Decisions with sufficient evidence	0
Core decisions	🟡 EXTEND: 37 · ⚪ INCONCLUSIVE: 14 · ✅ PROMOTE: 0 · ❌ REJECT: 0
Interaction safety holds	0

---

prompt_compression · agent-performance-analyzer.md

Readiness: 🟢 READY
Variants: caveman vs verbose · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric effective_tokens not yet observed

H0: no change in effective_tokens. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
caveman	60	██████████ 60/20 (100%)
verbose	51	██████████ 51/20 (100%)

Experiment : prompt_compression
Workflow   : agent-performance-analyzer.md
Hypothesis : H0: no change in effective_tokens. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| caveman          |    6 |   100.0% |          835.8 |    [769.5 , 902.2] | ██████████ 60/20 (100%) |
| verbose          |    4 |   100.0% |          731.8 |    [549.6 , 913.9] | ██████████ 51/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.3975 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_strategy · agent-persona-explorer.md

Readiness: 🟢 READY
Variants: batch vs per_scenario · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric effective_tokens not yet observed

H0: no change in effective_tokens or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
batch	58	██████████ 58/20 (100%)
per_scenario	53	██████████ 53/20 (100%)

Experiment : sub_agent_strategy
Workflow   : agent-persona-explorer.md
Hypothesis : H0: no change in effective_tokens or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| batch            |    4 |    75.0% |          578.8 |    [346.8 , 810.7] | ██████████ 58/20 (100%) |
| per_scenario     |    6 |    50.0% |          486.3 |    [400.9 , 571.7] | ██████████ 53/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.6403 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_variant · aw-failure-investigator.md

Readiness: 🟢 READY
Variants: assertive vs clinical vs narrative · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric output_length_chars not yet observed

H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
assertive	134	██████████ 134/20 (100%)
clinical	134	██████████ 134/20 (100%)
narrative	129	██████████ 129/20 (100%)

Experiment : tone_variant
Workflow   : aw-failure-investigator.md
Hypothesis : H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| assertive        |    3 |   100.0% |          620.0 |    [472.6 , 767.4] | ██████████ 134/20 (100%) |
| clinical         |    2 |   100.0% |          578.5 |    [394.3 , 762.7] | ██████████ 134/20 (100%) |
| narrative        |    5 |   100.0% |          618.4 |    [592.7 , 644.1] | ██████████ 129/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.9296 (variant assignment balance test, not outcome significance)


Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

Core decision: INCONCLUSIVE (unsupported_multi_variant) — automatic decisions currently require exactly two variants

---

prompt_style · ci-coach.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric token_count_per_run not yet observed

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	37	██████████ 37/20 (100%)
detailed	46	██████████ 46/20 (100%)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| concise          |    5 |    80.0% |         1209.0 |   [927.8 , 1490.2] | ██████████ 37/20 (100%) |
| detailed         |    5 |    60.0% |         1112.2 |   [791.9 , 1432.5] | ██████████ 46/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0 (variant assignment balance test, not outcome significance)

Guardrails:
  run_success_rate >=0.85 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

sub_agent_strategy · daily-agentrx-trace-optimizer.md

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric issue_evidence_completeness not yet observed

H0: no change in issue quality or run success rate. H1: sub_agents variant yields higher evidence completeness score with equal or lower token cost

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_agent	36	██████████ 36/20 (100%)
sub_agents	36	██████████ 36/20 (100%)

Experiment : sub_agent_strategy
Workflow   : daily-agentrx-trace-optimizer.md
Hypothesis : H0: no change in issue quality or run success rate. H1: sub_agents variant yields higher evidence completeness score with equal or lower token cost
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| single_agent     |    5 |    40.0% |         1335.6 |   [829.7 , 1841.5] | ██████████ 36/20 (100%) |
| sub_agents       |    5 |    80.0% |         2106.8 |   [689.0 , 3524.6] | ██████████ 36/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=1 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · daily-astrostylelite-markdown-spellcheck.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric effective_tokens not yet observed

Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	65	██████████ 65/20 (100%)
detailed	64	██████████ 64/20 (100%)

Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck.md
Hypothesis : Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| concise          |    3 |   100.0% |          642.3 |   [273.0 , 1011.7] | ██████████ 65/20 (100%) |
| detailed         |    7 |   100.0% |          608.4 |    [561.0 , 655.8] | ██████████ 64/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.8907 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · daily-community-attribution.md

Readiness: 🟢 READY
Variants: concise vs verbose · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	63	██████████ 63/20 (100%)
verbose	64	██████████ 64/20 (100%)

Experiment : prompt_style
Workflow   : daily-community-attribution.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| concise          |    5 |    60.0% |          600.4 |    [459.7 , 741.1] | ██████████ 63/20 (100%) |
| verbose          |    5 |    80.0% |          654.8 |    [496.6 , 813.0] | ██████████ 64/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.8903 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

reasoning_depth · daily-fact.md

Readiness: 🟢 READY
Variants: multi_candidate vs single_pass · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Evidence: metric discussion_reaction_count not yet observed

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
multi_candidate	33	██████████ 33/30 (100%)
single_pass	51	██████████ 51/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-fact.md
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| multi_candidate  |    4 |     0.0% |          305.5 |    [227.3 , 383.7] | ██████████ 33/30 (100%) |
| single_pass      |    6 |     0.0% |          276.5 |    [212.2 , 340.8] | ██████████ 51/30 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.04678 (variant assignment balance test, not outcome significance)

Guardrails:
  empty_output_rate <0.05 : UNSUPPORTED
  run_success_rate >=0.95 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "empty_output_rate" is not backed by a supported metric observation

---

prompt_style · daily-news.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric effective_token_count not yet observed

H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	53	██████████ 53/20 (100%)
detailed	46	██████████ 46/20 (100%)

Experiment : prompt_style
Workflow   : daily-news.md
Hypothesis : H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| concise          |    5 |    80.0% |          800.4 |    [626.5 , 974.3] | ██████████ 53/20 (100%) |
| detailed         |    5 |   100.0% |          726.4 |    [658.0 , 794.8] | ██████████ 46/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.4888 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

reasoning_depth · daily-security-red-team.md

Readiness: 🟢 READY
Variants: iterative vs single_pass · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 30 per variant · Evidence: metric false_positive_rate not yet observed

H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
iterative	72	██████████ 72/30 (100%)
single_pass	48	██████████ 48/30 (100%)

Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| iterative        |    5 |    20.0% |         3163.6 |  [1322.9 , 5004.3] | ██████████ 72/30 (100%) |
| single_pass      |    5 |    60.0% |         1765.0 |  [-579.8 , 4109.8] | ██████████ 48/30 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.0269 (variant assignment balance test, not outcome significance)

Guardrails:
  run_success_rate >=0.9 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

tool_verbosity · gpclean.md

Readiness: 🟢 READY
Variants: full_bash vs minimal_toolset · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric effective_token_count not yet observed

H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
full_bash	51	██████████ 51/20 (100%)
minimal_toolset	57	██████████ 57/20 (100%)

Experiment : tool_verbosity
Workflow   : gpclean.md
Hypothesis : H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| full_bash        |    7 |   100.0% |          720.4 |    [80.5 , 1360.4] | ██████████ 51/20 (100%) |
| minimal_toolset  |    3 |   100.0% |          397.0 |    [242.4 , 551.6] | ██████████ 57/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.5711 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

prompt_style · issue-arborist.md

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric links_created not yet observed

H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	37	██████████ 37/20 (100%)
detailed	30	██████████ 30/20 (100%)

Experiment : prompt_style
Workflow   : issue-arborist.md
Hypothesis : H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| concise          |    3 |    33.3% |          405.3 |    [-69.4 , 880.1] | ██████████ 37/20 (100%) |
| detailed         |    7 |    42.9% |          429.0 |    [307.4 , 550.6] | ██████████ 30/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.3969 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_strategy · smoke-antigravity.md

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_agent	99	██████████ 99/20 (100%)
sub_agents	109	██████████ 109/20 (100%)

Experiment : sub_agent_strategy
Workflow   : smoke-antigravity.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| single_agent     |    5 |   100.0% |          616.8 |    [234.7 , 998.9] | ██████████ 99/20 (100%) |
| sub_agents       |    5 |   100.0% |          720.2 |   [214.8 , 1225.6] | ██████████ 109/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.4952 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman · smoke-copilot.md

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
no	177	██████████ 177/20 (100%)
yes	177	██████████ 177/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| no               |    5 |    80.0% |          784.6 |    [651.2 , 918.0] | ██████████ 177/20 (100%) |
| yes              |    5 |    60.0% |          736.4 |    [643.0 , 829.8] | ██████████ 177/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=1 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

subagent_model · smoke-copilot.md

Readiness: 🟢 READY
Variants: large vs small · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
large	167	██████████ 167/20 (100%)
small	167	██████████ 167/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| large            |    5 |    40.0% |          728.8 |    [606.9 , 850.7] | ██████████ 167/20 (100%) |
| small            |    5 |   100.0% |          792.2 |    [692.1 , 892.3] | ██████████ 167/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=1 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman · smoke-copilot-aoai-apikey.md

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
no	89	██████████ 89/20 (100%)
yes	90	██████████ 90/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot-aoai-apikey.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| no               |    5 |    80.0% |          921.0 |   [734.6 , 1107.4] | ██████████ 89/20 (100%) |
| yes              |    5 |    80.0% |          746.4 |   [487.7 , 1005.1] | ██████████ 90/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.8986 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

subagent_model · smoke-copilot-aoai-apikey.md

Readiness: 🟢 READY
Variants: large vs small · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
large	90	██████████ 90/20 (100%)
small	89	██████████ 89/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot-aoai-apikey.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| large            |    5 |    60.0% |          794.2 |   [477.5 , 1110.9] | ██████████ 90/20 (100%) |
| small            |    5 |   100.0% |          873.2 |   [716.0 , 1030.4] | ██████████ 89/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.8986 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

caveman · smoke-copilot-aoai-entra.md

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
no	76	██████████ 76/20 (100%)
yes	76	██████████ 76/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot-aoai-entra.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| no               |    5 |    60.0% |          909.8 |   [508.3 , 1311.3] | ██████████ 76/20 (100%) |
| yes              |    5 |    60.0% |          802.6 |   [508.8 , 1096.4] | ██████████ 76/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=1 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

subagent_model · smoke-copilot-aoai-entra.md

Readiness: 🟢 READY
Variants: large vs small · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
large	76	██████████ 76/20 (100%)
small	76	██████████ 76/20 (100%)

Experiment : subagent_model
Workflow   : smoke-copilot-aoai-entra.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| large            |    5 |    60.0% |          882.6 |   [505.5 , 1259.7] | ██████████ 76/20 (100%) |
| small            |    5 |    60.0% |          829.8 |   [492.6 , 1167.0] | ██████████ 76/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=1 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_strategy · smoke-gemini.md

Readiness: 🟢 READY
Variants: single_agent vs sub_agents · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric effective_tokens not yet observed

H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_agent	122	██████████ 122/20 (100%)
sub_agents	140	██████████ 140/20 (100%)

Experiment : sub_agent_strategy
Workflow   : smoke-gemini.md
Hypothesis : H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| single_agent     |    6 |     0.0% |          329.5 |    [271.6 , 387.4] | ██████████ 122/20 (100%) |
| sub_agents       |    4 |     0.0% |          334.0 |    [325.7 , 342.3] | ██████████ 140/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.2653 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

sub_agent_decomposition · smoke-pi.md

Readiness: 🟢 READY
Variants: parallel_sub_agents vs single_agent · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: no primary metric declared

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
parallel_sub_agents	42	██████████ 42/20 (100%)
single_agent	34	██████████ 34/20 (100%)

Experiment : sub_agent_decomposition
Workflow   : smoke-pi.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| parallel_sub_agents |    5 |   100.0% |          259.2 |    [144.0 , 374.4] | ██████████ 42/20 (100%) |
| single_agent     |    5 |   100.0% |          198.8 |    [173.4 , 224.2] | ██████████ 34/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.3619 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

tone_style · typist.md

Readiness: 🟢 READY
Variants: conversational vs formal · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: metric discussion_engagement_score not yet observed

H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
conversational	42	██████████ 42/20 (100%)
formal	34	██████████ 34/20 (100%)

Experiment : tone_style
Workflow   : typist.md
Hypothesis : H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------------+----------------+--------------------+---------------+
| Variant          |    n |   Succ % |   Mean dur (s) |         95% CI (s) |   min_samples |
+------------------+------+----------------+----------------+--------------------+---------------+
| conversational   |    6 |   100.0% |         1299.2 |  [1042.4 , 1556.0] | ██████████ 42/20 (100%) |
| formal           |    4 |    75.0% |          940.0 |   [396.1 , 1483.9] | ██████████ 34/20 (100%) |
+------------------+------+----------------+----------------+--------------------+---------------+

Evidence: chi-square balance p_value=0.3619 (variant assignment balance test, not outcome significance)


Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

Interaction diagnostics: caveman × subagent_model (smoke-copilot family)

Two experiments run concurrently in smoke-copilot, smoke-copilot-aoai-apikey, and smoke-copilot-aoai-entra. Per-run assignment cells over the last 10 tracked runs:

smoke-copilot.md — cells: (no,large)=n3, (no,small)=n2, (yes,large)=n2, (yes,small)=n3 · sparsest cell n=2 (min_samples per variant experiment = 20) → SPARSE_CELL_RISK
smoke-copilot-aoai-apikey.md — cells: (no,large)=n1, (no,small)=n4, (yes,large)=n4, (yes,small)=n1 · sparsest cell n=1 (min_samples per variant experiment = 20) → SPARSE_CELL_RISK
smoke-copilot-aoai-entra.md — cells: (no,large)=n2, (no,small)=n3, (yes,large)=n3, (yes,small)=n2 · sparsest cell n=2 (min_samples per variant experiment = 20) → SPARSE_CELL_RISK

All three workflows are still EXTEND at the core-decision level (insufficient observations), so no promotion is at risk of an interaction-blind decision today. Once outcome-metric wiring lands, apply the interaction_underpowered safety hold before any PROMOTE on these workflows given the sparse 2×2 cells above.

---

📊 Summary

View Full Experiments Table

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
prompt_compression	agent-performance-analyzer	caveman	verbose	p=0.397	N/A	EXTEND	EXTEND
sub_agent_strategy	agent-persona-explorer	batch	per_scenario	p=0.64	N/A	EXTEND	EXTEND
sub_agent_strategy	architectureguardian	single_agent	sub_agents	p=0.524	N/A	EXTEND	EXTEND
audit_decomposition	auditworkflows	single_agent	phased_sub_agents	p=0.216	UNSUPPORTED	EXTEND	EXTEND
tone_variant	aw-failure-investigator	assertive	clinical	p=0.93	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	blogauditor	concise	detailed	p=0.487	N/A	EXTEND	EXTEND
tone_variant	breakingchangechecker	neutral	urgent	p=0.332	N/A	EXTEND	EXTEND
prompt_style	ci-coach	detailed	concise	p=0	UNSUPPORTED	EXTEND	EXTEND
output_format	copilotagentanalysis	prose	ste	p=4.76e-07	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	daily-agentrx-trace-optimizer	single_agent	sub_agents	p=1	N/A	EXTEND	EXTEND
detail_level	dailyarchitecturediagram	brief	comprehensive	p=0.319	N/A	EXTEND	EXTEND
prompt_style	daily-astrostylelite-markdown-spellcheck	concise	detailed	p=0.891	N/A	EXTEND	EXTEND
model_size	dailycachestrategyanalyzer	gpt-5.3-codex	gpt-5.3-codex-spark	p=0.571	UNSUPPORTED	EXTEND	EXTEND
model_size	dailycavemanoptimizer	agent	claude-haiku-4.5	p=3.7e-21	N/A	INCONCLUSIVE	INCONCLUSIVE
output_format	dailycodemetrics	executive_summary	full_detail	p=1.25e-06	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	daily-community-attribution	concise	verbose	p=0.89	N/A	EXTEND	EXTEND
output_format	dailycompilerquality	concise	detailed	p=2.46e-06	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	dailydochealer	agent	claude-haiku-4.5	p=9.47e-21	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	dailydocupdater	agent	claude-haiku-4.5	p=8.67e-19	N/A	INCONCLUSIVE	INCONCLUSIVE
reasoning_depth	daily-fact	single_pass	multi_candidate	p=0.0468	UNSUPPORTED	EXTEND	EXTEND
model_size	dailyfunctionnamer	?	(n/a)	p=0	N/A	EXTEND	EXTEND
output_format	dailyissuesreport	collapsible	inline	p=4.36e-06	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	daily-news	concise	detailed	p=0.489	N/A	EXTEND	EXTEND
remove_redundant_context_v1	dailyrenderingscriptsverifier	?	(n/a)	p=0	N/A	EXTEND	EXTEND
log_fetch_strategy	dailysafeoutputoptimizer	eager	lazy	p=0	UNSUPPORTED	EXTEND	EXTEND
reasoning_depth	daily-security-red-team	single_pass	iterative	p=0.0269	UNSUPPORTED	EXTEND	EXTEND
semgrep_output_format	dailysemgrepscan	bullet_list	prose	p=0.0232	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer	default	relaxed	p=0.781	N/A	INCONCLUSIVE	INCONCLUSIVE
caveman_mode	dataflowprdiscussiondataset	no	yes	p=0.41	N/A	EXTEND	EXTEND
output_format	deepreport	annotated_brief	executive_brief	p=4.82e-05	N/A	INCONCLUSIVE	INCONCLUSIVE
summary_detail	dependabotcampaign	brief	detailed	p=1	N/A	EXTEND	EXTEND
prompt_style	dependabotgochecker	concise	detailed	p=0.851	N/A	INCONCLUSIVE	INCONCLUSIVE
tool_verbosity	gpclean	full_bash	minimal_toolset	p=0.571	N/A	EXTEND	EXTEND
prompt_style	issue-arborist	concise	detailed	p=0.397	N/A	EXTEND	EXTEND
reasoning_depth	plan	baseline	deep	p=0.369	N/A	INCONCLUSIVE	INCONCLUSIVE
remove_redundant_context_v1	prsouschef	?	(n/a)	p=0	N/A	EXTEND	EXTEND
sub_agent_strategy	smoke-antigravity	single_agent	sub_agents	p=0.495	N/A	EXTEND	EXTEND
caveman	smoke-copilot	no	yes	p=1	N/A	EXTEND	EXTEND
subagent_model	smoke-copilot	large	small	p=1	N/A	EXTEND	EXTEND
caveman	smoke-copilot-aoai-apikey	no	yes	p=0.899	N/A	EXTEND	EXTEND
subagent_model	smoke-copilot-aoai-apikey	large	small	p=0.899	N/A	EXTEND	EXTEND
caveman	smoke-copilot-aoai-entra	no	yes	p=1	N/A	EXTEND	EXTEND
subagent_model	smoke-copilot-aoai-entra	large	small	p=1	N/A	EXTEND	EXTEND
sub_agent_strategy	smokecopilotsubagents	delegated_sequential	inline_strict	p=0.679	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	smoke-gemini	single_agent	sub_agents	p=0.265	N/A	EXTEND	EXTEND
sub_agent_decomposition	smoke-pi	parallel_sub_agents	single_agent	p=0.362	N/A	EXTEND	EXTEND
prompt_style_test	smokeproject	?	(n/a)	p=0	N/A	EXTEND	EXTEND
sub_agent_strategy	smoketemporaryid	single_agent	sub_agents	p=0	UNSUPPORTED	EXTEND	EXTEND
model_size	testqualitysentinel	?	(n/a)	p=0	N/A	EXTEND	EXTEND
tone_style	typist	conversational	formal	p=0.362	N/A	EXTEND	EXTEND
prefetch_strategy	weeklyblogpostwriter	eager	lazy	p=0.41	N/A	EXTEND	EXTEND

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire outcome-metric collection (token_count_per_run, run_success_rate, effective_token_count, etc.) into the analyze pipeline — every READY experiment today resolves to EXTEND/INCONCLUSIVE solely because insufficient_observations or guardrail_unsupported blocks the decision layer, not because evidence is weak.
2. Extend aw-failure-investigator's tone_variant experiment support beyond 2-variant comparisons, or split it into pairwise A/B tests — unsupported_multi_variant is the only blocker there with 397 total runs banked.
3. Consolidate the smoke-copilot* family's dual caveman × subagent_model factorial design — current per-cell sample sizes (as low as n=1) create SPARSE_CELL_RISK; either increase min_samples awareness for factorial designs or run sequentially.

Top 3 eval actions

1. Add or tighten eval:output_format_adherence graders for copilot-agent-analysis, daily-code-metrics, daily-semgrep-scan, daily-issues-report, and deep-report — these all declare eval: secondary metrics but no observation pipeline surfaced them in this run.
2. Add a run_success_rate / empty_output_rate grader wired to native Actions run conclusions for ci-coach, daily-fact, and daily-security-red-team — their guardrail metrics are declared but marked unsupported.
3. Add a token/cost grader for pr-sous-chef and daily-rendering-scripts-verifier, whose primary metric is a grader:execution-duration reference with no corresponding recorded observations yet.

Decision-pipeline gaps for next PR

• insufficient_observations (18 experiments) and guardrail_unsupported (3 experiments) account for all EXTEND decisions among READY experiments — the single highest-leverage fix is landing a generic per-run metric-observation writer that the experiments analyze decision layer can consume.
• unsupported_multi_variant (1 experiment, aw-failure-investigator) — the core decision engine only supports exactly 2 variants; either extend it or require workflows to declare pairwise variants.
• Factorial interaction diagnostics (SPARSE_CELL_RISK) are computed ad hoc in this report; they should become a normalized analyses[].interaction field in gh aw experiments analyze output so decisions can consume it directly instead of relying on reporting-layer recomputation.

Descriptive window: last 30 runs per workflow (subject to available run artifacts) · Decision thresholds: from decision_policy (min_samples=20–30 per experiment, confidence=0.95)
Run: 34325977768

Generated by 🧪 daily-experiment-report · copilot · auto · 157.6 AIC · ⌖ 36 AIC · ⊞ 16.5K · ◷

• expires on Sep 12, 2026, 12:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #59921.