[experiments] Daily Experiment Report — 2026-09-27

[github-actions[bot]]

Executive Summary

This report covers 51 experiments across 48 workflows with active experiments: frontmatter. 20 experiments have a linked tracking issue; 31 do not (legacy shorthand config or no issue: field, and 3 branches with no matching source file).

Decision distribution: EXTEND=37, INCONCLUSIVE=14

Readiness distribution: COLLECTING=25, READY=26

Warning

dailyfact is failing 100% of recent runs (30/30 failures) across both reasoning_depth variants (single_pass n=15, multi_candidate n=14) in the last 30 runs — confirmed against raw run conclusions, not a data-parsing artifact. This is a workflow health incident independent of the A/B test and should be triaged first.

Quick Stats

Metric	Value
Total experiment configs analysed	51
Workflows with active experiments	48
Experiments with tracking issue	20
Experiments READY for analysis	26
Experiments still COLLECTING	25
PROMOTE decisions	0
REJECT decisions	0
EXTEND decisions	37
INCONCLUSIVE decisions	14

Tracked Experiments (linked to a GitHub issue)

Per-experiment detail (20 experiments)

agentperformanceanalyzer — prompt_compression (issue #33280)

Display name: Agent Performance Analyzer - Meta-Orchestrator | Readiness: READY | Decision: EXTEND

Experiment : prompt_compression
Workflow   : agentperformanceanalyzer
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 29 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| caveman          |   18 |   72.2%  |          812.6 | [652.8 , 972.3]    | ████████ 70/20 (100%) |
| verbose          |   11 |   81.8%  |          770.1 | [595.9 , 944.3]    | ████████ 58/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.2890 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

auditworkflows — audit_decomposition (issue #43177)

Display name: Agentic Workflow Audit Agent | Readiness: COLLECTING | Decision: EXTEND

Experiment : audit_decomposition
Workflow   : auditworkflows
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 29 runs with artifacts
min_samples: 264 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_agent     |   15 |   26.7%  |          526.7 | [371.6 , 681.8]    | █░░░░░░░ 36/264 (14%) |
| phased_sub_agents|   14 |   42.9%  |          495.9 | [425.9 , 565.8]    | █░░░░░░░ 47/264 (18%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.2250 (is_balanced=True)

Guardrails:
  configured=True, passed=None

Core decision : EXTEND
Reason code   : insufficient_samples
Rationale     : one or more variants have fewer usable observations than min_samples
Report action : EXTEND

awfailureinvestigator — tone_variant (issue #36105)

Display name: [aw] Failure Investigator (6h) | Readiness: READY | Decision: INCONCLUSIVE

Experiment : tone_variant
Workflow   : awfailureinvestigator
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 3 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| assertive        |    1 |  100.0%  |          499.0 | N/A (n<2)          | ████████ 159/20 (100%) |
| clinical         |    1 |  100.0%  |          469.0 | N/A (n<2)          | ████████ 155/20 (100%) |
| narrative        |    1 |  100.0%  |          448.0 | N/A (n<2)          | ████████ 157/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.9628 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

breakingchangechecker — tone_variant (issue #42467)

Display name: Breaking Change Checker | Readiness: READY | Decision: EXTEND

Experiment : tone_variant
Workflow   : breakingchangechecker
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 21 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| neutral          |   10 |   40.0%  |          483.1 | [392.4 , 573.8]    | ████████ 27/20 (100%) |
| urgent           |   11 |   27.3%  |          732.5 | [538.8 , 926.3]    | ████████ 24/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.6773 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

cicoach — prompt_style (issue #32335)

Display name: CI Optimization Coach | Readiness: READY | Decision: EXTEND

Experiment : prompt_style
Workflow   : cicoach
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 30 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| detailed         |   20 |   70.0%  |         1321.0 | [1067.8 , 1574.3]  | ████████ 57/20 (100%) |
| concise          |   10 |   70.0%  |         1114.0 | [935.1 , 1292.9]   | ████████ 39/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0000 (is_balanced=True)

Guardrails:
  configured=True, passed=None

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

dailyarchitecturediagram — detail_level (issue #31926)

Display name: 1. List all Go packages with their doc comments | Readiness: COLLECTING | Decision: EXTEND

Experiment : detail_level
Workflow   : dailyarchitecturediagram
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 18 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| brief            |   11 |   90.9%  |          599.3 | [441.5 , 757.0]    | ████░░░░ 11/20 (55%) |
| comprehensive    |    7 |   85.7%  |          585.4 | [293.0 , 877.8]    | ███░░░░░ 7/20 (35%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.3484 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_samples
Rationale     : one or more variants have fewer usable observations than min_samples
Report action : EXTEND

dailycompilerquality — output_format (issue #32390)

Display name: Daily Compiler Quality Check | Readiness: COLLECTING | Decision: INCONCLUSIVE

Experiment : output_format
Workflow   : dailycompilerquality
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 29 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    8 |   50.0%  |          462.9 | [355.8 , 569.9]    | ████████ 55/20 (100%) |
| detailed         |   12 |   58.3%  |          546.0 | [426.5 , 665.5]    | ████████ 62/20 (100%) |
| ste              |    9 |   44.4%  |          536.9 | [404.5 , 669.3]    | ██████░░ 16/20 (80%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0000 (is_balanced=False)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

dailyfact — reasoning_depth (issue #31324)

Display name: Daily Fact | Readiness: READY | Decision: EXTEND

Experiment : reasoning_depth
Workflow   : dailyfact
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 29 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_pass      |   15 |    0.0%  |          325.4 | [264.2 , 386.6]    | ████████ 56/30 (100%) |
| multi_candidate  |   14 |    0.0%  |          364.6 | [307.3 , 422.0]    | ████████ 40/30 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0982 (is_balanced=True)

Guardrails:
  configured=True, passed=None

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
Report action : EXTEND

dailyissuesreport — output_format (issue #30573)

Display name: Daily Issues Report Generator | Readiness: READY | Decision: INCONCLUSIVE

Experiment : output_format
Workflow   : dailyissuesreport
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 28 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| collapsible      |   11 |   81.8%  |          732.5 | [642.6 , 822.5]    | ████████ 59/20 (100%) |
| inline           |    7 |  100.0%  |          748.9 | [676.9 , 820.8]    | ████████ 63/20 (100%) |
| ste              |   10 |   90.0%  |          787.2 | [661.2 , 913.2]    | ████████ 21/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0000 (is_balanced=False)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

dailynews — prompt_style (issue #31190)

Display name: Daily News | Readiness: READY | Decision: EXTEND

Experiment : prompt_style
Workflow   : dailynews
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 30 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |   17 |   94.1%  |          750.4 | [704.4 , 796.3]    | ████████ 62/20 (100%) |
| detailed         |   13 |   92.3%  |          717.7 | [688.4 , 746.9]    | ████████ 50/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.2557 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

dailysafeoutputoptimizer — log_fetch_strategy (issue #38094)

Display name: Daily Safe Output Tool Optimizer | Readiness: COLLECTING | Decision: EXTEND

Experiment : log_fetch_strategy
Workflow   : dailysafeoutputoptimizer
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 1 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| lazy             |    1 |  100.0%  |         2400.0 | N/A (n<2)          | ░░░░░░░░ 0/20 (0%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0000 (is_balanced=True)

Guardrails:
  configured=True, passed=None

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : at least two variants are required for a decision
Report action : EXTEND

dailysecurityredteam — reasoning_depth (issue #31673)

Display name: Cache directory setup | Readiness: READY | Decision: EXTEND

Experiment : reasoning_depth
Workflow   : dailysecurityredteam
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 29 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_pass      |   14 |   28.6%  |         2831.5 | [1904.4 , 3758.6]  | ████████ 56/30 (100%) |
| iterative        |   15 |   13.3%  |         3378.5 | [2731.2 , 4025.7]  | ████████ 81/30 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0309 (is_balanced=False)

Guardrails:
  configured=True, passed=None

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

dailysemgrepscan — semgrep_output_format (issue #32795)

Display name: Daily Semgrep Scan | Readiness: COLLECTING | Decision: INCONCLUSIVE

Experiment : semgrep_output_format
Workflow   : dailysemgrepscan
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 28 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| bullet_list      |    9 |  100.0%  |          565.0 | [430.9 , 699.1]    | ███████░ 28/30 (93%) |
| prose            |    9 |  100.0%  |          512.4 | [448.1 , 576.8]    | ███████░ 27/30 (90%) |
| structured_sectio|   10 |  100.0%  |          587.1 | [410.2 , 764.0]    | ██████░░ 22/30 (73%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.0232 (is_balanced=False)

Guardrails:
  configured=True, passed=None

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

dataflowprdiscussiondataset — caveman_mode (issue #37102)

Display name: DataFlow PR & Discussion Dataset Builder | Readiness: COLLECTING | Decision: EXTEND

Experiment : caveman_mode
Workflow   : dataflowprdiscussiondataset
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 16 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| no               |    6 |   50.0%  |        11246.2 | [-1047.7 , 23540.0] | ██░░░░░░ 6/20 (30%) |
| yes              |   10 |   40.0%  |        11171.0 | [3174.3 , 19167.7] | ████░░░░ 10/20 (50%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.3187 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_samples
Rationale     : one or more variants have fewer usable observations than min_samples
Report action : EXTEND

issuearborist — prompt_style (issue #30015)

Display name: Issue Arborist | Readiness: READY | Decision: EXTEND

Experiment : prompt_style
Workflow   : issuearborist
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 29 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |   13 |   46.2%  |          390.8 | [326.9 , 454.8]    | ████████ 46/20 (100%) |
| detailed         |   16 |   37.5%  |          402.1 | [344.7 , 459.6]    | ████████ 38/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.3868 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

plan — reasoning_depth (issue #42941)

Display name: Plan Command | Readiness: COLLECTING | Decision: INCONCLUSIVE

Experiment : reasoning_depth
Workflow   : plan
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 7 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| baseline         |    4 |  100.0%  |          447.5 | [332.8 , 562.2]    | ██░░░░░░ 4/20 (20%) |
| deep             |    2 |  100.0%  |          545.0 | [-916.2 , 2006.2]  | █░░░░░░░ 2/20 (10%) |
| shallow          |    1 |  100.0%  |          470.0 | N/A (n<2)          | ░░░░░░░░ 1/20 (5%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.3694 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

smokecopilotsubagents — sub_agent_strategy (issue #47551)

Display name: Smoke Copilot Sub Agents | Readiness: COLLECTING | Decision: INCONCLUSIVE

Experiment : sub_agent_strategy
Workflow   : smokecopilotsubagents
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 30 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| delegated_sequent|    8 |   87.5%  |          389.2 | [309.6 , 468.9]    | ████░░░░ 16/30 (53%) |
| inline_strict    |   10 |   90.0%  |          389.1 | [340.3 , 437.9]    | ████░░░░ 14/30 (47%) |
| single_agent_cont|   12 |   75.0%  |          392.6 | [349.1 , 436.0]    | █████░░░ 19/30 (63%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.6366 (is_balanced=True)

Guardrails:
  configured=True, passed=None

Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
Report action : INCONCLUSIVE

testqualitysentinel — model_size (issue #43530)

Display name: Test Quality Sentinel | Readiness: READY | Decision: EXTEND

Experiment : model_size
Workflow   : testqualitysentinel
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 22 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| claude-haiku-4.5 |   10 |   90.0%  |          830.9 | [285.6 , 1376.2]   | ████████ 40/20 (100%) |
| claude-sonnet-5  |   12 |  100.0%  |          500.3 | [378.0 , 622.7]    | ████████ 39/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.8761 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

typist — tone_style (issue #34032)

Display name: Typist - Go Type Analysis | Readiness: READY | Decision: EXTEND

Experiment : tone_style
Workflow   : typist
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 30 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| conversational   |   14 |   92.9%  |         1148.3 | [983.5 , 1313.1]   | ████████ 47/20 (100%) |
| formal           |   16 |   68.8%  |          826.6 | [661.8 , 991.4]    | ████████ 42/20 (100%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.6028 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

weeklyblogpostwriter — prefetch_strategy (issue #38590)

Display name: Weekly Blog Post Writer | Readiness: COLLECTING | Decision: EXTEND

Experiment : prefetch_strategy
Workflow   : weeklyblogpostwriter
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 15 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| eager            |    5 |  100.0%  |          635.8 | [591.4 , 680.2]    | ██░░░░░░ 5/20 (25%) |
| lazy             |   10 |  100.0%  |          767.9 | [696.3 , 839.5]    | ████░░░░ 10/20 (50%) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi-square assignment-balance test, p=0.1935 (is_balanced=True)

Guardrails:
  (no guardrails configured; native metric observation UNSUPPORTED -> data-pipeline gap, treated as not-passing evidence)

Core decision : EXTEND
Reason code   : insufficient_samples
Rationale     : one or more variants have fewer usable observations than min_samples
Report action : EXTEND

Untracked Experiments (no linked issue)

Summary (31 experiments — legacy shorthand config or missing source file)

Workflow	Experiment	Readiness	Decision	Reason code	Samples
agentpersonaexplorer	sub_agent_strategy	READY	EXTEND	insufficient_observations	batch=66, per_scenario=62
architectureguardian	sub_agent_strategy	COLLECTING	EXTEND	insufficient_samples	single_agent=17, sub_agents=21
blogauditor	prompt_style	COLLECTING	EXTEND	insufficient_samples	concise=4, detailed=7
copilotagentanalysis	output_format	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	prose=34, ste=3, structured=43
dailyagentrxtraceoptimizer	sub_agent_strategy	READY	EXTEND	insufficient_observations	single_agent=42, sub_agents=42
dailyastrostylelitemarkdownspellcheck	prompt_style	READY	EXTEND	insufficient_observations	concise=75, detailed=71
dailycachestrategyanalyzer	model_size	COLLECTING	EXTEND	insufficient_samples	gpt-5.3-codex=11, gpt-5.3-codex-spark=10
dailycavemanoptimizer	model_size	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	agent=1, claude-haiku-4.5=69, claude-sonnet-4.6=34, claude-sonnet-5=9, small-agent=2
dailycodemetrics	output_format	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	executive_summary=54, full_detail=58, ste=16
dailycommunityattribution	prompt_style	READY	EXTEND	insufficient_observations	concise=70, verbose=70
dailydochealer	model_size	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	agent=2, claude-haiku-4.5=60, claude-sonnet-4.6=43, claude-sonnet-5=9, small-agent=1
dailydocupdater	model_size	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	agent=1, claude-haiku-4.5=47, claude-sonnet-4.6=47, claude-sonnet-5=13, small-agent=2
dailyfunctionnamer	model_size	COLLECTING	EXTEND	insufficient_observations	None
dailyrenderingscriptsverifier	remove_redundant_context_v1	COLLECTING	EXTEND	insufficient_observations	None
dailysubagentoptimizer	timeout_setting	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	default=2, relaxed=1, tight=1
deepreport	output_format	READY	INCONCLUSIVE	unsupported_multi_variant	annotated_brief=59, executive_brief=83, full_briefing=58, ste=37
dependabotcampaign	summary_detail	COLLECTING	EXTEND	insufficient_samples	brief=6, detailed=6
dependabotgochecker	prompt_style	COLLECTING	INCONCLUSIVE	unsupported_multi_variant	concise=19, detailed=18, step_by_step=15
gpclean	tool_verbosity	READY	EXTEND	insufficient_observations	full_bash=58, minimal_toolset=67
prsouschef	remove_redundant_context_v1	COLLECTING	EXTEND	insufficient_observations	None
smokeantigravity	sub_agent_strategy	READY	EXTEND	insufficient_observations	single_agent=99, sub_agents=109
smokecopilot	caveman	READY	EXTEND	insufficient_observations	no=186, yes=185
smokecopilot	subagent_model	READY	EXTEND	insufficient_observations	large=176, small=175
smokecopilotaoaiapikey	caveman	READY	EXTEND	insufficient_observations	no=95, yes=94
smokecopilotaoaiapikey	subagent_model	READY	EXTEND	insufficient_observations	large=94, small=95
smokecopilotaoaientra	caveman	READY	EXTEND	insufficient_observations	no=81, yes=81
smokecopilotaoaientra	subagent_model	READY	EXTEND	insufficient_observations	large=81, small=81
smokegemini	sub_agent_strategy	READY	EXTEND	insufficient_observations	single_agent=129, sub_agents=143
smokepi	sub_agent_decomposition	READY	EXTEND	insufficient_observations	parallel_sub_agents=42, single_agent=34
smokeproject	prompt_style_test	COLLECTING	EXTEND	insufficient_observations	None
smoketemporaryid	sub_agent_strategy	COLLECTING	EXTEND	insufficient_observations	None

Factorial Interaction Diagnostics (multi-experiment workflows)

Three workflows run two simultaneous experiments (caveman × subagent_model). The table below tests whether variant assignment across the two experiments is statistically independent (orthogonal randomization) — this is a randomization-health check, not an outcome-interaction test, and it only gates report recommendations when a core decision is PROMOTE (none exist in this run).

smokecopilot

Cell (caveman|subagent_model)	n
yes	large
no	small
yes	small
no	large

interaction_p_value = 0.7086 | min cell n = 85 | interaction_risk_status = OK

smokecopilotaoaiapikey

Cell (caveman|subagent_model)	n
yes	small
no	large
yes	large
no	small

interaction_p_value = 0.0006 | min cell n = 35 | interaction_risk_status = OK

⚠️ Assignment independence is rejected at p<0.01 — the two experiments' variant assignments are NOT orthogonal in smokecopilotaoaiapikey. No core decision is PROMOTE here so no safety-hold applies, but this is worth flagging as a randomization/data-pipeline concern.

smokecopilotaoaientra

Cell (caveman|subagent_model)	n
no	large
yes	small
yes	large
no	small

interaction_p_value = 0.0593 | min cell n = 34 | interaction_risk_status = OK

Self-Tuning Continuation Plan

Top 3 experiment actions:

1. typist / tone_style and testqualitysentinel / model_size are READY with balanced assignment (chi-square not significant) — prioritize wiring a supported outcome metric (currently guardrail_unsupported/insufficient_observations) so these can progress past EXTEND.
2. dailyfact / reasoning_depth shows 0% success rate across both variants (single_pass n=15, multi_candidate n=14) in the last 30 runs — this is a real signal worth investigating as a workflow health issue, independent of the A/B comparison itself.
3. 14 of 51 experiment configs return unsupported_multi_variant (3+ variant experiments the core analyzer cannot yet compare pairwise) — extend the analyzer to support >2-variant comparisons (e.g., dailycompilerquality, awfailureinvestigator, dailyissuesreport).

Top 3 eval / data-pipeline actions:

1. Guardrails are universally unsupported across all 51 analyzed experiments — zero experiments have a supported/passing guardrail metric observation. This is the single largest gap blocking any PROMOTE/REJECT decision; wire native guardrail metric collection into the experiment framework.
2. smokecopilotaoaiapikey's two simultaneous experiments show a statistically significant assignment-interaction (p=0.0006) — audit the randomization logic for that workflow to confirm variant assignment is drawn independently per experiment.
3. 3 experiment branches (smokeantigravity, dependabotcampaign, dailysubagentoptimizer) have no matching .md source file — either restore/rename the source workflow or archive the stale experiment branch to avoid orphaned state.

Generated by 🧪 daily-experiment-report · copilot · auto · 292.4 AIC · ⌖ 19.2 AIC · ⊞ 16.3K · ◷

• expires on Sep 30, 2026, 12:11 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #63963.