[experiments] Daily Experiment Report — 2026-09-21

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-21

51 A/B experiments across 48 workflows in github/gh-aw were analysed via gh aw experiments analyze. None have decision-quality evidence yet: 24 are READY for a statistical read (data volume sufficient) but every one of them is blocked by a deterministic gate — either an unsupported outcome metric (guardrail_unsupported), too few usable observations (insufficient_observations), or ≥3 concurrent variants that the core analyzer does not yet resolve pairwise (unsupported_multi_variant). No tracking issue: field is configured on any experiment, so Steps 8/9 (issue comments, lifecycle labels) are skipped this run.

⚡ Quick Stats

Metric	Value
Active experiments	51 (across 48 workflows)
Ready for analysis (readiness=READY)	24
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	3 (all smoke-copilot* caveman×subagent_model pairs)
Reason-code breakdown	insufficient_observations: 26 · unsupported_multi_variant: 14 · insufficient_samples: 8 · guardrail_unsupported: 3

---

caveman × subagent_model · smoke-copilot (+ 2 sibling smoke workflows)

Readiness: 🟢 READY (both experiments)
Variants: caveman = no/yes · subagent_model = large/small · Window: last 30 runs · Analysed: 10 runs with artifacts (matched)
min_samples: 20 per variant · Evidence: none — both experiments blocked by insufficient_observations

These three smoke workflows (smoke-copilot, smoke-copilot-aoai-apikey, smoke-copilot-aoai-entra) run two concurrent 2×2 factorial experiments. A cross-experiment interaction safety hold applies: the 2×2 cells are sparse (max cell n=4, well under min_samples=20) in all three workflows, so even though neither individual experiment's core decision is PROMOTE today, this hold is recorded pre-emptively for when either becomes ready to promote.

📈 View Detailed Statistics — smoke-copilot

Sample Sizes & Progress

Variant	Runs (state)	Progress
caveman=no	183	██████████ 183/20 (100%)
caveman=yes	182	██████████ 182/20 (100%)
subagent_model=large	173	██████████ 173/20 (100%)
subagent_model=small	172	██████████ 172/20 (100%)

(State-branch counts are well past min_samples; insufficient_observations means the outcome-metric pipeline has not yet attached enough per-run observation records — a data-pipeline gap, not a sample-size gap.)

Experiment : caveman
Workflow   : smoke-copilot
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| no (control)     |   5  |  40.0%   |    758.2       | [558.0 , 958.4]    | 20/20 (100%)  |
| yes              |   5  |  20.0%   |    669.4       | [454.8 , 884.0]    | 20/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Interaction cells (caveman × subagent_model), n < 20 in every cell → SPARSE_CELL_RISK:
  (yes, large):  n=4  succ=25% dur=656s
  (no,  small):  n=3  succ=33% dur=733s
  (no,  large):  n=2  succ=50% dur=796s
  (yes, small):  n=1  succ=0%  dur=722s

Evidence: none (below min_samples for outcome observations)

Guardrails: not configured for this experiment

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : outcome-metric observations below threshold despite state-branch assignment counts exceeding min_samples
Report action : EXTEND (interaction_underpowered would additionally apply if either core decision reached PROMOTE)

Core decision: EXTEND (insufficient_observations) — assignment volume is high (170+ per variant) but usable outcome observations remain thin; Report action: EXTEND (interaction_underpowered) — 2×2 cells are sparse across all three smoke-copilot workflows, so promotion is held even prospectively.

---

prompt_style · ci-coach

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: guardrail_unsupported — mandatory run_success_rate guardrail has no supported metric observation

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs (state)	Progress
concise	37	██████████ 37/20 (185%)
detailed	54	██████████ 54/20 (270%)

Experiment : prompt_style
Workflow   : ci-coach
Hypothesis : H0: no change in PR merge rate/proposal quality. H1: concise reduces tokens >=25% w/o quality loss
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| detailed (ctrl)  |   9  |  77.8%   |   1493.9       | [986.5 , 2001.3]   | 54/20 (270%)  |
| concise          |   1  | 100.0%   |   1280.0       | N/A (n<2)          | 37/20 (185%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: none — run_success_rate guardrail unsupported (no backing observation record)

Guardrails:
  run_success_rate >=0.85 : UNSUPPORTED
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory run_success_rate guardrail is not backed by a supported metric observation.

---

reasoning_depth · daily-fact, daily-security-red-team

Readiness: 🟢 READY (both)
Variants: single_pass vs multi_candidate/iterative · min_samples: 30
Evidence: guardrail_unsupported for both — empty_output_rate/run_success_rate unsupported

daily-fact hypothesis: H0 no change in engagement; H1 multi_candidate yields +20% reactions. daily-security-red-team hypothesis: H0 no change in finding quality; H1 iterative cuts false-positive rate ≥20% at ≤30% token overhead.

📈 View Detailed Statistics — daily-fact

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_pass      |   4  |   0.0%   |    442.8       | [192.0 , 693.5]    | 54/30 (180%)  |
| multi_candidate  |   6  |   0.0%   |    330.5       | [311.3 , 349.7]    | 37/30 (123%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Guardrails: empty_output_rate <0.05 UNSUPPORTED · run_success_rate >=0.95 UNSUPPORTED
Core decision : EXTEND  |  Reason code : guardrail_unsupported

Note: the 0% observed success rate in this 30-run window is a run-conclusion artifact (multiple recent runs failed for reasons unrelated to the experiment split — see workflow run logs); it is descriptive context only and does not feed the core decision.

Core decision: EXTEND (guardrail_unsupported) for both experiments.

---

tool_verbosity · gpclean

Readiness: 🟢 READY · Variants: full_bash vs minimal_toolset · min_samples: 20 · Evidence: none, insufficient_observations

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| full_bash        |   6  |  16.7%   |    320.5       | [258.2 , 382.8]    | 57/20 (285%)  |
| minimal_toolset  |   4  |   0.0%   |    332.2       | [262.5 , 402.0]    | 62/20 (310%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Core decision : EXTEND  |  Reason code : insufficient_observations

Core decision: EXTEND (insufficient_observations).

---

📊 Summary

View Full Experiments Table (all 51 experiments)

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
prompt_compression	agent-performance-analyzer	caveman	verbose	none	n/a	EXTEND (insufficient_observations)	EXTEND
sub_agent_strategy	agent-persona-explorer	batch	per_scenario	none	n/a	EXTEND (insufficient_observations)	EXTEND
sub_agent_strategy	architecture-guardian	single_agent	sub_agents	none	n/a	EXTEND (insufficient_samples)	EXTEND
audit_decomposition	audit-workflows	single_agent	phased_sub_agents	none	n/a	EXTEND (insufficient_samples)	EXTEND
tone_variant	aw-failure-investigator	assertive/clinical/narrative	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
prompt_style	blog-auditor	concise	detailed	none	n/a	EXTEND (insufficient_samples)	EXTEND
tone_variant	breaking-change-checker	neutral	urgent	none	n/a	EXTEND (insufficient_observations)	EXTEND
prompt_style	ci-coach	detailed	concise	guardrail_unsupported	UNSUPPORTED	EXTEND	EXTEND
output_format	copilot-agent-analysis	prose/structured/ste	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
sub_agent_strategy	daily-agentrx-trace-optimizer	single_agent	sub_agents	none	n/a	EXTEND (insufficient_observations)	EXTEND
detail_level	daily-architecture-diagram	brief	comprehensive	none	n/a	EXTEND (insufficient_samples)	EXTEND
prompt_style	daily-astrostylelite-markdown-spellcheck	concise	detailed	none	n/a	EXTEND (insufficient_observations)	EXTEND
model_size	daily-cache-strategy-analyzer	gpt-5.3-codex	gpt-5.3-codex-spark	none	n/a	EXTEND (insufficient_samples)	EXTEND
model_size	daily-caveman-optimizer	multiple (5)	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
output_format	daily-code-metrics	executive_summary/full_detail/ste	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
prompt_style	daily-community-attribution	concise	verbose	none	n/a	EXTEND (insufficient_observations)	EXTEND
output_format	daily-compiler-quality	concise/detailed/ste	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
model_size	daily-doc-healer	multiple (5)	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
model_size	daily-doc-updater	multiple (5)	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
reasoning_depth	daily-fact	single_pass	multi_candidate	guardrail_unsupported	UNSUPPORTED	EXTEND	EXTEND
model_size	daily-function-namer	(n/a, 3 runs)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
output_format	daily-issues-report	collapsible/inline/ste	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
prompt_style	daily-news	concise	detailed	none	n/a	EXTEND (insufficient_observations)	EXTEND
remove_redundant_context_v1	daily-rendering-scripts-verifier	(n/a, 1 run)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
log_fetch_strategy	daily-safe-output-optimizer	(n/a, 1 run)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
reasoning_depth	daily-security-red-team	single_pass	iterative	guardrail_unsupported	UNSUPPORTED	EXTEND	EXTEND
semgrep_output_format	daily-semgrep-scan	bullet_list/prose/structured_sections	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
timeout_setting	daily-subagent-optimizer	default/relaxed/tight	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
caveman_mode	dataflow-pr-discussion-dataset	no	yes	none	n/a	EXTEND (insufficient_samples)	EXTEND
output_format	deep-report	annotated_brief/executive_brief/full_briefing/ste	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
summary_detail	dependabot-campaign	brief	detailed	none	n/a	EXTEND (insufficient_samples)	EXTEND
prompt_style	dependabot-go-checker	concise/detailed/step_by_step	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
tool_verbosity	gpclean	full_bash	minimal_toolset	none	n/a	EXTEND (insufficient_observations)	EXTEND
prompt_style	issue-arborist	concise	detailed	none	n/a	EXTEND (insufficient_observations)	EXTEND
reasoning_depth	plan	baseline/deep/shallow	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
remove_redundant_context_v1	pr-sous-chef	(n/a, 1 run)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
sub_agent_strategy	smoke-antigravity	single_agent	sub_agents	none	n/a	EXTEND (insufficient_observations)	EXTEND
caveman	smoke-copilot	no	yes	none	n/a	EXTEND (insufficient_observations)	EXTEND (interaction_underpowered)
subagent_model	smoke-copilot	large	small	none	n/a	EXTEND (insufficient_observations)	EXTEND (interaction_underpowered)
caveman	smoke-copilot-aoai-apikey	no	yes	none	n/a	EXTEND (insufficient_observations)	EXTEND (interaction_underpowered)
subagent_model	smoke-copilot-aoai-apikey	large	small	none	n/a	EXTEND (insufficient_observations)	EXTEND (interaction_underpowered)
caveman	smoke-copilot-aoai-entra	no	yes	none	n/a	EXTEND (insufficient_observations)	EXTEND (interaction_underpowered)
subagent_model	smoke-copilot-aoai-entra	large	small	none	n/a	EXTEND (insufficient_observations)	EXTEND (interaction_underpowered)
sub_agent_strategy	smoke-copilot-sub-agents	delegated_sequential/inline_strict/single_agent_control	—	none	n/a	INCONCLUSIVE (unsupported_multi_variant)	INCONCLUSIVE
sub_agent_strategy	smoke-gemini	single_agent	sub_agents	none	n/a	EXTEND (insufficient_observations)	EXTEND
sub_agent_decomposition	smoke-pi	single_agent	parallel_sub_agents	none	n/a	EXTEND (insufficient_observations)	EXTEND
prompt_style_test	smoke-project	(n/a, 1 run)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
sub_agent_strategy	smoke-temporary-id	(n/a, 1 run)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
model_size	test-quality-sentinel	(n/a, 6 runs)	—	none	n/a	EXTEND (insufficient_observations)	EXTEND
tone_style	typist	conversational	formal	none	n/a	EXTEND (insufficient_observations)	EXTEND
prefetch_strategy	weekly-blog-post-writer	eager	lazy	none	n/a	EXTEND (insufficient_samples)	EXTEND

🛠️ Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire outcome-observation backing for run_success_rate / empty_output_rate — 3 experiments (ci-coach, daily-fact, daily-security-red-team) are READY by sample count but permanently stuck at guardrail_unsupported; without a native metric source these can never reach PROMOTE/REJECT.
2. Add pairwise/sequential-decision support for ≥3-variant experiments — 14 experiments (output_format across 6 workflows, model_size across 3 workflows, plus tone_variant, semgrep_output_format, timeout_setting, reasoning_depth, sub_agent_strategy variants) are stuck at unsupported_multi_variant; this is the single largest blocked cohort.
3. Promote interaction diagnostics to a core signal for smoke-copilot* — all three factorial (caveman × subagent_model) workflows have sparse 2×2 cells; increase run cadence or reduce concurrent variant count so cells clear min_samples before either sub-experiment nears PROMOTE.

Top 3 eval actions

1. Tighten daily-fact's eval question set to explicitly grade "novelty"/"reaction count" proxies so the multi_candidate hypothesis has a supported observation path instead of relying on unsupported guardrails.
2. Add an eval question to daily-security-red-team that grades false-positive rate directly, mapped to the iterative vs single_pass prompt variants, to unblock run_success_rate/empty_output_rate guardrail support.
3. For output_format experiments (deep-report, daily-code-metrics, daily-compiler-quality, daily-issues-report, copilot-agent-analysis), add a shared eval grading verbosity/readability trade-offs per format so ste vs structured/prose can be compared with a common observation schema.

Decision-pipeline gaps for next PR

• guardrail_unsupported (3 experiments) and insufficient_observations (26 experiments) both point to the same root cause: assignment counts in the state branch vastly exceed min_samples, but per-run outcome-metric observations are not being captured/attached at a comparable rate. This is the top data-pipeline gap to close.
• unsupported_multi_variant (14 experiments, the largest single reason code) indicates the core analyzer needs a K-variant (K≥3) decision path — currently only pairwise control/candidate comparisons are supported.
• Factorial interaction diagnostics (buildFactorialInteractionCells) are currently a reporting-layer-only computation; promote this to a normalized core analysis signal so interaction_underpowered safety holds are enforced deterministically rather than via reporting convention.

Descriptive window: last 30 runs per workflow (where fetched) · Decision thresholds: from decision_policy (min effect 0, confidence 0.95)
Run: 35575352350

Generated by 🧪 daily-experiment-report · copilot · auto · 136.6 AIC · ⌖ 13.4 AIC · ⊞ 16.3K · ◷

• expires on Sep 24, 2026, 12:11 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #62585.