[experiments] Daily Experiment Report — 2026-09-24

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-24

51 experiments analysed across 48 workflow branches. 26 have reached min_samples (READY); the remaining 25 are still collecting. No experiment has a deterministic PROMOTE or REJECT decision yet — every core decision is EXTEND (37) or INCONCLUSIVE (14), mostly because outcome-metric plumbing (guardrail_unsupported, insufficient_observations) or ≥3-variant support (unsupported_multi_variant) is not yet wired into the decision layer.

⚡ Quick Stats

Metric	Value
Active experiments (workflow × experiment)	51
Ready for analysis	26
Decisions with sufficient evidence for automatic promotion	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	3 (smoke-copilot 2×2 caveman × subagent_model factorials)
Workflows with tracking issues notified today	11
Stale experiment branches (workflow no longer declares experiments:)	6

Descriptive window: last 10 recorded runs per experiment (from recent_runs), cross-referenced against the last 30 completed Action runs. Decision source of truth: gh aw experiments analyze.

---

prompt_style · ci-coach.md

Readiness: 🟢 READY · Variants: detailed vs concise · Analysed: 10/10 recent runs matched
min_samples: 20 per variant (detailed=56, concise=38) · Evidence: mann_whitney on token_count_per_run, guardrails unsupported

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
detailed	56	██████████ 56/20 (100%)
concise	38	██████████ 38/20 (100%)

Descriptive outcomes (last 10 recent runs)

Variant	n	Succ %	Mean dur (s)	95% CI (s)
detailed	9	66.7%	1334.6	[854.6, 1814.5]
concise	1	100.0%	1110.0	N/A (n<2)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise reduces tokens ≥25%
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| detailed         |   9  |  66.7%   |   1334.6       | [854.6 , 1814.5]   | 56/20 (100%)  |
| concise          |   1  | 100.0%   |   1110.0       | N/A (n<2)          | 38/20 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: mann_whitney on token_count_per_run — p_value not surfaced (native metric not observed yet)

Guardrails:
  run_success_rate >=0.85 : UNSUPPORTED (no native observation pipeline)
  empty_output_rate <=0.05 : UNSUPPORTED (no native observation pipeline)

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail run_success_rate has no supported metric observation yet; sample size is sufficient (94 runs) but the guardrail pipeline is the blocker.

---

reasoning_depth · daily-fact.md

Readiness: 🟢 READY · Variants: single_pass vs multi_candidate · Analysed: 10/10 recent runs matched
min_samples: 30 per variant (single_pass=55, multi_candidate=39) · Evidence: guardrail unsupported

H0: no change in discussion engagement rate. H1: multi_candidate produces more novel/verified facts

⚠️ Data-pipeline gap: all 10 most-recent runs across both variants show conclusion=failure in the Actions API — 0% success in the descriptive sample despite insufficient_observations/guardrail_unsupported masking it in the core decision. This looks like an infrastructure or workflow regression independent of the A/B assignment and should be triaged before drawing any A/B conclusion.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_pass	55	██████████ 55/30 (100%)
multi_candidate	39	██████████ 39/30 (100%)

Descriptive outcomes (last 10 recent runs)

Variant	n	Succ %	Mean dur (s)	95% CI (s)
single_pass	4	0.0%	429.5	[167.1, 691.9]
multi_candidate	6	0.0%	357.5	[300.3, 414.7]

Experiment : reasoning_depth
Workflow   : daily-fact.md
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel facts
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_pass      |   4  |   0.0%   |    429.5       | [167.1 , 691.9]    | 55/30 (100%)  |
| multi_candidate  |   6  |   0.0%   |    357.5       | [300.3 , 414.7]    | 39/30 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: descriptive metric is discussion_reaction_count; native observation not yet wired

Guardrails:
  empty_output_rate <0.05 : UNSUPPORTED
  run_success_rate >=0.95 : UNSUPPORTED (descriptively 0/10 succeeded — investigate before trusting)

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — sample size sufficient (94 runs) but flagged: recent-run success rate is 0% for both variants, which likely reflects an infrastructure/workflow break, not an experiment signal.

---

reasoning_depth · daily-security-red-team.md

Readiness: 🟢 READY · Variants: single_pass vs iterative · Analysed: 10/10 recent runs matched
min_samples: 30 per variant (single_pass=54, iterative=80) · Evidence: guardrail unsupported

H0: no change in finding quality. H1: iterative reduces false-positive rate

⚠️ Data-pipeline gap: all 10 most-recent runs for both variants show conclusion=cancelled. This is very likely a timeout/cancellation pattern (mean duration ≈3800s ≈63 min in both variants) unrelated to the reasoning-depth assignment, and should be investigated as an infra issue before any A/B read.

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
single_pass	54	██████████ 54/30 (100%)
iterative	80	██████████ 80/30 (100%)

Descriptive outcomes (last 10 recent runs — all cancelled)

Variant	n	Succ %	Mean dur (s)	95% CI (s)
single_pass	6	0.0% (all cancelled)	3813.8	[3773.4, 3854.3]
iterative	4	0.0% (all cancelled)	3822.3	[3774.8, 3869.7]

Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 30 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| single_pass      |   6  |   0.0%   |   3813.8       | [3773.4 , 3854.3]  | 54/30 (100%)  |
| iterative        |   4  |   0.0%   |   3822.3       | [3774.8 , 3869.7]  | 80/30 (100%)  |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: metric false_positive_rate; native observation unsupported

Guardrails:
  run_success_rate >=0.9 : UNSUPPORTED (descriptively 0/10 succeeded, all cancelled — near-identical durations suggest a shared timeout)
  empty_output_rate <=0.05 : UNSUPPORTED

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — flagged: all 10 recent runs cancelled at ~63 minutes in both variants, consistent with a timeout-minutes ceiling being hit regardless of assignment. Recommend checking the workflow's timeout budget before continuing this experiment.

---

🟢 Other READY experiments (2-variant, `insufficient_observations`)

These experiments have reached min_samples but the decision layer still requires more matched outcome observations (native metric pipeline gaps), not more runs. EXTEND here means "keep collecting outcome telemetry," not "keep running more assignments."

Workflow	Experiment	Variants (n)	Descriptive succ %	Issue
agent-performance-analyzer.md	prompt_compression	caveman=67, verbose=58	caveman 20% / verbose 60% (n=5 each)	#33280
agent-persona-explorer.md	sub_agent_strategy	batch=65, per_scenario=60	batch 60% / per_scenario 20% (n=5 each)	—
breaking-change-checker.md	tone_variant	urgent=24, neutral=25	urgent 28.6% / neutral 0% (n=7/3)	#42467
daily-agentrx-trace-optimizer.md	sub_agent_strategy	single_agent=42, sub_agents=41	—	—
daily-astrostylelite-markdown-spellcheck.md	prompt_style	concise=73, detailed=70	—	—
daily-community-attribution.md	prompt_style	concise=70, verbose=70	—	—
daily-news.md	prompt_style	concise=60, detailed=50	concise 100% / detailed 100% (n=7/3)	#31190
gpclean.md	tool_verbosity	minimal_toolset=64, full_bash=58	0% both (n=4/6) ⚠️	—
issue-arborist.md	prompt_style	concise=44, detailed=37	concise 20% / detailed 0% (n=5 each)	#30015
smoke-antigravity.md*	sub_agent_strategy	single_agent=99, sub_agents=109	—	—
smoke-copilot.md	caveman	no=184, yes=185	see interaction cells below	—
smoke-copilot.md	subagent_model	small=175, large=174	see interaction cells below	—
smoke-copilot-aoai-apikey.md	caveman	no=93, yes=94	see interaction cells below	—
smoke-copilot-aoai-apikey.md	subagent_model	small=93, large=94	see interaction cells below	—
smoke-copilot-aoai-entra.md	caveman	no=80, yes=80	see interaction cells below	—
smoke-copilot-aoai-entra.md	subagent_model	small=80, large=80	see interaction cells below	—
smoke-gemini.md	sub_agent_strategy	single_agent=127, sub_agents=143	—	—
smoke-pi.md*	sub_agent_decomposition	single_agent=34, parallel_sub_agents=42	100% both (n=5 each), parallel slower (259s vs 199s)	—
test-quality-sentinel.md	model_size	claude-sonnet-5=25, claude-haiku-4.5=27	100% both (n=6/4); haiku slower (1580s vs 972s)	#43530
typist.md	tone_style	conversational=46, formal=41	conversational 75% / formal 50% (n=4/6)	#34032

* = workflow no longer declares experiments: in frontmatter but the branch has residual history; treat as concluded/stale.

---

🔀 Factorial interaction: caveman × subagent_model (smoke-copilot family)

Three workflows (smoke-copilot.md, smoke-copilot-aoai-apikey.md, smoke-copilot-aoai-entra.md) run two simultaneous 2-variant experiments. Both individual experiments are core EXTEND (insufficient_observations), but a 2×2 interaction diagnostic was computed from the last 10 matched runs per workflow:

Workflow	Cell (caveman, subagent)	n	Succ %	interaction_p_value	interaction_risk_status
smoke-copilot	(no, large)	3	33.3%	0.519	🔴 SPARSE_CELL_RISK
smoke-copilot	(no, small)	2	50.0%
smoke-copilot	(yes, large)	1	100.0%
smoke-copilot	(yes, small)	4	25.0%
smoke-copilot-aoai-apikey	(no, large)	1	0.0%	0.206	🔴 SPARSE_CELL_RISK
smoke-copilot-aoai-apikey	(no, small)	4	25.0%
smoke-copilot-aoai-apikey	(yes, large)	4	25.0%
smoke-copilot-aoai-apikey	(yes, small)	1	0.0%
smoke-copilot-aoai-entra	(no, large)	4	0.0%	0.206	🔴 SPARSE_CELL_RISK
smoke-copilot-aoai-entra	(no, small)	1	0.0%
smoke-copilot-aoai-entra	(yes, large)	1	0.0%
smoke-copilot-aoai-entra	(yes, small)	4	25.0%

All 12 cells have n < 20 (min_samples). core_decision: EXTEND for both underlying experiments in all three workflows, so the sparse-cell safety hold does not change anything today (it only downgrades a would-be PROMOTE), but is logged here since these are the only multi-experiment factorial workflows in the fleet.

---

⚪ INCONCLUSIVE experiments (3–4 variants, `unsupported_multi_variant`)

The core decision engine currently only computes automatic PROMOTE/REJECT for exactly 2 variants. These 6 experiments have ≥3 variants and enough samples, but return INCONCLUSIVE / unsupported_multi_variant by design — they need either a K-variant decision extension or manual analyst review.

Workflow	Experiment	Variants (counts)	Balance (χ2, p)	Issue
aw-failure-investigator.md	tone_variant	assertive=154, clinical=154, narrative=150	χ2=0.07, p=0.954 (balanced)	#36105
daily-issues-report.md	output_format	collapsible=59, inline=61, ste=20	χ2=22.9, p<0.001 (imbalanced — ste under-assigned)	#30573
deep-report.md	output_format	annotated_brief=53, full_briefing=58, ste=32, executive_brief=82	imbalanced	—
copilot-agent-analysis.md	output_format	structured=43, prose=34, ste=3	ste far below min_samples (30) despite total_runs≥min	—
daily-caveman-optimizer.md	model_size	5 variants incl. legacy agent/small-agent labels	stale variant labels present	—
daily-code-metrics.md	output_format	full_detail=58, executive_summary=54, ste=16	ste below min_samples	#1 (likely placeholder/typo — not a valid tracking issue)
daily-compiler-quality.md	output_format	concise=54, detailed=61, ste=15	—	#32390
daily-doc-healer.md / daily-doc-updater.md	model_size	5 variants incl. legacy labels	stale variant labels present	—
daily-semgrep-scan.md	semgrep_output_format	bullet_list=28, prose=27, structured_sections=22	—	#32795
dependabot-go-checker.md	prompt_style	concise=18, detailed=18, step_by_step=15	—	—
plan.md	reasoning_depth	shallow=1, baseline=4, deep=2	far below min_samples (20)	#42941
smoke-copilot-sub-agents.md	sub_agent_strategy	inline_strict=13, delegated_sequential=15, single_agent_control=19	—	#47551

---

🟡 COLLECTING experiments (below `min_samples`)

Workflow	Experiment	Progress	Issue
audit-workflows.md	audit_decomposition	phased_sub_agents 44/264 (17%), single_agent 36/264 (14%)	#43177
blog-auditor.md	prompt_style	concise 4/20 (20%), detailed 7/20 (35%)	—
daily-architecture-diagram.md	detail_level	brief 11/20 (55%), comprehensive 7/20 (35%)	#31926
daily-cache-strategy-analyzer.md	model_size	9/20 each (45%)	—
daily-safe-output-optimizer.md	log_fetch_strategy	1 run total	#38094
dataflow-pr-discussion-dataset.md	caveman_mode	no 6/20 (30%), yes 9/20 (45%)	#37102
dependabot-campaign (removed workflow)	summary_detail	6/20 each (30%)	—
daily-rendering-scripts-verifier.md / pr-sous-chef.md	remove_redundant_context_v1	1 run each	—
weekly-blog-post-writer.md	prefetch_strategy	eager 5/20 (25%), lazy 10/20 (50%)	#38590
daily-function-namer / architecture-guardian (removed workflows)	model_size / sub_agent_strategy	low n, stale	—

📊 Summary

View Full Experiments Table (READY only)

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
prompt_style	ci-coach.md	detailed	concise	mann_whitney / token_count_per_run	UNSUPPORTED	EXTEND	EXTEND
reasoning_depth	daily-fact.md	single_pass	multi_candidate	discussion_reaction_count	UNSUPPORTED (⚠️ 0% succ)	EXTEND	EXTEND
reasoning_depth	daily-security-red-team.md	single_pass	iterative	false_positive_rate	UNSUPPORTED (⚠️ all cancelled)	EXTEND	EXTEND
prompt_compression	agent-performance-analyzer.md	verbose	caveman	aic	UNSUPPORTED	EXTEND	EXTEND
tone_variant	breaking-change-checker.md	neutral	urgent	issue_engagement_rate	n/a	EXTEND	EXTEND
prompt_style	daily-news.md	detailed	concise	aic	n/a	EXTEND	EXTEND
prompt_style	issue-arborist.md	concise	detailed	links_created	n/a	EXTEND	EXTEND
model_size	test-quality-sentinel.md	claude-sonnet-5	claude-haiku-4.5	review_usefulness_acceptance_rate	n/a	EXTEND	EXTEND
tone_style	typist.md	formal	conversational	discussion_engagement_score	n/a	EXTEND	EXTEND
tone_variant	aw-failure-investigator.md	—	3-variant	balanced (p=0.954)	n/a	INCONCLUSIVE	INCONCLUSIVE
output_format	daily-issues-report.md	—	3-variant	imbalanced (p<0.001)	n/a	INCONCLUSIVE	INCONCLUSIVE

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Investigate daily-fact.md (0/10 recent runs succeeded) and daily-security-red-team.md (10/10 recent runs cancelled, ~63 min each) as likely infra/timeout regressions unrelated to their A/B assignments — file/triage before trusting either experiment's future decision.
2. Fix the invalid tracking-issue reference issue: 1 on daily-code-metrics.md's output_format experiment frontmatter (points to an unrelated closed issue) so lifecycle notifications route correctly.
3. Extend the core decision engine to support ≥3-variant experiments (currently unsupported_multi_variant for 6 workflows including deep-report.md, copilot-agent-analysis.md, daily-issues-report.md) — these have enough samples and, in daily-issues-report.md's case, a statistically significant assignment imbalance (p<0.001) worth a native K-variant test.

Top 3 eval actions

1. copilot-agent-analysis.md, daily-code-metrics.md, daily-compiler-quality.md, deep-report.md, and daily-issues-report.md all use an ste (Simplified Technical English) output_format variant that is consistently under-assigned relative to peers (e.g. 3/34/43 for copilot-agent-analysis) — tighten weighting or an eval question asking "was the ste variant actually selected and rendered distinctly from structured/prose?" to confirm the variant isn't silently degrading to a default template.
2. For daily-fact.md and daily-security-red-team.md, add an eval question mapped to their prompts asking "did the run complete without failure/cancellation?" so guardrail-blocking infra issues surface as a graded observation rather than only via manual Action-run inspection.
3. daily-caveman-optimizer.md / daily-doc-healer.md / daily-doc-updater.md carry stale variant labels (agent, small-agent) alongside current ones (claude-sonnet-5, claude-haiku-4.5) — add an eval/grading check that flags variant-label drift when experiments: frontmatter is edited without a corresponding branch-state migration.

Decision-pipeline gaps for the next PR

• guardrail_unsupported blocks 3 READY, high-sample experiments (ci-coach, daily-fact, daily-security-red-team) purely because run_success_rate / empty_output_rate have no native metric observation wired — this is the single highest-leverage fix to unblock automatic decisions today.
• unsupported_multi_variant blocks 14 experiments (6 workflows) from ever reaching PROMOTE/REJECT; the 2-cell factorial interaction helper built for this report (§ smoke-copilot family) should become a normalized core analysis signal rather than a reporting-only diagnostic, especially once K-variant support lands.
• 6 experiment branches (smoke-antigravity, dependabot-campaign, daily-subagent-optimizer, architecture-guardian, daily-function-namer, smoke-pi) have historical state on experiments/* branches but no matching experiments: frontmatter in any current workflow file — these should be explicitly archived/pruned or their frontmatter restored, otherwise gh aw experiments list will keep surfacing stale data indefinitely.

Descriptive window: last 30 runs per workflow (10 matched via recent_runs) · Decision thresholds: from decision_policy (confidence 0.95, minimum_effect 0, regression_tolerance 0)
Run: 35971955572

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

• files.pythonhosted.org
• github.com
• o205451.ingest.us.sentry.io
• proxy.golang.org
• pypi.org
• storage.googleapis.com
• sum.golang.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "files.pythonhosted.org"
    - "github.com"
    - "o205451.ingest.us.sentry.io"
    - "proxy.golang.org"
    - "pypi.org"
    - "storage.googleapis.com"
    - "sum.golang.org"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · copilot · auto · 225.4 AIC · ⌖ 21.2 AIC · ⊞ 16.4K · ◷

• expires on Sep 27, 2026, 12:09 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #63394.