[experiments] Daily Experiment Report — 2026-09-17

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-17

47 experiment workflow branches are active in github/gh-aw, comprising 51 individual A/B experiments across 48 workflows. None have decision-quality evidence sufficient for a PROMOTE/REJECT verdict yet — core decisions are 24 READY (evidence gates hit but metric pipeline incomplete) and 27 COLLECTING (still accumulating samples). No promotions or rejections today; every deterministic decision remains EXTEND or INCONCLUSIVE.

⚡ Quick Stats

Metric	Value
Active experiments	51 (across 48 workflows)
Ready for analysis	24
Decisions with sufficient evidence (non-sample-gated reason codes)	17
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no PROMOTE decisions exist yet to hold)

---

prompt_style · ci-coach

Readiness: 🟢 READY
Variants: detailed (control) vs concise · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: mann_whitney on token_count_per_run; both mandatory guardrails (run_success_rate, empty_output_rate) are unsupported metrics, blocking a deterministic verdict

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
detailed	52	██████████ 52/20 (100%+)
concise	37	██████████ 37/20 (100%+)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| detailed         |  9   |  77.8%   |    1525.8      | [1046.7 , 2004.9]  | 52/20 (100%+) |
| concise          |  1   | 100.0%   |    1280.0      | N/A (n<2)          | 37/20 (100%+) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: mann_whitney — primary metric token_count_per_run has no supported observation source yet

Guardrails:
  run_success_rate >=0.85 : UNSUPPORTED (no observation source)
  empty_output_rate <=0.05 : UNSUPPORTED (no observation source)

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

tone_style · typist

Readiness: 🟢 READY
Variants: conversational (control) vs formal · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: chi-square balance test only (p=0.38, balanced); primary metric observations unavailable

H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
conversational	45	██████████ 45/20 (100%+)
formal	37	██████████ 37/20 (100%+)

Experiment : tone_style
Workflow   : typist.md
Hypothesis : H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| conversational   |  6   |  83.3%   |    960.0       | [676.4 , 1243.6]   | 45/20 (100%+) |
| formal           |  4   |  75.0%   |    931.3        | [315.6 , 1546.9]   | 37/20 (100%+) |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: chi_square balance test only (χ2=0.78, p=0.38, balanced); discussion_engagement_score primary metric has no supported observation source

Guardrails: none configured

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

📊 Summary

View Full Experiments Table

Experiment	Workflow	Control	Candidate	Readiness	Core decision	Reason code
prompt_compression	agent-performance-analyzer.md	caveman	verbose	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	agent-persona-explorer.md	batch	per_scenario	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	architecture-guardian.md	single_agent	sub_agents	🟡 COLLECTING	EXTEND	insufficient_samples
audit_decomposition	audit-workflows.md	single_agent	phased_sub_agents	🟡 COLLECTING	EXTEND	insufficient_samples
tone_variant	aw-failure-investigator.md	assertive/clinical/narrative	—	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blog-auditor.md	concise	detailed	🟡 COLLECTING	EXTEND	insufficient_samples
tone_variant	breaking-change-checker.md	neutral	urgent	🟢 READY	EXTEND	insufficient_observations
prompt_style	ci-coach.md	detailed	concise	🟢 READY	EXTEND	guardrail_unsupported
output_format	copilot-agent-analysis.md	prose/ste/structured	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	daily-agentrx-trace-optimizer.md	single_agent	sub_agents	🟢 READY	EXTEND	insufficient_observations
detail_level	daily-architecture-diagram.md	brief	comprehensive	🟡 COLLECTING	EXTEND	insufficient_samples
prompt_style	daily-astrostylelite-markdown-spellcheck.md	concise	detailed	🟢 READY	EXTEND	insufficient_observations
model_size	daily-cache-strategy-analyzer.md	gpt-5.3-codex	gpt-5.3-codex-spark	🟡 COLLECTING	EXTEND	insufficient_samples
model_size	daily-caveman-optimizer.md	claude-sonnet-5/haiku/etc	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
output_format	daily-code-metrics.md	executive_summary/full_detail/ste	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prompt_style	daily-community-attribution.md	concise	verbose	🟢 READY	EXTEND	insufficient_observations
output_format	daily-compiler-quality.md	concise/detailed/ste	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	daily-doc-healer.md	claude-sonnet-5/haiku/etc	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	daily-doc-updater.md	claude-sonnet-5/haiku/etc	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	daily-fact.md	single_pass	multi_candidate	🟢 READY	EXTEND	guardrail_unsupported
output_format	daily-issues-report.md	collapsible/inline/ste	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prompt_style	daily-news.md	concise	detailed	🟢 READY	EXTEND	insufficient_observations
remove_redundant_context_v1	daily-rendering-scripts-verifier.md	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
log_fetch_strategy	daily-safe-output-optimizer.md	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
reasoning_depth	daily-security-red-team.md	single_pass	iterative	🟢 READY	EXTEND	guardrail_unsupported
semgrep_output_format	daily-semgrep-scan.md	bullet_list/prose/structured_sections	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflow-pr-discussion-dataset.md	no	yes	🟡 COLLECTING	EXTEND	insufficient_samples
output_format	deep-report.md	full_briefing/executive_brief/annotated_brief/ste	—	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabot-campaign (stale branch, no workflow file)	brief	detailed	🟡 COLLECTING	EXTEND	insufficient_samples
prompt_style	dependabot-go-checker.md	concise/detailed/step_by_step	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean.md	full_bash	minimal_toolset	🟢 READY	EXTEND	insufficient_observations
prompt_style	issue-arborist.md	detailed	concise	🟢 READY	EXTEND	insufficient_observations
reasoning_depth	plan.md	baseline/deep/shallow	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	pr-sous-chef.md	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smoke-antigravity (stale branch, no workflow file)	single_agent	sub_agents	🟢 READY	EXTEND	insufficient_observations
caveman / subagent_model	smoke-copilot.md	no/large	yes/small	🟢 READY	EXTEND	insufficient_observations
caveman / subagent_model	smoke-copilot-aoai-apikey.md	no/large	yes/small	🟢 READY	EXTEND	insufficient_observations
caveman / subagent_model	smoke-copilot-aoai-entra.md	no/large	yes/small	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	smoke-copilot-sub-agents.md	inline_strict/delegated_sequential/single_agent_control	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smoke-gemini.md	single_agent	sub_agents	🟢 READY	EXTEND	insufficient_observations
sub_agent_decomposition	smoke-pi (stale branch, no workflow file)	single_agent	parallel_sub_agents	🟢 READY	EXTEND	insufficient_observations
prompt_style_test	smoke-project.md	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smoke-temporary-id.md	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
model_size	test-quality-sentinel.md	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
tone_style	typist.md	conversational	formal	🟢 READY	EXTEND	insufficient_observations
prefetch_strategy	weekly-blog-post-writer.md	eager/lazy	—	🟡 COLLECTING	EXTEND	insufficient_samples
model_size	daily-function-namer (stale branch)	—	—	🟡 COLLECTING	EXTEND	insufficient_observations
timeout_setting	daily-sub-agent-optimizer (stale branch)	default/relaxed/tight	—	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Unblock guardrail_unsupported verdicts (ci-coach, daily-fact, daily-security-red-team): run_success_rate / empty_output_rate guardrail metrics have no supported observation source — wire these into the metrics pipeline so READY experiments can reach a real PROMOTE/REJECT verdict instead of perpetually re-extending.
2. Resolve unsupported_multi_variant (14 experiments — aw-failure-investigator, copilot-agent-analysis, daily-caveman-optimizer, daily-code-metrics, daily-compiler-quality, daily-doc-healer, daily-doc-updater, daily-issues-report, daily-semgrep-scan, deep-report, dependabot-go-checker, plan, smoke-copilot-sub-agents, and the stale daily-sub-agent-optimizer branch): 3+ variant designs currently cannot produce a core decision at all — prioritize adding a supported ≥3-arm test (e.g. Kruskal-Wallis + pairwise Bonferroni) to core analysis.
3. Clean up stale experiment branches for workflows with no matching .md source (smoke-antigravity, dependabot-campaign, daily-sub-agent-optimizer, smoke-pi, daily-function-namer) — these accumulate runs but can never reach a decision since the workflow definition was removed/renamed.

Top 3 eval actions

1. output_format_adherence eval (used by copilot-agent-analysis, daily-code-metrics, daily-compiler-quality, daily-issues-report, daily-semgrep-scan) should be tightened to also emit a machine-readable per-run score usable as a supported multi-variant metric input, not just a qualitative grade.
2. execution-duration grader-backed experiments (daily-rendering-scripts-verifier, pr-sous-chef) are technically COLLECTING but show total_runs: 1 — verify the grader is actually firing on scheduled runs, not just manual triggers.
3. Map discussion_engagement_score (used by 4 workflows) to a normalized 0-1 eval score so cross-experiment comparison of engagement-driven experiments becomes possible without re-deriving reaction/comment counts per workflow.

Decision-pipeline gaps for next PR

• guardrail_unsupported and insufficient_observations account for 37/51 decisions — the core analyzer has no path from "guardrail declared in frontmatter" to "guardrail backed by a real per-run observation." This is the single largest blocker to any PROMOTE/REJECT verdict in the repo today.
• No interaction diagnostics were computable this run beyond raw cell counts for smoke-copilot* (caveman × subagent_model) — cells are extremely sparse (n=1–4) at the recent_runs window of 10; this needs to become a normalized core signal fed by the full 30-run window rather than left as a reporting-layer helper.

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy
Run: 35196671921

Generated by 🧪 daily-experiment-report · copilot · auto · 219.5 AIC · ⌖ 21.3 AIC · ⊞ 16.9K · ◷

• expires on Sep 20, 2026, 12:09 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #61760.