[experiments] Daily Experiment Report — 2026-09-15

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-15

51 experiments analysed across 48 workflows. 24 have reached min_samples and are core-analysis ready; none show a deterministic PROMOTE/REJECT outcome yet — all resolved analyses land on EXTEND (guardrail/observation data gaps) or INCONCLUSIVE (3+ variant experiments, which the core decision engine does not yet auto-resolve). One 2-variant interaction pair (smokecopilot caveman × subagent_model) is sparse and held from promotion consideration.

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis	24
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	1

---

caveman · smokecopilot

Readiness: 🟢 READY
Variants: no vs yes · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square p=0.9118112136945463

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
no	181	██████████ 181/20 (100%)
yes	180	██████████ 180/20 (100%)

Experiment : caveman
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+------------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples            |
+------------------+------+----------+----------------+--------------------+------------------------+
| no               |    5 |    40.0% |          680.8 | [428.7 , 932.9]    | ██████████ 181/20 (100%) |
| yes              |    5 |    20.0% |          662.6 | [460.5 , 864.7]    | ██████████ 180/20 (100%) |
+------------------+------+----------+----------------+--------------------+------------------------+

Evidence: chi_square p_value=0.9118112136945463, is_balanced=True

Guardrails: (none declared)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

Report action: EXTEND (interaction_underpowered) — Interaction cells for caveman × subagent_model are sparse (min cell n=1, target min_samples=20); interaction_p_value=0.206, interaction_risk_status=SPARSE_CELL_RISK. This is a reporting safety hold pending normalized interaction analysis in core.

---

prompt_style · dailycommunityattribution

Readiness: 🟢 READY
Variants: concise vs verbose · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Evidence: chi_square p=1

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	66	██████████ 66/20 (100%)
verbose	66	██████████ 66/20 (100%)

Experiment : prompt_style
Workflow   : daily-community-attribution.md
Hypothesis : (not specified)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+------------------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples            |
+------------------+------+----------+----------------+--------------------+------------------------+
| concise          |    5 |    60.0% |          642.0 | [605.6 , 678.4]    | ██████████ 66/20 (100%) |
| verbose          |    5 |    80.0% |          605.2 | [493.6 , 716.8]    | ██████████ 66/20 (100%) |
+------------------+------+----------+----------------+--------------------+------------------------+

Evidence: chi_square p_value=1, is_balanced=True

Guardrails: (none declared)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

[Repeat pattern above applies to all remaining experiments — summarized in the compact table below]

📊 Summary

View Full Experiments Table

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
tone_variant	aw-failure-investigator	(multi-variant)	—	chi2 p=0.879	N/A	INCONCLUSIVE	INCONCLUSIVE
caveman	smoke-copilot	no	yes	chi2 p=0.912	N/A	EXTEND	EXTEND (interaction hold)
subagent_model	smoke-copilot	large	small	chi2 p=0.911	N/A	EXTEND	EXTEND
sub_agent_strategy	smoke-gemini	single_agent	sub_agents	chi2 p=0.395	N/A	EXTEND	EXTEND
sub_agent_strategy	smokeantigravity (decommissioned)	single_agent	sub_agents	chi2 p=0.495	N/A	EXTEND	EXTEND
output_format	deep-report	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
caveman	smoke-copilot-aoai-apikey	no	yes	chi2 p=0.899	N/A	EXTEND	EXTEND
subagent_model	smoke-copilot-aoai-apikey	large	small	chi2 p=0.899	N/A	EXTEND	EXTEND
caveman	smoke-copilot-aoai-entra	no	yes	chi2 p=1.000	N/A	EXTEND	EXTEND
subagent_model	smoke-copilot-aoai-entra	large	small	chi2 p=1.000	N/A	EXTEND	EXTEND
prompt_style	daily-astrostylelite-markdown-spellcheck	concise	detailed	chi2 p=0.839	N/A	EXTEND	EXTEND
output_format	daily-issues-report	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	daily-community-attribution	concise	verbose	chi2 p=1.000	N/A	EXTEND	EXTEND
reasoning_depth	daily-security-red-team	single_pass	iterative	chi2 p=0.012	UNSUPPORTED	EXTEND	EXTEND
output_format	daily-compiler-quality	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
output_format	daily-code-metrics	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_compression	agent-performance-analyzer	caveman	verbose	chi2 p=0.356	N/A	EXTEND	EXTEND
sub_agent_strategy	agent-persona-explorer	batch	per_scenario	chi2 p=0.585	N/A	EXTEND	EXTEND
tool_verbosity	gpclean	full_bash	minimal_toolset	chi2 p=0.402	N/A	EXTEND	EXTEND
model_size	daily-caveman-optimizer	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style	daily-news	concise	detailed	chi2 p=0.379	N/A	EXTEND	EXTEND
model_size	daily-doc-healer	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
model_size	daily-doc-updater	(multi-variant)	—	chi2 p=0.000	N/A	INCONCLUSIVE	INCONCLUSIVE
reasoning_depth	daily-fact	single_pass	multi_candidate	chi2 p=0.052	UNSUPPORTED	EXTEND	EXTEND
prompt_style	ci-coach	detailed	concise	chi2 p=0.000	UNSUPPORTED	EXTEND	EXTEND
output_format	copilot-agent-analysis	(multi-variant)	—	chi2 p=0.000	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
tone_style	typist	conversational	formal	chi2 p=0.510	N/A	EXTEND	EXTEND
semgrep_output_format	daily-semgrep-scan	(multi-variant)	—	chi2 p=0.023	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	daily-agentrx-trace-optimizer	single_agent	sub_agents	chi2 p=1.000	N/A	EXTEND	EXTEND
sub_agent_decomposition	smoke-pi	parallel_sub_agents	single_agent	chi2 p=0.362	N/A	EXTEND	EXTEND
audit_decomposition	audit-workflows	single_agent	phased_sub_agents	chi2 p=0.348	UNSUPPORTED	EXTEND	EXTEND
prompt_style	issue-arborist	concise	detailed	chi2 p=0.348	N/A	EXTEND	EXTEND
prompt_style	dependabot-go-checker	(multi-variant)	—	chi2 p=0.716	N/A	INCONCLUSIVE	INCONCLUSIVE
sub_agent_strategy	smoke-copilot-sub-agents	(multi-variant)	—	chi2 p=0.694	UNSUPPORTED	INCONCLUSIVE	INCONCLUSIVE
tone_variant	breaking-change-checker	neutral	urgent	chi2 p=0.752	N/A	EXTEND	EXTEND
sub_agent_strategy	architecture-guardian	single_agent	sub_agents	chi2 p=0.524	N/A	EXTEND	EXTEND
detail_level	daily-architecture-diagram	brief	comprehensive	chi2 p=0.223	N/A	EXTEND	EXTEND
prefetch_strategy	weekly-blog-post-writer	eager	lazy	chi2 p=0.285	N/A	EXTEND	EXTEND
caveman_mode	dataflow-pr-discussion-dataset	no	yes	chi2 p=0.285	N/A	EXTEND	EXTEND
summary_detail	dependabotcampaign (decommissioned)	brief	detailed	chi2 p=1.000	N/A	EXTEND	EXTEND
prompt_style	blog-auditor	concise	detailed	chi2 p=0.319	N/A	EXTEND	EXTEND
model_size	daily-cache-strategy-analyzer	gpt-5.3-codex	gpt-5.3-codex-spark	chi2 p=0.736	UNSUPPORTED	EXTEND	EXTEND
reasoning_depth	plan	(multi-variant)	—	chi2 p=0.369	N/A	INCONCLUSIVE	INCONCLUSIVE
timeout_setting	dailysubagentoptimizer (decommissioned)	(multi-variant)	—	chi2 p=0.781	N/A	INCONCLUSIVE	INCONCLUSIVE
prompt_style_test	smoke-project	(multi-variant)	—	chi2 p=0.000	N/A	EXTEND	EXTEND
remove_redundant_context_v1	daily-rendering-scripts-verifier	(multi-variant)	—	chi2 p=0.000	N/A	EXTEND	EXTEND
log_fetch_strategy	daily-safe-output-optimizer	eager	lazy	chi2 p=0.000	UNSUPPORTED	EXTEND	EXTEND
sub_agent_strategy	smoke-temporary-id	single_agent	sub_agents	chi2 p=0.000	UNSUPPORTED	EXTEND	EXTEND
remove_redundant_context_v1	pr-sous-chef	(multi-variant)	—	chi2 p=0.000	N/A	EXTEND	EXTEND
model_size	daily-function-namer	(multi-variant)	—	chi2 p=0.000	N/A	EXTEND	EXTEND
model_size	test-quality-sentinel	(multi-variant)	—	chi2 p=0.000	N/A	EXTEND	EXTEND

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy (minimum_effect=0, confidence=0.95) · 3 experiment branches (smokeantigravity, dependabotcampaign, dailysubagentoptimizer) belong to workflows removed from the repo; their data is historical only.
Run: 34943854343

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. dailysecurityredteam / reasoning_depth (READY, 126 runs) — guardrails run_success_rate and empty_output_rate are guardrail_unsupported; wire native metric emission so the guardrail-backed PROMOTE/REJECT path can activate (10/10 sampled runs were cancelled, which itself warrants investigation into scheduling/dedup behavior independent of the experiment).
2. awfailureinvestigator / tone_variant and deepreport / output_format (both READY, 3–4 variants) — core decision engine currently reports INCONCLUSIVE / unsupported_multi_variant for any experiment with more than two variants. Prioritize extending core analysis to K-variant (ANOVA/Kruskal-Wallis style) decisions so these long-running, well-powered experiments (418 and 189 runs respectively) can resolve.
3. smokecopilot caveman × subagent_model factorial pair — interaction cells are sparse (min cell n=1 vs min_samples=20); continue collecting joint assignments before considering promotion of either factor in isolation.

Top 3 eval actions

1. Guardrail-backed experiments (dailyfact, cicoach, dailysecurityredteam) declare guardrails (empty_output_rate, run_success_rate) that are unsupported — add graders/evals that emit these as native metric observations so decision_guardrails can move from configured=True to a pass/fail verdict.
2. Experiments using insufficient_observations as their reason code (33 of 51) indicate the primary metric: field (e.g. discussion_engagement_score, effective_tokens, token_count_per_run) is not yet populated by an eval/grader pipeline for these workflows — tighten eval question mappings so primary-metric observations flow into gh aw experiments analyze for READY experiments first (see list above).
3. For output_format / semgrep_output_format style experiments (deepreport, dailycodemetrics, dailyissuesreport, dailysemgrepscan, dailycompilerquality), add secondary-metric evals for output_length_chars / output_token_count so verbosity trade-offs are visible even while the primary K-variant decision remains INCONCLUSIVE.

Decision-pipeline gaps for the next PR

• guardrail_unsupported (3 experiments) and insufficient_observations (33 experiments) are the two dominant reason codes blocking every READY experiment from a deterministic PROMOTE/REJECT — this is a metric-plumbing gap, not a statistical-power gap, since most of these experiments already exceed min_samples.
• unsupported_multi_variant (14 experiments, including two of the largest by sample size) is a core-analysis gap: extend the decision engine beyond two-variant comparisons.
• Factorial interaction diagnostics (chi-square over 2D contingency tables) are currently only computed here in the reporting layer; promote this to a normalized core analysis signal so interaction_underpowered can become a first-class reason_code rather than a report-only override.

Generated by 🧪 daily-experiment-report · copilot · auto · 217.1 AIC · ⌖ 11.2 AIC · ⊞ 16.7K · ◷

• expires on Sep 18, 2026, 12:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #61311.