[experiments] Daily Experiment Report — 2026-09-26

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-26

51 experiments analysed across 48 workflows. 26 are READY for analysis (sample thresholds met), 25 are still COLLECTING samples. No experiment currently has a deterministic PROMOTE or REJECT decision — all decisions are EXTEND (metric-observation gaps) or INCONCLUSIVE (multi-variant, unsupported by pairwise core analysis).

⚡ Quick Stats

Metric	Value
Active experiments	51
Workflows with experiments	48
Ready for analysis	26
Collecting samples	25
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no PROMOTE decisions triggered; 3 multi-experiment workflows checked, all sparse)

---

📋 Reason Code Breakdown (why nothing is READY_FOR_ANALYSIS with a promote/reject)

Reason code	Count	Meaning
insufficient_observations	26	primary metric has no supported observation source yet (metric field declared but not backed by native data or evals)
unsupported_multi_variant	14	core decision engine currently only supports exactly 2 variants; 3+ variant experiments report INCONCLUSIVE
insufficient_samples	8	at least one variant has not reached min_samples yet
guardrail_unsupported	3	a mandatory guardrail metric is declared but not backed by a supported observation source

---

prompt_style · ci-coach

Readiness: 🟢 READY
Variants: concise vs detailed · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Primary metric: token_count_per_run

H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
concise	39	██████████ 39/20 (100%)
detailed	57	██████████ 57/20 (100%)

Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| concise          |    2 |   100.0% |         1262.5 | [-675.2 , 3200.2]  | 2/20 (10%)    |
| detailed         |    8 |    62.5% |         1352.0 | [788.4 , 1915.6]   | 8/20 (40%)    |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: mann_whitney / chi_square p=0.000 (balance test, not outcome significance)

Guardrails:
  mandatory guardrail "run_success_rate" is not backed by a supported metric observation

Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
Report action : EXTEND

Core decision: EXTEND (guardrail_unsupported) — mandatory guardrail "run_success_rate" is not backed by a supported metric observation

---

tool_verbosity · gpclean

Readiness: 🟢 READY
Variants: full_bash vs minimal_toolset · Window: last 30 runs · Analysed: 10 runs with artifacts
min_samples: 20 per variant · Primary metric: aic

H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)

📈 View Detailed Statistics

Sample Sizes & Progress

Variant	Runs	Progress
full_bash	58	██████████ 58/20 (100%)
minimal_toolset	66	██████████ 66/20 (100%)

Experiment : tool_verbosity
Workflow   : gpclean.md
Hypothesis : H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)
Window     : last 30 runs  |  Analysed: 10 runs with artifacts
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         | min_samples   |
+------------------+------+----------+----------------+--------------------+---------------+
| full_bash        |    5 |     0.0% |          279.6 | [259.8 , 299.4]    | 5/20 (25%)    |
| minimal_toolset  |    5 |     0.0% |          275.4 | [252.4 , 298.4]    | 5/20 (25%)    |
+------------------+------+----------+----------------+--------------------+---------------+

Evidence: mann_whitney / chi_square p=0.479 (balance test, not outcome significance)

Guardrails:
    (none declared or not evaluated — see reason below)

Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
Report action : EXTEND

Core decision: EXTEND (insufficient_observations) — primary metric observations are not available for statistical comparison

---

📊 Summary

View Full Experiments Table (all 51 experiments across 48 workflows)

Experiment	Workflow	Variants	Samples	Readiness	Core decision	Reason code
prompt_compression	agent-performance-analyzer	caveman, verbose	caveman=69, verbose=58	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	agent-persona-explorer	batch, per_scenario	batch=65, per_scenario=62	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	architecture-guardian	single_agent, sub_agents	single_agent=17, sub_agents=21	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
audit_decomposition	audit-workflows	phased_sub_agents, single_agent	phased_sub_agents=46, single_agent=36	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
tone_variant	aw-failure-investigator	assertive, clinical, narrative	assertive=157, clinical=154, narrative=156	🟢 READY	⚪ INCONCLUSIVE	unsupported_multi_variant
prompt_style	blog-auditor	concise, detailed	concise=4, detailed=7	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
tone_variant	breaking-change-checker	neutral, urgent	neutral=27, urgent=24	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style	ci-coach	concise, detailed	concise=39, detailed=57	🟢 READY	🟡 EXTEND	guardrail_unsupported
output_format	copilot-agent-analysis	prose, ste, structured	prose=34, ste=3, structured=43	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	daily-agentrx-trace-optimizer	single_agent, sub_agents	single_agent=42, sub_agents=42	🟢 READY	🟡 EXTEND	insufficient_observations
detail_level	daily-architecture-diagram	brief, comprehensive	brief=11, comprehensive=7	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
prompt_style	daily-astrostylelite-markdown-spellcheck	concise, detailed	concise=74, detailed=71	🟢 READY	🟡 EXTEND	insufficient_observations
model_size	daily-cache-strategy-analyzer	gpt-5.3-codex, gpt-5.3-codex-spark	gpt-5.3-codex=10, gpt-5.3-codex-spark=10	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
model_size	daily-caveman-optimizer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	agent=1, claude-haiku-4.5=68, claude-sonnet-4.6=34, claude-sonnet-5=9, small-agent=2	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
output_format	daily-code-metrics	executive_summary, full_detail, ste	executive_summary=54, full_detail=58, ste=16	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
prompt_style	daily-community-attribution	concise, verbose	concise=70, verbose=70	🟢 READY	🟡 EXTEND	insufficient_observations
output_format	daily-compiler-quality	concise, detailed, ste	concise=54, detailed=62, ste=16	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	daily-doc-healer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	agent=2, claude-haiku-4.5=60, claude-sonnet-4.6=43, claude-sonnet-5=8, small-agent=1	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
model_size	daily-doc-updater	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	agent=1, claude-haiku-4.5=46, claude-sonnet-4.6=47, claude-sonnet-5=13, small-agent=2	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	daily-fact	multi_candidate, single_pass	multi_candidate=40, single_pass=56	🟢 READY	🟡 EXTEND	guardrail_unsupported
model_size	daily-function-namer			🟡 COLLECTING	🟡 EXTEND	insufficient_observations
output_format	daily-issues-report	collapsible, inline, ste	collapsible=59, inline=62, ste=21	🟢 READY	⚪ INCONCLUSIVE	unsupported_multi_variant
prompt_style	daily-news	concise, detailed	concise=62, detailed=50	🟢 READY	🟡 EXTEND	insufficient_observations
remove_redundant_context_v1	daily-rendering-scripts-verifier			🟡 COLLECTING	🟡 EXTEND	insufficient_observations
log_fetch_strategy	daily-safe-output-optimizer			🟡 COLLECTING	🟡 EXTEND	insufficient_observations
reasoning_depth	daily-security-red-team	iterative, single_pass	iterative=81, single_pass=55	🟢 READY	🟡 EXTEND	guardrail_unsupported
semgrep_output_format	daily-semgrep-scan	bullet_list, prose, structured_sections	bullet_list=28, prose=27, structured_sections=22	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	default, relaxed, tight	default=2, relaxed=1, tight=1	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflow-pr-discussion-dataset	no, yes	no=6, yes=10	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
output_format	deep-report	annotated_brief, executive_brief, full_briefing, ste	annotated_brief=56, executive_brief=83, full_briefing=58, ste=36	🟢 READY	⚪ INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	brief, detailed	brief=6, detailed=6	🟡 COLLECTING	🟡 EXTEND	insufficient_samples
prompt_style	dependabot-go-checker	concise, detailed, step_by_step	concise=19, detailed=18, step_by_step=15	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	full_bash, minimal_toolset	full_bash=58, minimal_toolset=66	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style	issue-arborist	concise, detailed	concise=45, detailed=38	🟢 READY	🟡 EXTEND	insufficient_observations
reasoning_depth	plan	baseline, deep, shallow	baseline=4, deep=2, shallow=1	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	pr-sous-chef			🟡 COLLECTING	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	single_agent, sub_agents	single_agent=99, sub_agents=109	🟢 READY	🟡 EXTEND	insufficient_observations
caveman	smoke-copilot	no, yes	no=185, yes=185	🟢 READY	🟡 EXTEND	insufficient_observations
subagent_model	smoke-copilot	large, small	large=175, small=175	🟢 READY	🟡 EXTEND	insufficient_observations
caveman	smoke-copilot-aoai-apikey	no, yes	no=94, yes=94	🟢 READY	🟡 EXTEND	insufficient_observations
subagent_model	smoke-copilot-aoai-apikey	large, small	large=94, small=94	🟢 READY	🟡 EXTEND	insufficient_observations
caveman	smoke-copilot-aoai-entra	no, yes	no=81, yes=80	🟢 READY	🟡 EXTEND	insufficient_observations
subagent_model	smoke-copilot-aoai-entra	large, small	large=80, small=81	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smoke-copilot-sub-agents	delegated_sequential, inline_strict, single_agent_control	delegated_sequential=15, inline_strict=14, single_agent_control=19	🟡 COLLECTING	⚪ INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smoke-gemini	single_agent, sub_agents	single_agent=128, sub_agents=143	🟢 READY	🟡 EXTEND	insufficient_observations
sub_agent_decomposition	smoke-pi	parallel_sub_agents, single_agent	parallel_sub_agents=42, single_agent=34	🟢 READY	🟡 EXTEND	insufficient_observations
prompt_style_test	smoke-project			🟡 COLLECTING	🟡 EXTEND	insufficient_observations
sub_agent_strategy	smoke-temporary-id			🟡 COLLECTING	🟡 EXTEND	insufficient_observations
model_size	test-quality-sentinel	claude-haiku-4.5, claude-sonnet-5	claude-haiku-4.5=36, claude-sonnet-5=33	🟢 READY	🟡 EXTEND	insufficient_observations
tone_style	typist	conversational, formal	conversational=47, formal=42	🟢 READY	🟡 EXTEND	insufficient_observations
prefetch_strategy	weekly-blog-post-writer	eager, lazy	eager=5, lazy=10	🟡 COLLECTING	🟡 EXTEND	insufficient_samples

🔀 Multi-Experiment Interaction Diagnostics

Three smoke-test workflows run two simultaneous experiments each (caveman × subagent_model): smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra. Interaction cell counts are highly sparse (1–4 runs per 2×2 cell across all three), well below min_samples (20). interaction_risk_status = SPARSE_CELL_RISK for all three. No PROMOTE decision exists in this report, so the interaction safety hold does not currently override any core decision — this is recorded for forward readiness once sample sizes grow.

🎯 Recommendations

• No experiment is ready for a PROMOTE/REJECT call today; all 51 experiments are either EXTEND (metric-observation pipeline gaps) or INCONCLUSIVE (3+ variant designs unsupported by the pairwise core engine).
• Highest-value next step is closing the metric observation gap: 26 experiments across 45 workflows are gated on insufficient_observations or guardrail_unsupported even though sample thresholds are met (READY).
• 14 experiments (awfailureinvestigator, dailyissuesreport, deepreport, and others) use 3+ variants and will never resolve to PROMOTE/REJECT until the core engine supports multi-variant comparisons (or they are reduced to pairwise designs).

🔧 Self-Tuning Continuation Plan

View continuation plan

Top 3 experiment actions

1. guardrail_unsupported (3 experiments — cicoach, dailyfact, dailysecurityredteam): wire the declared guardrail metrics (run_success_rate, native evals) into the grader/observation pipeline so decision_guardrails.passed can be computed instead of defaulting to unsupported.
2. insufficient_observations (26 experiments, the largest bucket): audit metric: fields (aic, token_count, ai_credits_total, etc.) against what the analyze pipeline can currently ingest from run artifacts vs. eval results, and prioritize native support for the most common ones (aic appears 6+ times).
3. unsupported_multi_variant (14 experiments incl. deepreport, awfailureinvestigator, dailyissuesreport): either extend the core decision engine to support K≥3 variant comparisons with Bonferroni correction (already computed as bonferroni_alpha but unused for a decision), or split these into sequential pairwise experiments.

Top 3 eval actions

1. Several experiments declare eval: prefixed secondary metrics (e.g. copilot-pr-nlp-analysis → eval:insights_report_produced, deep-report → eval:output_format_goal_met). Confirm these graders are actually running and populating results so they can back the primary metric: once promoted from secondary.
2. pr-sous-chef and daily-rendering-scripts-verifier use grader:execution-duration as the primary metric — verify this custom grader source is registered and producing per-run observations distinct from run_duration_ms.
3. Tighten eval questions for workflows stuck on guardrail_unsupported (daily-fact, daily-security-red-team) so guardrail pass/fail can be derived from an eval-backed proxy while native support is built.

Decision-pipeline gaps for next PR

• Native metric support is the single largest blocker: 26/51 experiments are READY on samples but blocked purely on insufficient_observations/guardrail_unsupported.
• Multi-variant (K≥3) decision support is the second largest blocker (14/51 experiments); chi_square/bonferroni_alpha are already computed but not consumed by the decision layer.
• Interaction diagnostics (factorial cells for smokecopilot* workflows) should become a normalized core analysis signal rather than a report-only computation, so SPARSE_CELL_RISK can gate PROMOTE decisions automatically instead of via prose.

Descriptive window: last 30 runs per workflow (377 total run-variant records matched across 48 workflows) · Decision thresholds: from decision_policy (95% confidence, 0 minimum effect / regression tolerance as configured)
Run: 36228023293

Generated by 🧪 daily-experiment-report · copilot · auto · 154.8 AIC · ⌖ 8.67 AIC · ⊞ 16.3K · ◷

• expires on Sep 29, 2026, 12:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #63805.