[experiments] Daily Experiment Report — 2026-09-20

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-20

51 experiments analysed across 48 workflows in github/gh-aw. 24 have reached READY sample thresholds, 27 are still COLLECTING. No PROMOTE or REJECT decisions today — every core decision is EXTEND (37) or INCONCLUSIVE (14), driven almost entirely by missing per-run outcome-metric wiring rather than by weak effects.

⚡ Quick Stats

Metric	Value
Active experiments	51
Workflows with experiments	48
Ready for analysis	24
Decisions with sufficient evidence (PROMOTE/REJECT)	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no multi-experiment workflow reached a PROMOTE core decision)
Tracking issues configured	0 (no experiment declares an issue: field — Step 8/9 notifications and labels not applicable this run)

📈 Why nothing reached PROMOTE/REJECT today

Reason code	Count	Meaning
insufficient_observations	26	Sample size reached, but primary-metric per-run observations aren't available to the decision layer (most common gap)
unsupported_multi_variant	14	Experiment has 3+ variants; automatic core decisions currently require exactly 2
insufficient_samples	8	Still below min_samples for at least one variant
guardrail_unsupported	3	A mandatory guardrail metric has no supported native observation source

---

🟢 READY experiments (24) — all EXTEND/INCONCLUSIVE, no usable outcome evidence yet

📋 View all 24 READY experiments

Experiment	Workflow	Variants	n	min_samples	Decision	Reason code
prompt_compression	agentperformanceanalyzer	caveman, verbose	121	20	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	batch, per_scenario	121	20	EXTEND	insufficient_observations
tone_variant	awfailureinvestigator	assertive, clinical, narrative	439	20	INCONCLUSIVE	unsupported_multi_variant
tone_variant	breakingchangechecker	neutral, urgent	46	20	EXTEND	insufficient_observations
prompt_style	cicoach	concise, detailed	91	20	EXTEND	guardrail_unsupported
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent, sub_agents	79	20	EXTEND	insufficient_observations
prompt_style	dailyastrostylelitemarkdownspellcheck	concise, detailed	139	20	EXTEND	insufficient_observations
prompt_style	dailycommunityattribution	concise, verbose	137	20	EXTEND	insufficient_observations
reasoning_depth	dailyfact	multi_candidate, single_pass	91	30	EXTEND	guardrail_unsupported
prompt_style	dailynews	concise, detailed	107	20	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	iterative, single_pass	130	30	EXTEND	guardrail_unsupported
output_format	deepreport	annotated_brief, executive_brief, full_briefing, ste	209	20	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	full_bash, minimal_toolset	118	20	EXTEND	insufficient_observations
prompt_style	issuearborist	concise, detailed	77	20	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	single_agent, sub_agents	208	20	EXTEND	insufficient_observations
caveman	smokecopilot	no, yes	364	20	EXTEND	insufficient_observations
subagent_model	smokecopilot	large, small	344	20	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	no, yes	185	20	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	large, small	185	20	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	no, yes	158	20	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	large, small	158	20	EXTEND	insufficient_observations
sub_agent_strategy	smokegemini	single_agent, sub_agents	268	20	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	parallel_sub_agents, single_agent	76	20	EXTEND	insufficient_observations
tone_style	typist	conversational, formal	84	20	EXTEND	insufficient_observations

Notable evidence signals (descriptive, non-authoritative — core decision unchanged):

• dailyagentrxtraceoptimizer / sub_agent_strategy: balanced (χ2=0.013, p=0.876), single_agent n=39 vs sub_agents n=40 — ready for a real decision as soon as issue_evidence_completeness observations are wired in.
• awfailureinvestigator / tone_variant: 3-way balanced split (χ2=0.087, p=0.947, n=439) — biggest READY experiment by volume, but blocked purely by the 2-variant limitation.
• cicoach, dailyfact, dailysecurityredteam: guardrail metrics (run_success_rate, discussion_reaction_count, false_positive_rate) have no supported native observation source yet — flagged as a data-pipeline gap, not a guardrail pass.

---

🟡 COLLECTING experiments (27)

📋 View all 27 COLLECTING experiments

Experiment	Workflow	Variants	n	min_samples	Decision	Reason code
sub_agent_strategy	architectureguardian	single_agent, sub_agents	38	20	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	phased_sub_agents, single_agent	76	264	EXTEND	insufficient_samples
prompt_style	blogauditor	concise, detailed	10	20	EXTEND	insufficient_samples
output_format	copilotagentanalysis	prose, ste, structured	80	30	INCONCLUSIVE	unsupported_multi_variant
detail_level	dailyarchitecturediagram	brief, comprehensive	17	20	EXTEND	insufficient_samples
model_size	dailycachestrategyanalyzer	gpt-5.3-codex, gpt-5.3-codex-spark	14	20	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	108	20	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	executive_summary, full_detail, ste	124	20	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycompilerquality	concise, detailed, ste	126	20	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	108	20	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	agent, claude-haiku-4.5, claude-sonnet-4.6, claude-sonnet-5, small-agent	105	20	INCONCLUSIVE	unsupported_multi_variant
model_size	dailyfunctionnamer	(<2 variants recorded)	3	20	EXTEND	insufficient_observations
output_format	dailyissuesreport	collapsible, inline, ste	136	20	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	dailyrenderingscriptsverifier	(<2 variants recorded)	1	20	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	(<2 variants recorded)	1	20	EXTEND	insufficient_observations
semgrep_output_format	dailysemgrepscan	bullet_list, prose, structured_sections	77	30	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	default, relaxed, tight	4	20	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	no, yes	15	20	EXTEND	insufficient_samples
summary_detail	dependabotcampaign	brief, detailed	12	20	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	concise, detailed, step_by_step	49	20	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	plan	baseline, deep, shallow	7	20	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef	(<2 variants recorded)	1	20	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	delegated_sequential, inline_strict, single_agent_control	45	30	INCONCLUSIVE	unsupported_multi_variant
prompt_style_test	smokeproject	(<2 variants recorded)	1	20	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	(<2 variants recorded)	1	20	EXTEND	insufficient_observations
model_size	testqualitysentinel	(<2 variants recorded)	6	20	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	eager, lazy	14	20	EXTEND	insufficient_samples

Note: auditworkflows has an unusually high combined min_samples (264) relative to its per-variant target — likely a min_samples × variant-count multiplier worth double-checking in the workflow's experiments: frontmatter.

---

📊 Summary

View Full Experiments Table (all 51)

See the READY and COLLECTING tables above — combined they cover all 51 experiments across 48 workflows. No experiment reached PROMOTE or REJECT this run.

No experiment in this run declares a tracking issue: field, so Step 8 (issue notifications) and Step 9 (lifecycle labels) have nothing to act on today. Adding issue: to active experiments would enable automatic readiness/promotion/guardrail notifications going forward.

🔁 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire primary-metric observations for insufficient_observations experiments (26 of 51). These are sample-size-ready but the core decision layer has no per-run value for metrics like issue_evidence_completeness, token_count_per_run-adjacent proxies, and similar custom metrics. Highest-value targets: dailyagentrxtraceoptimizer (n=79, balanced), smokeantigravity/smokegemini (n=208/268, balanced, high volume).
2. Relax or extend the 2-variant limit for unsupported_multi_variant experiments (14 of 51). awfailureinvestigator (n=439, 3-way balanced, p=0.947) and deepreport (n=209, 4-way) are the largest READY experiments blocked purely by this constraint — prioritize pairwise/ANOVA-style comparison support here.
3. Resolve guardrail_unsupported metrics for cicoach, dailyfact, dailysecurityredteam. run_success_rate, discussion_reaction_count, and false_positive_rate need a supported native observation source so mandatory guardrails can actually pass/fail instead of blocking every decision.

Top 3 eval actions

1. For workflows using prompt_style/tone_variant/output_format experiments (e.g. cicoach, awfailureinvestigator, deepreport, dailycodemetrics), add or tighten eval questions that directly score the declared primary metric (verbosity, tone quality, completeness) per run — today the assignment layer works but downstream eval scoring isn't feeding the decision layer.
2. For sub_agent_strategy/sub_agent_decomposition experiments (agentpersonaexplorer, dailyagentrxtraceoptimizer, smokepi, architectureguardian), tighten evals around evidence completeness and cost, since these are the closest to being decision-ready once observations land.
3. For model_size experiments with 5 variants (dailycavemanoptimizer, dailydochealer, dailydocupdater), consider trimming to 2 primary comparison arms (e.g. best small vs. best large model) so the core analysis pipeline can produce a real decision instead of perpetual INCONCLUSIVE.

Decision-pipeline gaps for the next PR

• insufficient_observations (26 experiments) is the single largest blocker — the core analysis needs a general path from declared metric:/secondary_metrics: names to actual per-run numeric observations (currently many are custom/derived and not captured anywhere).
• unsupported_multi_variant (14 experiments) — extend automatic decisions beyond exactly-2-variant comparisons (e.g. omnibus test + pairwise follow-up) so 3+ variant experiments like awfailureinvestigator and deepreport aren't permanently stuck at INCONCLUSIVE.
• guardrail_unsupported (3 experiments) — no interaction-diagnostic cells were computed this run since no workflow has 2+ simultaneously active experiments with PROMOTE-eligible evidence; once observation wiring lands, revisit whether the factorial interaction helper should become a normalized core signal rather than a reporting-only safety hold.

---

Descriptive window: last 30 runs per workflow (git-branch state used for counts; artifact-based outcome correlation not performed this run since core decisions are still evidence-gated). Decision thresholds: from each experiment's decision_policy in state.json.
Run: 35497964745

Generated by 🧪 daily-experiment-report · copilot · auto · 111.6 AIC · ⌖ 23.4 AIC · ⊞ 16.3K · ◷

• expires on Sep 23, 2026, 12:01 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #62336.