[experiments] Daily Experiment Report — 2026-10-27

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-27

53 experiments analysed across 50 workflows (experiments/* branches). No experiment has decision-quality evidence: 44 are EXTEND, 9 are INCONCLUSIVE. No PROMOTE or REJECT today.

⚡ Quick Stats

Metric	Value
Active experiments	53
🟢 READY / 🟡 COLLECTING	27 / 26
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 44 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 9
Interaction safety holds	0 (only smokecopilot* have 2 experiments; no promote candidates)

Reason codes

reason_code	Count	Meaning
insufficient_observations	17	Metric observations missing
guardrail_unsupported	15	Mandatory guardrail has no supported metric observation
insufficient_samples	12	Variants below min_samples
unsupported_multi_variant	9	3+ variants; core cannot pick candidate (all INCONCLUSIVE)

📋 Full experiments table

Workflow	Experiment	Readiness	Core decision	Reason code	Samples
agentperformanceanalyzer	prompt_compression	🟢	EXTEND	guardrail_unsupported	caveman 85 / verbose 73
agentpersonaexplorer	sub_agent_strategy	🟡	EXTEND	insufficient_observations	–
architectureguardian	sub_agent_strategy	🟡	EXTEND	insufficient_samples	17 / 21 (min 20)
auditworkflows	audit_decomposition	🟡	EXTEND	insufficient_samples	64 / 48 (min 264 obs)
awfailureinvestigator	tone_variant	🟡	EXTEND	insufficient_observations	–
blogauditor	prompt_style	🟡	EXTEND	insufficient_samples	5 / 10 (min 20)
breakingchangechecker	tone_variant	🟡	EXTEND	insufficient_samples	37 / 34 (min 194)
cicoach	prompt_style	🟢	EXTEND	guardrail_unsupported	50 / 67
copilotagentanalysis	output_format	🟡	INCONCLUSIVE	unsupported_multi_variant	44/6/51
copilotprnlpanalysis	nlp_prompt_style	🟡	EXTEND	insufficient_samples	0 / 0
dailyagentrxtraceoptimizer	sub_agent_strategy	🟢	EXTEND	guardrail_unsupported	53 / 53
dailyarchitecturediagram	detail_level	🟡	EXTEND	insufficient_samples	13 / 9 (min 10)
dailyastrostylelitemarkdownspellcheck	prompt_style	🟢	EXTEND	guardrail_unsupported	89 / 86
dailycachestrategyanalyzer	model_size	🟢	EXTEND	guardrail_unsupported	25 / 26
dailycavemanoptimizer	model_size	🟢	EXTEND	guardrail_unsupported	22 / 84
dailycodemetrics	output_format	🟢	INCONCLUSIVE	unsupported_multi_variant	55/59/21
dailycommunityattribution	prompt_style	🟢	EXTEND	insufficient_observations	81 / 81
dailycompilerquality	output_format	🟢	INCONCLUSIVE	unsupported_multi_variant	56/66/22
dailydochealer	model_size	🟢	EXTEND	guardrail_unsupported	25 / 74
dailydocupdater	model_size	🟢	EXTEND	guardrail_unsupported	23 / 58
dailyfact	reasoning_depth	🟢	EXTEND	guardrail_unsupported	67 / 50
dailyfunctionnamer	model_size	🟡	EXTEND	insufficient_observations	–
dailyissuesreport	output_format	🟡	INCONCLUSIVE	unsupported_multi_variant	67/64/23
dailynews	prompt_style	🟢	EXTEND	guardrail_unsupported	71 / 62
dailyrenderingscriptsverifier	remove_redundant_context_v1	🟡	EXTEND	insufficient_samples	0 / 0
dailysafeoutputoptimizer	log_fetch_strategy	🟡	EXTEND	insufficient_observations	–
dailysecurityredteam	reasoning_depth	🟢	EXTEND	guardrail_unsupported	74 / 93
dailysemgrepscan	semgrep_output_format	🟡	INCONCLUSIVE	unsupported_multi_variant	28/27/22
dailysubagentoptimizer	timeout_setting	🟡	INCONCLUSIVE	unsupported_multi_variant	2/1/1
dataflowprdiscussiondataset	caveman_mode	🟡	EXTEND	insufficient_samples	8 / 12
deepreport	output_format	🟢	INCONCLUSIVE	unsupported_multi_variant	71/92/71/54
dependabotcampaign	summary_detail	🟡	EXTEND	insufficient_samples	6 / 6
dependabotgochecker	prompt_style	🟡	INCONCLUSIVE	unsupported_multi_variant	24/21/20
gofan	prompt_tightening_v1	🟡	EXTEND	insufficient_samples	0 / 0
gpclean	tool_verbosity	🟢	EXTEND	guardrail_unsupported	74 / 81
issuearborist	prompt_style	🟢	EXTEND	guardrail_unsupported	62 / 52
plan	reasoning_depth	🟡	INCONCLUSIVE	unsupported_multi_variant	4/2/1
prsouschef	remove_redundant_context_v1	🟡	EXTEND	insufficient_observations	–
smokeantigravity	sub_agent_strategy	🟢	EXTEND	insufficient_observations	99 / 109
smokecopilot	caveman, subagent_model	🟢	EXTEND	insufficient_observations	~210 / ~209 each
smokecopilotaoaiapikey	caveman, subagent_model	🟢	EXTEND	insufficient_observations	~107 each
smokecopilotaoaientra	caveman, subagent_model	🟢	EXTEND	insufficient_observations	93 / 93 each
smokecopilotsubagents	sub_agent_strategy	🟡	EXTEND	insufficient_samples	3 / 1 obs
smokegemini	sub_agent_strategy	🟢	EXTEND	guardrail_unsupported	141 / 156
smokepi	sub_agent_decomposition	🟢	EXTEND	insufficient_observations	42 / 34
smokeproject	prompt_style_test	🟡	EXTEND	insufficient_observations	–
smoketemporaryid	sub_agent_strategy	🟡	EXTEND	insufficient_observations	–
testqualitysentinel	model_size	🟡	EXTEND	insufficient_observations	–
typist	tone_style	🟢	EXTEND	guardrail_unsupported	51 / 59
weeklyblogpostwriter	prefetch_strategy	🟡	EXTEND	insufficient_samples	8 / 12

⚠️ Limitations

• gh aw experiments list failed in this sandbox (branch fetch error); experiment branches were enumerated via the API and each analysed with a locally built gh aw experiments analyze.
• Per-run outcome metrics (success/duration, 95% CI) and charts were not produced this run; core decisions above are canonical. Tracking-issue comments/labels skipped (no promote/guardrail-failure conditions; issue fields not read).

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire supported metric observations for the 15 guardrail_unsupported experiments (e.g. run_success_rate, empty_output_rate) — cicoach, dailynews, gpclean, dailysecurityredteam, model_size family.
2. Fix observation pipeline for 17 insufficient_observations experiments, esp. the READY smoke* ones (no metric declared or no observations recorded).
3. Extend multi-variant support in core analysis (9 INCONCLUSIVE, mostly output_format with ste), or reduce to pairwise comparisons.

Top eval actions

1. Add evals/graders feeding eval:* / grader:* metrics for copilotprnlpanalysis, smokecopilotsubagents, dailyrenderingscriptsverifier, gofan (observations near zero).
2. Add an engagement/quality eval for output_format workflows (discussion_engagement_score) to give ste comparisons observations.
3. Add declared metrics to smoke* experiments (currently metric unset) mapped to their workflow prompts.

Decision-pipeline gaps

• Native metrics unresolved: aic, ai_credits_total, token_count_per_run, discussion_engagement_score.
• Promote smoke caveman×subagent_model interaction diagnostics into a normalized core signal.
• Fix experiments list branch fetching failure.

Run: 40470174176

Generated by 🧪 daily-experiment-report · copilot · auto · ~30 AIC · ⌖ ~1 AIC · ⊞ 17.1K · ◷

• expires on Oct 30, 2026, 12:04 AM UTC-08:00