[experiments] Daily Experiment Report — 2026-10-14

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-14

52 experiments across 49 workflows were analysed from the experiments/* state branches. No experiment has a PROMOTE or REJECT decision. 22 are READY on sample count, but none has decision-quality evidence yet.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready (sample gate)	25
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (multi-experiment workflows: smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra; neither is PROMOTE, so no hold is needed)

Reason-code breakdown

Reason code	Count	Meaning
insufficient_observations	26	Assignments exist, but no usable metric observations were recorded
unsupported_multi_variant	14	Auto-decision needs exactly 2 variants
insufficient_samples	9	Below min_samples
guardrail_unsupported	3	cicoach, dailyfact, dailysecurityredteam: mandatory guardrail metric unsupported (data-pipeline gap)

📈 All experiments (core decisions, verbatim from `gh aw experiments analyze`)

Workflow	Experiment	Samples per variant	Readiness	Core decision	Reason code
agentperformanceanalyzer	prompt_compression	caveman:80, verbose:65	🟢 READY	EXTEND	insufficient_observations
agentpersonaexplorer	sub_agent_strategy		🟡 COLLECTING	EXTEND	insufficient_observations
architectureguardian	sub_agent_strategy	single_agent:17, sub_agents:21	🟡 COLLECTING	EXTEND	insufficient_samples
auditworkflows	audit_decomposition	phased_sub_agents:58, single_agent:41	🟡 COLLECTING	EXTEND	insufficient_samples
awfailureinvestigator	tone_variant	assertive:228, clinical:208, narrative:202	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
blogauditor	prompt_style	concise:4, detailed:9	🟡 COLLECTING	EXTEND	insufficient_samples
breakingchangechecker	tone_variant	neutral:32, urgent:30	🟢 READY	EXTEND	insufficient_observations
cicoach	prompt_style	concise:46, detailed:62	🟢 READY	EXTEND	guardrail_unsupported
copilotagentanalysis	output_format	prose:38, ste:4, structured:46	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
copilotprnlpanalysis	nlp_prompt_style	concise:0, structured:0	🟡 COLLECTING	EXTEND	insufficient_samples
dailyagentrxtraceoptimizer	sub_agent_strategy	single_agent:50, sub_agents:47	🟢 READY	EXTEND	insufficient_observations
dailyarchitecturediagram	detail_level	brief:13, comprehensive:8	🟡 COLLECTING	EXTEND	insufficient_samples
dailyastrostylelitemarkdownspellcheck	prompt_style	concise:84, detailed:78	🟢 READY	EXTEND	insufficient_observations
dailycachestrategyanalyzer	model_size	gpt-5.3-codex:22, gpt-5.3-codex-spark:16	🟡 COLLECTING	EXTEND	insufficient_samples
dailycavemanoptimizer	model_size	agent:1, claude-haiku-4.5:78, claude-sonnet-4.6:34, claude-sonnet-5:15, small-agent:2	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycodemetrics	output_format	executive_summary:54, full_detail:58, ste:16	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailycommunityattribution	prompt_style	concise:74, verbose:75	🟢 READY	EXTEND	insufficient_observations
dailycompilerquality	output_format	concise:56, detailed:66, ste:22	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
dailydochealer	model_size	agent:2, claude-haiku-4.5:66, claude-sonnet-4.6:43, claude-sonnet-5:20, small-agent:1	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailydocupdater	model_size	agent:1, claude-haiku-4.5:55, claude-sonnet-4.6:47, claude-sonnet-5:18, small-agent:2	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailyfact	reasoning_depth	multi_candidate:46, single_pass:62	🟢 READY	EXTEND	guardrail_unsupported
dailyfunctionnamer	model_size		🟡 COLLECTING	EXTEND	insufficient_observations
dailyissuesreport	output_format	collapsible:67, inline:64, ste:23	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailynews	prompt_style	concise:66, detailed:58	🟢 READY	EXTEND	insufficient_observations
dailyrenderingscriptsverifier	remove_redundant_context_v1		🟡 COLLECTING	EXTEND	insufficient_observations
dailysafeoutputoptimizer	log_fetch_strategy		🟡 COLLECTING	EXTEND	insufficient_observations
dailysecurityredteam	reasoning_depth	iterative:88, single_pass:66	🟢 READY	EXTEND	guardrail_unsupported
dailysemgrepscan	semgrep_output_format	bullet_list:28, prose:27, structured_sections:22	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dailysubagentoptimizer	timeout_setting	default:2, relaxed:1, tight:1	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dataflowprdiscussiondataset	caveman_mode	no:7, yes:11	🟡 COLLECTING	EXTEND	insufficient_samples
deepreport	output_format	annotated_brief:71, executive_brief:91, full_briefing:71, ste:53	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
dependabotcampaign	summary_detail	brief:6, detailed:6	🟡 COLLECTING	EXTEND	insufficient_samples
dependabotgochecker	prompt_style	concise:21, detailed:19, step_by_step:19	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	full_bash:67, minimal_toolset:75	🟢 READY	EXTEND	insufficient_observations
issuearborist	prompt_style	concise:54, detailed:47	🟢 READY	EXTEND	insufficient_observations
plan	reasoning_depth	baseline:4, deep:2, shallow:1	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prsouschef	remove_redundant_context_v1		🟡 COLLECTING	EXTEND	insufficient_observations
smokeantigravity	sub_agent_strategy	single_agent:99, sub_agents:109	🟢 READY	EXTEND	insufficient_observations
smokecopilot	caveman	no:194, yes:194	🟢 READY	EXTEND	insufficient_observations
smokecopilot	subagent_model	large:184, small:184	🟢 READY	EXTEND	insufficient_observations
smokecopilotaoaiapikey	caveman	no:100, yes:101	🟢 READY	EXTEND	insufficient_observations
smokecopilotaoaiapikey	subagent_model	large:100, small:101	🟢 READY	EXTEND	insufficient_observations
smokecopilotaoaientra	caveman	no:87, yes:87	🟢 READY	EXTEND	insufficient_observations
smokecopilotaoaientra	subagent_model	large:87, small:87	🟢 READY	EXTEND	insufficient_observations
smokecopilotsubagents	sub_agent_strategy	delegated_sequential:19, inline_strict:18, single_agent_control:20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
smokegemini	sub_agent_strategy	single_agent:137, sub_agents:148	🟢 READY	EXTEND	insufficient_observations
smokepi	sub_agent_decomposition	parallel_sub_agents:42, single_agent:34	🟢 READY	EXTEND	insufficient_observations
smokeproject	prompt_style_test		🟡 COLLECTING	EXTEND	insufficient_observations
smoketemporaryid	sub_agent_strategy		🟡 COLLECTING	EXTEND	insufficient_observations
testqualitysentinel	model_size	claude-haiku-4.5:27, claude-sonnet-5:30	🟢 READY	EXTEND	insufficient_observations
typist	tone_style	conversational:56, formal:45	🟢 READY	EXTEND	insufficient_observations
weeklyblogpostwriter	prefetch_strategy	eager:7, lazy:11	🟡 COLLECTING	EXTEND	insufficient_samples

Outcome charts (success rate and duration) were not generated this run. The decision layer has no usable per-run metric observations, so descriptive outcome stats would not change any decision. Tracking-issue comments and labels were skipped: no experiment is PROMOTE, none has a guardrail_failed rejection, and the experiments list CLI was unusable here (it returned null for the repo). I analysed the fetched branches directly instead.

🔧 Self-Tuning Continuation Plan

Top experiment actions

1. Make observations flow: 26 experiments are READY or collecting but return insufficient_observations. Emit the declared metric and secondary_metrics into the experiment state (starting with the smoke-* and prompt_style workflows with 100+ runs).
2. Support multi-variant decisions, or prune variants: 14 experiments (awfailureinvestigator, deepreport, dailycompilerquality, the model_size family, and others) are stuck on unsupported_multi_variant. Stray variants such as agent and small-agent in model_size also need cleanup.
3. Implement the native guardrail metric for cicoach, dailyfact and dailysecurityredteam (guardrail_unsupported).

Top eval actions

1. No eval scores were available to this run. Wire graders into metric observations for the highest-volume workflows (awfailureinvestigator, smokecopilot, deepreport).
2. Tighten the prompt_style evals for dailynews, dailycommunityattribution and issuearborist, where 100+ runs have no observations.
3. Add output-length and verbosity evals for the output_format experiments.

Decision-pipeline gaps for the next PR

• Unresolved: insufficient_observations (26) and guardrail_unsupported (3).
• Promote the K×K interaction diagnostic (the smokecopilot* caveman × subagent_model pairs) into a normalized core analysis signal.
• gh aw experiments list/analyze --repo fails when the CLI shells out to gh api (unknown flag: --repo). Fix the repo-scoped branch fetch.

Run: 38531338119

Generated by 🧪 daily-experiment-report · copilot · auto · 23.6 AIC · ⌖ 0.937 AIC · ⊞ 16.3K · ◷

• expires on Oct 17, 2026, 12:00 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #68989.