[experiments] Daily Experiment Report — 2026-10-13

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-13

52 experiments analysed across 49 workflows. None has decision-quality evidence: 0 PROMOTE, 0 REJECT, 38 EXTEND, 14 INCONCLUSIVE. No interaction safety holds were needed, because no core decision was PROMOTE.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis (READY)	25
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0

Reason codes: insufficient_observations 25, unsupported_multi_variant 14, insufficient_samples 10, guardrail_unsupported 3.

📊 All experiments (core decision from `gh aw experiments analyze`)

Experiment	Workflow	Readiness	Samples	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	🟢 READY	caveman:80, verbose:64	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	🟢 READY	batch:77, per_scenario:67	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	🟡 COLLECTING	single_agent:17, sub_agents:21	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	🟡 COLLECTING	phased_sub_agents:57, single_agent:41	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	🟢 READY	assertive:227, clinical:206, narrative:201	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	🟡 COLLECTING	concise:4, detailed:9	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	🟢 READY	neutral:32, urgent:30	EXTEND	insufficient_observations
prompt_style	cicoach	🟢 READY	concise:46, detailed:61	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	🟡 COLLECTING	prose:37, ste:4, structured:46	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	🟡 COLLECTING	concise:0, structured:0	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	🟢 READY	single_agent:50, sub_agents:46	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	🟡 COLLECTING	brief:13, comprehensive:8	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	🟢 READY	concise:83, detailed:78	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	🟡 COLLECTING	gpt-5.3-codex:22, gpt-5.3-codex-spark:15	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	🟡 COLLECTING	agent:1, claude-haiku-4.5:77, claude-sonnet-4.6:34, claude-sonnet-5:15, small-agent:2	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	🟡 COLLECTING	executive_summary:54, full_detail:58, ste:16	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	🟢 READY	concise:74, verbose:74	EXTEND	insufficient_observations
output_format	dailycompilerquality	🟢 READY	concise:56, detailed:66, ste:22	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	🟡 COLLECTING	agent:2, claude-haiku-4.5:66, claude-sonnet-4.6:43, claude-sonnet-5:19, small-agent:1	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	🟡 COLLECTING	agent:1, claude-haiku-4.5:54, claude-sonnet-4.6:47, claude-sonnet-5:18, small-agent:2	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	🟢 READY	multi_candidate:46, single_pass:61	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
output_format	dailyissuesreport	🟡 COLLECTING	collapsible:67, inline:64, ste:23	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	🟢 READY	concise:66, detailed:57	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	🟢 READY	iterative:88, single_pass:65	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	🟡 COLLECTING	bullet_list:28, prose:27, structured_sections:22	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	🟡 COLLECTING	default:2, relaxed:1, tight:1	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	🟡 COLLECTING	no:7, yes:11	EXTEND	insufficient_samples
output_format	deepreport	🟢 READY	annotated_brief:71, executive_brief:91, full_briefing:71, ste:53	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	🟡 COLLECTING	brief:6, detailed:6	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	🟡 COLLECTING	concise:21, detailed:19, step_by_step:19	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	🟢 READY	full_bash:67, minimal_toolset:74	EXTEND	insufficient_observations
prompt_style	issuearborist	🟢 READY	concise:54, detailed:46	EXTEND	insufficient_observations
reasoning_depth	plan	🟡 COLLECTING	baseline:4, deep:2, shallow:1	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	🟢 READY	single_agent:99, sub_agents:109	EXTEND	insufficient_observations
caveman	smokecopilot	🟢 READY	no:194, yes:194	EXTEND	insufficient_observations
subagent_model	smokecopilot	🟢 READY	large:184, small:184	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	🟢 READY	no:100, yes:101	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	🟢 READY	large:100, small:101	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	🟢 READY	no:87, yes:87	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	🟢 READY	large:87, small:87	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	🟡 COLLECTING	delegated_sequential:19, inline_strict:18, single_agent_control:20	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	🟢 READY	single_agent:137, sub_agents:148	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	🟢 READY	parallel_sub_agents:42, single_agent:34	EXTEND	insufficient_observations
prompt_style_test	smokeproject	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
model_size	testqualitysentinel	🟡 COLLECTING	claude-haiku-4.5:9, claude-sonnet-5:12	EXTEND	insufficient_samples
tone_style	typist	🟢 READY	conversational:56, formal:44	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	🟡 COLLECTING	eager:7, lazy:11	EXTEND	insufficient_samples

Outcome-metric charts (success rate and duration) were not generated. Per-variant outcome observations are not available to core analysis, which is why most decisions are insufficient_observations. Tracking-issue comments and labels were skipped because no experiment qualified (no PROMOTE, no guardrail_failed REJECT).

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Wire outcome observations (metric values per run) for the 25 insufficient_observations experiments. Many already have READY sample counts, for example the smokecopilot* variants, gpclean, typist and dailynews.
2. Add multi-variant analysis support in core for the 14 unsupported_multi_variant experiments (K≥3). These include tone_variant, output_format and model_size. Until then, consider collapsing them to 2 variants.
3. Back the mandatory guardrails of cicoach (run_success_rate), dailyfact and dailysecurityredteam with supported metrics. These are the guardrail_unsupported cases.

Top eval actions

1. Add an observation for eval:insights_report_produced for copilotprnlpanalysis, which has 0 samples per variant.
2. Add graders for the discussion_reaction_count and false_positive_rate metrics.
3. Add native run_success_rate and token_count_per_run emission to the experiment state.

Decision-pipeline gaps for next PR

• Native metrics lacking observations: token_count_per_run, run_success_rate, discussion_reaction_count, false_positive_rate, ai_credits_*.
• Make factorial interaction diagnostics a normalized core signal (smokecopilot* run caveman × subagent_model).
• gh aw experiments has no working --repo flag in this environment, so the report used a local build and fetched branches.

Run: 38366510665

Generated by 🧪 daily-experiment-report · copilot · auto · 21.8 AIC · ⌖ 0.931 AIC · ⊞ 16.3K · ◷

• expires on Oct 16, 2026, 12:07 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #68637.