[experiments] Daily Experiment Report — 2026-10-02

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-02

52 experiments analysed across 49 workflows. No PROMOTE or REJECT decisions; 38 are EXTEND and 14 INCONCLUSIVE. No interaction safety holds (no workflow runs multiple experiments with a PROMOTE decision).

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready for analysis	26
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Reason codes	insufficient_observations: 26, unsupported_multi_variant: 14, insufficient_samples: 9, guardrail_unsupported: 3

⚠️ Per-run outcome charts (success rate / duration) were not generated in this run; decisions below come directly from gh aw experiments analyze.

📊 All experiments

Experiment	Workflow	Usable samples	Readiness	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	caveman=73, verbose=60	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	batch=69, per_scenario=64	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	single_agent=17, sub_agents=21	🟡 COLLECTING	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	phased_sub_agents=49, single_agent=38	🟡 COLLECTING	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	assertive=171, clinical=166, narrative=166	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	concise=4, detailed=8	🟡 COLLECTING	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	neutral=30, urgent=25	🟢 READY	EXTEND	insufficient_observations
prompt_style	cicoach	concise=41, detailed=59	🟢 READY	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	prose=34, ste=3, structured=43	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	concise=0, structured=0	🟡 COLLECTING	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent=44, sub_agents=45	🟢 READY	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	brief=12, comprehensive=7	🟡 COLLECTING	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	concise=79, detailed=72	🟢 READY	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	gpt-5.3-codex=15, gpt-5.3-codex-spark=11	🟡 COLLECTING	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	agent=1, claude-haiku-4.5=73, claude-sonnet-4.6=34, claude-sonnet-5=9, small-agent=2	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	executive_summary=54, full_detail=58, ste=16	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	concise=70, verbose=70	🟢 READY	EXTEND	insufficient_observations
output_format	dailycompilerquality	concise=55, detailed=65, ste=18	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	agent=2, claude-haiku-4.5=63, claude-sonnet-4.6=43, claude-sonnet-5=11, small-agent=1	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	agent=1, claude-haiku-4.5=50, claude-sonnet-4.6=47, claude-sonnet-5=14, small-agent=2	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	multi_candidate=42, single_pass=58	🟢 READY	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer	—	🟡 COLLECTING	EXTEND	insufficient_observations
output_format	dailyissuesreport	collapsible=63, inline=63, ste=22	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	concise=63, detailed=53	🟢 READY	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier	—	🟡 COLLECTING	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	—	🟡 COLLECTING	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	iterative=82, single_pass=60	🟢 READY	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	bullet_list=28, prose=27, structured_sections=22	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	default=2, relaxed=1, tight=1	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	no=6, yes=11	🟡 COLLECTING	EXTEND	insufficient_samples
output_format	deepreport	annotated_brief=66, executive_brief=85, full_briefing=61, ste=45	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	brief=6, detailed=6	🟡 COLLECTING	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	concise=20, detailed=18, step_by_step=16	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	full_bash=62, minimal_toolset=68	🟢 READY	EXTEND	insufficient_observations
prompt_style	issuearborist	concise=49, detailed=40	🟢 READY	EXTEND	insufficient_observations
reasoning_depth	plan	baseline=4, deep=2, shallow=1	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef	—	🟡 COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	single_agent=99, sub_agents=109	🟢 READY	EXTEND	insufficient_observations
caveman	smokecopilot	no=189, yes=189	🟢 READY	EXTEND	insufficient_observations
subagent_model	smokecopilot	large=179, small=179	🟢 READY	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	no=97, yes=97	🟢 READY	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	large=97, small=97	🟢 READY	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	no=83, yes=84	🟢 READY	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	large=84, small=83	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	delegated_sequential=17, inline_strict=15, single_agent_control=19	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	single_agent=132, sub_agents=145	🟢 READY	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	parallel_sub_agents=42, single_agent=34	🟢 READY	EXTEND	insufficient_observations
prompt_style_test	smokeproject	—	🟡 COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	—	🟡 COLLECTING	EXTEND	insufficient_observations
model_size	testqualitysentinel	claude-haiku-4.5=58, claude-sonnet-5=77	🟢 READY	EXTEND	insufficient_observations
tone_style	typist	conversational=51, formal=42	🟢 READY	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	eager=6, lazy=10	🟡 COLLECTING	EXTEND	insufficient_samples

🔁 Self-Tuning Continuation Plan

Experiment actions

1. 26 experiments are READY on assignments but lack metric observations: wire primary metric observations (evals/graders) into run state.
2. 14 multi-variant experiments (incl. many ste output_format arms) are INCONCLUSIVE: add pairwise/multi-arm decision support or reduce to control vs. one candidate.
3. guardrail_unsupported (cicoach, dailyfact, dailysecurityredteam): provide native guardrail metric observations.

Eval actions: tighten low-scoring eval questions for the workflows with insufficient_observations first (smoke*, typist, issuearborist, gpclean).

Decision-pipeline gaps: the local experiments list/analyze requires experiment branches to be fetched first; --repo is not accepted by the CLI; interaction diagnostics should become normalized core signals.

Run: 36981133048

Generated by 🧪 daily-experiment-report · copilot · auto · 24.3 AIC · ⌖ 10.8 AIC · ⊞ 16.3K · ◷

• expires on Oct 5, 2026, 12:02 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #65286.