[experiments] Daily Experiment Report — 2026-10-01

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-01

52 experiments analysed across 43 workflows. No experiment has a PROMOTE or REJECT decision. Core decisions are 38 EXTEND and 14 INCONCLUSIVE. Outcome-metric observations are mostly not yet wired into the decision layer.

⚡ Quick Stats

Metric	Value
Active experiments	52
Ready (all variants ≥ min_samples)	26
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no PROMOTE decisions to hold)

Reason codes: 26 insufficient_observations, 3 guardrail_unsupported, 9 insufficient_samples, 14 unsupported_multi_variant.

📊 Full Experiments Table

Experiment	Workflow	Variant assignments	min_samples	Readiness	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	caveman=73, verbose=59	20	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	batch=69, per_scenario=63	20	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	single_agent=17, sub_agents=21	20	🟡 COLLECTING	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	phased_sub_agents=48, single_agent=38	264	🟡 COLLECTING	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	assertive=168, clinical=165, narrative=165	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	blogauditor	concise=4, detailed=8	20	🟡 COLLECTING	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	neutral=29, urgent=25	20	🟢 READY	EXTEND	insufficient_observations
prompt_style	cicoach	concise=40, detailed=59	20	🟢 READY	EXTEND	guardrail_unsupported
output_format	copilotagentanalysis	prose=34, ste=3, structured=43	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
nlp_prompt_style	copilotprnlpanalysis	concise=0, structured=1	30	🟡 COLLECTING	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent=44, sub_agents=44	20	🟢 READY	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	brief=12, comprehensive=7	20	🟡 COLLECTING	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	concise=78, detailed=72	20	🟢 READY	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	gpt-5.3-codex=14, gpt-5.3-codex-spark=11	20	🟡 COLLECTING	EXTEND	insufficient_samples
model_size	dailycavemanoptimizer	agent=1, claude-haiku-4.5=72, claude-sonnet-4.6=34, claude-sonnet-5=9, small-agent=2	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	executive_summary=54, full_detail=58, ste=16	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailycommunityattribution	concise=70, verbose=70	20	🟢 READY	EXTEND	insufficient_observations
output_format	dailycompilerquality	concise=55, detailed=65, ste=17	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	agent=2, claude-haiku-4.5=62, claude-sonnet-4.6=43, claude-sonnet-5=11, small-agent=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	agent=1, claude-haiku-4.5=49, claude-sonnet-4.6=47, claude-sonnet-5=14, small-agent=2	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	dailyfact	multi_candidate=42, single_pass=57	30	🟢 READY	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer		20	🟡 COLLECTING	EXTEND	insufficient_observations
output_format	dailyissuesreport	collapsible=62, inline=63, ste=22	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dailynews	concise=63, detailed=52	20	🟢 READY	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier		20	🟡 COLLECTING	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer		20	🟡 COLLECTING	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	iterative=82, single_pass=59	30	🟢 READY	EXTEND	guardrail_unsupported
semgrep_output_format	dailysemgrepscan	bullet_list=28, prose=27, structured_sections=22	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	default=2, relaxed=1, tight=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
caveman_mode	dataflowprdiscussiondataset	no=6, yes=10	20	🟡 COLLECTING	EXTEND	insufficient_samples
output_format	deepreport	annotated_brief=65, executive_brief=85, full_briefing=60, ste=43	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
summary_detail	dependabotcampaign	brief=6, detailed=6	20	🟡 COLLECTING	EXTEND	insufficient_samples
prompt_style	dependabotgochecker	concise=20, detailed=18, step_by_step=16	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
tool_verbosity	gpclean	full_bash=61, minimal_toolset=68	20	🟢 READY	EXTEND	insufficient_observations
prompt_style	issuearborist	concise=49, detailed=39	20	🟢 READY	EXTEND	insufficient_observations
reasoning_depth	plan	baseline=4, deep=2, shallow=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
remove_redundant_context_v1	prsouschef		20	🟡 COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	single_agent=99, sub_agents=109	20	🟢 READY	EXTEND	insufficient_observations
caveman	smokecopilot	no=188, yes=188	20	🟢 READY	EXTEND	insufficient_observations
subagent_model	smokecopilot	large=178, small=178	20	🟢 READY	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	no=97, yes=97	20	🟢 READY	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	large=97, small=97	20	🟢 READY	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	no=83, yes=84	20	🟢 READY	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	large=84, small=83	20	🟢 READY	EXTEND	insufficient_observations
sub_agent_strategy	smokecopilotsubagents	delegated_sequential=17, inline_strict=15, single_agent_control=19	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokegemini	single_agent=132, sub_agents=145	20	🟢 READY	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	parallel_sub_agents=42, single_agent=34	20	🟢 READY	EXTEND	insufficient_observations
prompt_style_test	smokeproject		20	🟡 COLLECTING	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid		20	🟡 COLLECTING	EXTEND	insufficient_observations
model_size	testqualitysentinel	claude-haiku-4.5=54, claude-sonnet-5=58	20	🟢 READY	EXTEND	insufficient_observations
tone_style	typist	conversational=50, formal=42	20	🟢 READY	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	eager=6, lazy=10	20	🟡 COLLECTING	EXTEND	insufficient_samples

🔎 Highlights

• cicoach / prompt_style (concise=40 vs detailed=59, metric token_count_per_run): READY but EXTEND (guardrail_unsupported). run_success_rate and empty_output_rate have no supported metric observation.
• dailyfact and dailysecurityredteam: same guardrail_unsupported gap.
• awfailureinvestigator / tone_variant (3 variants, ~165 runs each): INCONCLUSIVE (unsupported_multi_variant).
• Several multi-variant experiments (ste output_format variants, model_size) show assignment imbalance (chi-square p ≪ 0.05). They are INCONCLUSIVE because core analysis does not support K≥3 comparisons.
• No per-run outcome (success/duration) correlation or charts were produced this run. The experiment state records assignments only, and most experiments declare no native metric.

🛠️ Self-Tuning Continuation Plan

Top experiment actions

1. Wire observations for the 26 READY experiments stuck on insufficient_observations: declare a native metric: or an eval-backed metric.
2. Add supported observations for run_success_rate and empty_output_rate so cicoach, dailyfact and dailysecurityredteam can clear guardrail_unsupported.
3. Investigate assignment imbalance in the ste / model_size experiments (dailycodemetrics, dailycompilerquality, deepreport, dailydochealer, dailydocupdater, dailycavemanoptimizer).

Top eval actions

1. Add an eval question and grader per READY workflow without a metric, mapped to its prompt-variant dimension (start with the smoke* and daily* prompt_style workflows).
2. Tighten eval:insights_report_produced for copilotprnlpanalysis: it has 1 observation, so the eval yields no usable samples.
3. Add an output-quality eval to the output_format experiments so verbosity/readability variants (ste) can be compared beyond length.

Decision-pipeline gaps for the next PR

• Unsupported native metrics/guardrails (run_success_rate, empty_output_rate) cause guardrail_unsupported; missing observations cause insufficient_observations.
• Core analysis should support K≥3 variants to clear unsupported_multi_variant.
• Factorial interaction diagnostics for smokecopilot* (caveman × subagent_model) should become a normalized core signal.

Descriptive window: assignment state per workflow branch · Decision thresholds: from decision_policy
Run: 36833264548

Generated by 🧪 daily-experiment-report · copilot · auto · 29.8 AIC · ⌖ 16.6 AIC · ⊞ 16.3K · ◷

• expires on Oct 4, 2026, 12:02 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #64967.