[experiments] Daily Experiment Report — 2026-10-07

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-07

52 experiments analysed across 49 workflows. No experiment has decision-quality evidence: 38 are EXTEND and 14 are INCONCLUSIVE. No PROMOTE or REJECT decisions.

⚡ Quick Stats

Metric	Value
Active experiments	52
🟢 READY (sample gate)	27
🟡 COLLECTING	25
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 38 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no PROMOTE decisions)

Reason codes: insufficient_observations: 26, unsupported_multi_variant: 14, insufficient_samples: 9, guardrail_unsupported: 3

Outcome charts (success rate / duration) were not generated: the main blocker is that native outcome observations are missing for most experiments, so descriptive run-level charts would not inform any decision.

📊 Full Experiments Table

Experiment	Workflow	Control → Candidate	Readiness	Sample progress	Core decision	Reason code
prompt_compression	agentperformanceanalyzer	caveman → verbose	🟢 READY	caveman █████ 75/20; verbose █████ 63/20	EXTEND	insufficient_observations
sub_agent_strategy	agentpersonaexplorer	batch → per_scenario	🟢 READY	batch █████ 73/20; per_scenario █████ 65/20	EXTEND	insufficient_observations
sub_agent_strategy	architectureguardian	single_agent → sub_agents	🟡 COLLECTING	single_agent ████░ 17/20; sub_agents █████ 21/20	EXTEND	insufficient_samples
audit_decomposition	auditworkflows	single_agent → phased_sub_agents	🟡 COLLECTING	phased_sub_agents █░░░░ 53/264; single_agent █░░░░ 39/264	EXTEND	insufficient_samples
prompt_style	blogauditor	concise → detailed	🟡 COLLECTING	concise █░░░░ 4/20; detailed ██░░░ 8/20	EXTEND	insufficient_samples
tone_variant	breakingchangechecker	neutral → urgent	🟢 READY	neutral █████ 31/20; urgent █████ 27/20	EXTEND	insufficient_observations
prompt_style	cicoach	detailed → concise	🟢 READY	concise █████ 42/20; detailed █████ 61/20	EXTEND	guardrail_unsupported
nlp_prompt_style	copilotprnlpanalysis	concise → structured	🟡 COLLECTING	concise ░░░░░ 0/30; structured ░░░░░ 0/30	EXTEND	insufficient_samples
sub_agent_strategy	dailyagentrxtraceoptimizer	single_agent → sub_agents	🟢 READY	single_agent █████ 46/20; sub_agents █████ 46/20	EXTEND	insufficient_observations
detail_level	dailyarchitecturediagram	brief → comprehensive	🟡 COLLECTING	brief ███░░ 12/20; comprehensive ██░░░ 8/20	EXTEND	insufficient_samples
prompt_style	dailyastrostylelitemarkdownspellcheck	concise → detailed	🟢 READY	concise █████ 82/20; detailed █████ 74/20	EXTEND	insufficient_observations
model_size	dailycachestrategyanalyzer	gpt-5.3-codex → gpt-5.3-codex-spark	🟡 COLLECTING	gpt-5.3-codex █████ 19/20; gpt-5.3-codex-spark ███░░ 12/20	EXTEND	insufficient_samples
prompt_style	dailycommunityattribution	concise → verbose	🟢 READY	concise █████ 71/20; verbose █████ 71/20	EXTEND	insufficient_observations
reasoning_depth	dailyfact	single_pass → multi_candidate	🟢 READY	multi_candidate █████ 43/30; single_pass █████ 60/30	EXTEND	guardrail_unsupported
model_size	dailyfunctionnamer	multi-variant / n/a	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
prompt_style	dailynews	concise → detailed	🟢 READY	concise █████ 64/20; detailed █████ 55/20	EXTEND	insufficient_observations
remove_redundant_context_v1	dailyrenderingscriptsverifier	multi-variant / n/a	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
log_fetch_strategy	dailysafeoutputoptimizer	eager → lazy	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
reasoning_depth	dailysecurityredteam	single_pass → iterative	🟢 READY	iterative █████ 85/30; single_pass █████ 62/30	EXTEND	guardrail_unsupported
caveman_mode	dataflowprdiscussiondataset	no → yes	🟡 COLLECTING	no ██░░░ 6/20; yes ███░░ 11/20	EXTEND	insufficient_samples
summary_detail	dependabotcampaign	brief → detailed	🟡 COLLECTING	brief ██░░░ 6/20; detailed ██░░░ 6/20	EXTEND	insufficient_samples
tool_verbosity	gpclean	full_bash → minimal_toolset	🟢 READY	full_bash █████ 64/20; minimal_toolset █████ 71/20	EXTEND	insufficient_observations
prompt_style	issuearborist	concise → detailed	🟢 READY	concise █████ 51/20; detailed █████ 43/20	EXTEND	insufficient_observations
remove_redundant_context_v1	prsouschef	multi-variant / n/a	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
sub_agent_strategy	smokeantigravity	single_agent → sub_agents	🟢 READY	single_agent █████ 99/20; sub_agents █████ 109/20	EXTEND	insufficient_observations
caveman	smokecopilot	no → yes	🟢 READY	no █████ 192/20; yes █████ 192/20	EXTEND	insufficient_observations
subagent_model	smokecopilot	large → small	🟢 READY	large █████ 182/20; small █████ 182/20	EXTEND	insufficient_observations
caveman	smokecopilotaoaiapikey	no → yes	🟢 READY	no █████ 99/20; yes █████ 99/20	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaiapikey	large → small	🟢 READY	large █████ 99/20; small █████ 99/20	EXTEND	insufficient_observations
caveman	smokecopilotaoaientra	no → yes	🟢 READY	no █████ 85/20; yes █████ 86/20	EXTEND	insufficient_observations
subagent_model	smokecopilotaoaientra	large → small	🟢 READY	large █████ 85/20; small █████ 86/20	EXTEND	insufficient_observations
sub_agent_strategy	smokegemini	single_agent → sub_agents	🟢 READY	single_agent █████ 134/20; sub_agents █████ 147/20	EXTEND	insufficient_observations
sub_agent_decomposition	smokepi	parallel_sub_agents → single_agent	🟢 READY	parallel_sub_agents █████ 42/20; single_agent █████ 34/20	EXTEND	insufficient_observations
prompt_style_test	smokeproject	multi-variant / n/a	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
sub_agent_strategy	smoketemporaryid	single_agent → sub_agents	🟡 COLLECTING	n/a	EXTEND	insufficient_observations
model_size	testqualitysentinel	claude-haiku-4.5 → claude-sonnet-5	🟢 READY	claude-haiku-4.5 █████ 130/20; claude-sonnet-5 █████ 125/20	EXTEND	insufficient_observations
tone_style	typist	conversational → formal	🟢 READY	conversational █████ 53/20; formal █████ 43/20	EXTEND	insufficient_observations
prefetch_strategy	weeklyblogpostwriter	eager → lazy	🟡 COLLECTING	eager ██░░░ 7/20; lazy ██░░░ 10/20	EXTEND	insufficient_samples
tone_variant	awfailureinvestigator	multi-variant / n/a	🟢 READY	assertive █████ 198/20; clinical █████ 188/20; narrative █████ 183/20	INCONCLUSIVE	unsupported_multi_variant
output_format	copilotagentanalysis	multi-variant / n/a	🟡 COLLECTING	prose █████ 35/30; ste ░░░░░ 3/30; structured █████ 43/30	INCONCLUSIVE	unsupported_multi_variant
model_size	dailycavemanoptimizer	multi-variant / n/a	🟡 COLLECTING	agent ░░░░░ 1/20; claude-haiku-4.5 █████ 75/20; claude-sonnet-4.6 █████ 34/20; claude-sonnet-5 ███░░ 11/20; small-agent ░░░░░ 2/20	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycodemetrics	multi-variant / n/a	🟡 COLLECTING	executive_summary █████ 54/20; full_detail █████ 58/20; ste ████░ 16/20	INCONCLUSIVE	unsupported_multi_variant
output_format	dailycompilerquality	multi-variant / n/a	🟢 READY	concise █████ 56/20; detailed █████ 66/20; ste █████ 21/20	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydochealer	multi-variant / n/a	🟡 COLLECTING	agent ░░░░░ 2/20; claude-haiku-4.5 █████ 66/20; claude-sonnet-4.6 █████ 43/20; claude-sonnet-5 ███░░ 13/20; small-agent ░░░░░ 1/20	INCONCLUSIVE	unsupported_multi_variant
model_size	dailydocupdater	multi-variant / n/a	🟡 COLLECTING	agent ░░░░░ 1/20; claude-haiku-4.5 █████ 51/20; claude-sonnet-4.6 █████ 47/20; claude-sonnet-5 ████░ 16/20; small-agent ░░░░░ 2/20	INCONCLUSIVE	unsupported_multi_variant
output_format	dailyissuesreport	multi-variant / n/a	🟢 READY	collapsible █████ 67/20; inline █████ 63/20; ste █████ 23/20	INCONCLUSIVE	unsupported_multi_variant
semgrep_output_format	dailysemgrepscan	multi-variant / n/a	🟡 COLLECTING	bullet_list █████ 28/30; prose ████░ 27/30; structured_sections ████░ 22/30	INCONCLUSIVE	unsupported_multi_variant
timeout_setting	dailysubagentoptimizer	multi-variant / n/a	🟡 COLLECTING	default ░░░░░ 2/20; relaxed ░░░░░ 1/20; tight ░░░░░ 1/20	INCONCLUSIVE	unsupported_multi_variant
output_format	deepreport	multi-variant / n/a	🟢 READY	annotated_brief █████ 69/20; executive_brief █████ 90/20; full_briefing █████ 65/20; ste █████ 52/20	INCONCLUSIVE	unsupported_multi_variant
prompt_style	dependabotgochecker	multi-variant / n/a	🟡 COLLECTING	concise █████ 20/20; detailed ████░ 18/20; step_by_step ████░ 18/20	INCONCLUSIVE	unsupported_multi_variant
reasoning_depth	plan	multi-variant / n/a	🟡 COLLECTING	baseline █░░░░ 4/20; deep ░░░░░ 2/20; shallow ░░░░░ 1/20	INCONCLUSIVE	unsupported_multi_variant
sub_agent_strategy	smokecopilotsubagents	multi-variant / n/a	🟡 COLLECTING	delegated_sequential ███░░ 17/30; inline_strict ███░░ 17/30; single_agent_control ███░░ 20/30	INCONCLUSIVE	unsupported_multi_variant

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. 26 experiments are insufficient_observations despite many READY ones (e.g. smokecopilot caveman/subagent_model, 192 samples each): wire each declared metric: to a native observation so decisions can resolve.
2. 14 multi-variant experiments return unsupported_multi_variant (e.g. deepreport, awfailureinvestigator tone_variant, dailycompilerquality/dailyissuesreport output_format): collapse to a control-vs-candidate pair or add core multi-arm support.
3. Imbalanced output_format experiments (ste arm under-assigned: 21–52 vs 56–90) and model_size experiments with stray variants (agent, small-agent) in dailydochealer/dailycavemanoptimizer/dailydocupdater: fix assignment/variant-name hygiene.

Top eval actions

1. copilotprnlpanalysis metric eval:insights_report_produced has 0 samples for both variants: check the eval is emitted and mapped to the nlp_prompt_style prompt.
2. ci-coach run_success_rate/empty_output_rate guardrails are unsupported: add graders feeding those observations.
3. smoketemporaryid metric aic and dailysafeoutputoptimizer run_duration_ms have no observations: add collection in the respective workflows.

Decision-pipeline gaps

• guardrail_unsupported (3 experiments, e.g. ci-coach) and insufficient_observations (26) need native metric backing.
• Interaction diagnostics (K1×K2 cells) should become normalized core signals; currently only workflows with ≥2 experiments (smokecopilot, smokecopilotaoaiapikey, smokecopilotaoaientra) are affected, all EXTEND.

Run: 37590607085

Generated by 🧪 daily-experiment-report · copilot · auto · 23 AIC · ⌖ 10.2 AIC · ⊞ 16.3K · ◷

• expires on Oct 10, 2026, 12:03 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #66834.