[experiments] Daily Experiment Report — 2026-10-19

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-10-19

32 experiments (one per workflow) were analysed from experiments/* branch state. None has a PROMOTE or REJECT decision. 16 are 🟢 READY on sample size, but most lack outcome observations.

⚡ Quick Stats

Metric	Value
Active experiments	32
Ready for analysis	16
Decisions with sufficient evidence	0
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 21 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 11
Interaction safety holds	0 (no workflow has more than one experiment)

📊 Core decisions by experiment

Workflow	Experiment	Runs	Variants	Readiness	Core decision	Reason code
agent-performance-analyzer	prompt_compression	150	caveman 82 / verbose 68	🟢 READY	EXTEND	insufficient_observations
audit-workflows	audit_decomposition	104	phased_sub_agents 60 / single_agent 44	🟢 READY	EXTEND	insufficient_observations
breaking-change-checker	tone_variant	65	neutral 34 / urgent 31	🟢 READY	EXTEND	insufficient_observations
ci-coach	prompt_style	111	concise 47 / detailed 64	🟢 READY	EXTEND	insufficient_observations
daily-agentrx-trace-optimizer	sub_agent_strategy	100	single_agent 51 / sub_agents 49	🟢 READY	EXTEND	insufficient_observations
daily-astrostylelite-markdown-spellcheck	prompt_style	167	concise 86 / detailed 81	🟢 READY	EXTEND	insufficient_observations
daily-community-attribution	prompt_style	154	concise 77 / verbose 77	🟢 READY	EXTEND	insufficient_observations
daily-fact	reasoning_depth	111	multi_candidate 47 / single_pass 64	🟢 READY	EXTEND	insufficient_observations
daily-news	prompt_style	127	concise 67 / detailed 60	🟢 READY	EXTEND	insufficient_observations
daily-security-red-team	reasoning_depth	159	iterative 90 / single_pass 69	🟢 READY	EXTEND	insufficient_observations
issue-arborist	prompt_style	106	concise 57 / detailed 49	🟢 READY	EXTEND	insufficient_observations
gpclean	tool_verbosity	147	full_bash 69 / minimal_toolset 78	🟢 READY	EXTEND	guardrail_unsupported
blog-auditor	prompt_style	14	concise 5 / detailed 9	🟡 COLLECTING	EXTEND	insufficient_samples
daily-architecture-diagram	detail_level	21	brief 13 / comprehensive 8	🟡 COLLECTING	EXTEND	insufficient_samples
dataflow-pr-discussion-dataset	caveman_mode	19	no 8 / yes 11	🟡 COLLECTING	EXTEND	insufficient_samples
agent-persona-explorer, aw-failure-investigator, copilot-pr-nlp-analysis, daily-rendering-scripts-verifier, daily-safe-output-optimizer, go-fan	various	1 each	n/a	🟡 COLLECTING	EXTEND	insufficient_observations
daily-compiler-quality	output_format	144	concise 56 / detailed 66 / ste 22	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
daily-issues-report	output_format	154	collapsible 67 / inline 64 / ste 23	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
daily-semgrep-scan	semgrep_output_format	77	bullet_list 28 / prose 27 / structured_sections 22	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
deep-report	output_format	286	annotated_brief 71 / executive_brief 91 / full_briefing 71 / ste 53	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
copilot-agent-analysis	output_format	93	prose 41 / ste 4 / structured 48	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-code-metrics	output_format	128	executive_summary 54 / full_detail 58 / ste 16	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dependabot-go-checker	prompt_style	61	concise 22 / detailed 19 / step_by_step 20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-doc-healer, daily-doc-updater	model_size	125–137	5–8 variants incl. stray agent/small-agent	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant

⚠️ Data-quality notes

• State branches are healthy (for example experiments/cicoach holds state.json with runs).
• Outcome observations are missing for most experiments, so 11 READY experiments sit at insufficient_observations.
• Workflows with 3+ variants are INCONCLUSIVE with unsupported_multi_variant.
• ste is badly imbalanced in copilot-agent-analysis (4 of 93 runs).
• model_size experiments have stray variants (agent, small-agent) and are unbalanced.
• About 13 workflows (for example awf-reflect, plan, typist, model-resolution) returned output my summariser could not parse, so they are excluded from the counts.
• Charts and duration CIs are omitted because per-run outcome data was not collected in this run. No tracking-issue comments or labels were posted.

🔧 Self-Tuning Continuation Plan

Top experiment actions

1. Wire a primary-metric observation for the 11 READY experiments stuck at insufficient_observations.
2. Add multi-variant support (pairwise vs control with Bonferroni) to unblock 11 unsupported_multi_variant results.
3. Normalize model_size variants and implement the missing guardrail metric for gpclean (guardrail_unsupported).

Top eval actions

1. Add readability and verbosity evals for deep-report, daily-issues-report and daily-compiler-quality output formats.
2. Add correctness evals for ci-coach and daily-fact prompts.
3. Add a tool-sufficiency eval for gpclean (minimal_toolset).

Decision-pipeline gaps for the next PR

• experiments list returns null unless branches are fetched locally, and --repo is not supported.
• No interaction diagnostics are needed yet, since no workflow has more than one experiment.

Descriptive window: last 30 runs per workflow · Run: 39219756595

Generated by 🧪 daily-experiment-report · copilot · auto · 30 AIC · ⌖ 0.859 AIC · ⊞ 17K · ◷

• expires on Oct 22, 2026, 12:07 AM UTC-08:00