[experiments] Daily Experiment Report — 2026-08-26

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-08-26

47 A/B experiments across 44 workflows in github/gh-aw were analysed. None reached a PROMOTE or REJECT core decision yet — 32 are EXTEND (mostly blocked by unsupported outcome-metric observations or low sample counts) and 15 are INCONCLUSIVE (declared 3+ variant designs that core analysis doesn't yet support). 23 experiments are READY on sample count but still lack usable outcome evidence.

⚡ Quick Stats

Metric	Value
Active experiments	47
Ready (sample-size)	23
Collecting	24
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 32 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 15
Interaction safety holds	0

📈 Full Experiments Table (all 47)

Workflow	Experiment	Variants (n)	min_samples	Readiness	Decision	Reason code
agent-performance-analyzer	prompt_compression	caveman=51, verbose=46	20	🟢 READY	EXTEND	insufficient_observations
agent-persona-explorer	sub_agent_strategy	batch=54, per_scenario=43	20	🟢 READY	EXTEND	insufficient_observations
architecture-guardian	sub_agent_strategy	single_agent=17, sub_agents=21	20	🟡 COLLECTING	EXTEND	insufficient_samples
audit-workflows	audit_decomposition	phased_sub_agents=32, single_agent=20	264	🟡 COLLECTING	EXTEND	insufficient_samples
aw-failure-investigator	tone_variant	assertive=113, clinical=125, narrative=105	20	🟢 READY	INCONCLUSIVE	unsupported_multi_variant
blog-auditor	prompt_style	concise=3, detailed=3	20	🟡 COLLECTING	EXTEND	insufficient_samples
breaking-change-checker	tone_variant	neutral=18, urgent=13	20	🟡 COLLECTING	EXTEND	insufficient_samples
ci-coach	prompt_style	concise=32, detailed=41	20	🟢 READY	EXTEND	guardrail_unsupported
copilot-agent-analysis	output_format	prose=33, ste=3, structured=42	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-agentrx-trace-optimizer	sub_agent_strategy	single_agent=31, sub_agents=31	20	🟢 READY	EXTEND	insufficient_observations
daily-architecture-diagram	detail_level	brief=8, comprehensive=6	20	🟡 COLLECTING	EXTEND	insufficient_samples
daily-astrostylelite-markdown-spellcheck	prompt_style	concise=61, detailed=54	20	🟢 READY	EXTEND	insufficient_observations
daily-cache-strategy-analyzer	model_size	6 variants (agent=1..gpt-5.4=37)	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-caveman-optimizer	model_size	agent=1, claude-haiku-4.5=47, claude-sonnet-4.6=33, small-agent=2	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-code-metrics	output_format	executive_summary=42, full_detail=53, ste=6	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-community-attribution	prompt_style	concise=56, verbose=57	20	🟢 READY	EXTEND	insufficient_observations
daily-compiler-quality	output_format	concise=46, detailed=50, ste=6	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-doc-healer	model_size	agent=2, claude-haiku-4.5=41, claude-sonnet-4.6=39, small-agent=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-doc-updater	model_size	agent=1, claude-haiku-4.5=34, claude-sonnet-4.6=46, small-agent=2	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-fact	reasoning_depth	multi_candidate=29, single_pass=45	30	🟡 COLLECTING	EXTEND	insufficient_samples
daily-function-namer	model_size	n=3 total (below all thresholds)	20	🟡 COLLECTING	EXTEND	insufficient_observations
daily-issues-report	output_format	collapsible=47, inline=55, ste=11	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-news	prompt_style	concise=48, detailed=41	20	🟢 READY	EXTEND	insufficient_observations
daily-safe-output-optimizer	log_fetch_strategy	eager=27, lazy=26	20	🟢 READY	EXTEND	guardrail_unsupported
daily-security-red-team	reasoning_depth	iterative=65, single_pass=41	30	🟢 READY	EXTEND	guardrail_unsupported
daily-semgrep-scan	semgrep_output_format	bullet_list=28, prose=27, structured_sections=22	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
daily-sub-agent-optimizer	timeout_setting	default=2, relaxed=1, tight=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dataflow-pr-discussion-dataset	caveman_mode	no=4, yes=7	10	🟡 COLLECTING	EXTEND	insufficient_samples
deep-report	output_format	annotated_brief=29, executive_brief=42, full_briefing=32, ste=10	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
dependabot-campaign	summary_detail	brief=6, detailed=6	20	🟡 COLLECTING	EXTEND	insufficient_samples
dependabot-go-checker	prompt_style	concise=13, detailed=13, step_by_step=12	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
gpclean	tool_verbosity	full_bash=41, minimal_toolset=53	20	🟢 READY	EXTEND	insufficient_observations
issue-arborist	prompt_style	concise=31, detailed=22	20	🟢 READY	EXTEND	insufficient_observations
plan	reasoning_depth	baseline=4, deep=2, shallow=1	20	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
smoke-antigravity	sub_agent_strategy	single_agent=99, sub_agents=109	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot · caveman	caveman	no=170, yes=170	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot · subagent_model	subagent_model	large=160, small=160	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot-aoai-apikey · caveman	caveman	no=84, yes=85	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot-aoai-apikey · subagent_model	subagent_model	large=85, small=84	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot-aoai-entra · caveman	caveman	no=71, yes=71	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot-aoai-entra · subagent_model	subagent_model	large=71, small=71	20	🟢 READY	EXTEND	insufficient_observations
smoke-copilot-sub-agents	sub_agent_strategy	delegated_sequential=13, inline_strict=7, single_agent_control=12	30	🟡 COLLECTING	INCONCLUSIVE	unsupported_multi_variant
smoke-gemini	sub_agent_strategy	single_agent=116, sub_agents=136	20	🟢 READY	EXTEND	insufficient_observations
smoke-pi	sub_agent_decomposition	parallel_sub_agents=42, single_agent=34	20	🟢 READY	EXTEND	insufficient_observations
test-quality-sentinel	model_size	claude-haiku-4.5=615, claude-sonnet-4.6=565	20	🟢 READY	EXTEND	insufficient_observations
typist	tone_style	conversational=36, formal=30	20	🟢 READY	EXTEND	insufficient_observations
weekly-blog-post-writer	prefetch_strategy	eager=4, lazy=7	20	🟡 COLLECTING	EXTEND	insufficient_samples

🔍 Interpretation

• insufficient_observations (20 experiments): sample counts satisfy min_samples, but the workflow's declared metric (e.g. token_count_per_run, custom quality scores) is not being written to per-run observation data that core analysis can consume. This is a data-pipeline gap, not evidence against any variant.
• unsupported_multi_variant (15 experiments): core statistical analysis currently supports 2-variant A/B comparisons; experiments declaring 3+ variants (output_format, model_size, reasoning_depth families) cannot yet produce a PROMOTE/REJECT decision even with abundant samples.
• guardrail_unsupported (3 experiments): cicoach, daily-safe-output-optimizer, daily-security-red-team are READY but blocked because a mandatory guardrail metric (e.g. run_success_rate) has no supported observation source.
• insufficient_samples (9 experiments): still accumulating runs toward min_samples.
• No interaction diagnostics were computed this run: no workflow currently runs two concurrent experiments with enough overlapping data for a meaningful K1×K2 cross-tab, so no SPARSE_CELL_RISK holds applied.

🛠️ Self-Tuning Continuation Plan

Top 3 experiment actions

1. Wire per-run outcome-metric observations (token_count_per_run, quality scores, run_success_rate) into the experiment state pipeline — this alone would unblock 23 insufficient_observations/guardrail_unsupported experiments that are already at full sample size.
2. Extend core analysis to support K≥3 variant designs (with Bonferroni correction, already partially modeled via bonferroni_alpha) — this unblocks 15 INCONCLUSIVE experiments, mostly output_format/model_size families.
3. Continue run accumulation for the 9 insufficient_samples experiments; no action needed beyond normal scheduling.

Top 3 eval actions

1. Add graders/evals that score output_format variants (structured, prose, ste, etc.) on verbosity and reader comprehension, feeding output_length_chars/output_token_count as secondary metrics — directly relevant to daily-code-metrics, daily-compiler-quality, daily-issues-report, deep-report, copilot-agent-analysis.
2. Add success/failure graders for model_size experiments (daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-doc-healer, daily-doc-updater) to produce the run_success_rate guardrail observation these workflows already declare but cannot evaluate.
3. Add a lightweight quality grader for prompt_style experiments (ci-coach, daily-astrostylelite-markdown-spellcheck, daily-news, issue-arborist) so insufficient_observations converts into usable evidence once sample counts are already sufficient.

Decision-pipeline gaps for next PR

• Normalize a "multi-variant chi-square / Kruskal-Wallis" decision path in core so 3+ variant experiments stop reporting unsupported_multi_variant once samples are sufficient.
• Add a supported native metric source for run_success_rate and empty_output_rate guardrails so guardrail_unsupported experiments (ci-coach, daily-safe-output-optimizer, daily-security-red-team) can reach a real guardrail pass/fail.
• Promote the factorial-interaction helper described in this reporting workflow into a first-class core analysis signal once two workflows run overlapping concurrent experiments with enough joint samples.

Descriptive window: gh aw experiments analyze state as of this run · Decision thresholds: from each experiment's decision_policy
Run: 32945795113

Generated by 🧪 daily-experiment-report · copilot · auto · 67.1 AIC · ⌖ 5 AIC · ⊞ 17K · ◷

• expires on Aug 29, 2026, 12:13 AM UTC-08:00