[experiments] Daily Experiment Report — 2026-09-11 #60223

2026-09-11T08:05:12Z

github-actions[bot]
Bot Sep 11, 2026

🧪 Daily Experiment Report — 2026-09-11

51 experiments analysed across 48 workflows. 22 have reached minimum sample size (READY). No experiment reached a PROMOTE or REJECT decision today — the core decision layer currently reports EXTEND or INCONCLUSIVE for every experiment, driven almost entirely by a data-pipeline gap: per-run native primary-metric observations (e.g. token counts, success rates, quality scores) are not yet wired into the decision layer for most workflows, and core analysis does not yet support statistical comparison across 3+ variants.

⚡ Quick Stats

Metric	Value
Active experiments	51
Ready for analysis (`READY`)	22
Collecting (`COLLECTING`)	29
Decisions with sufficient evidence	0 (all blocked by data-pipeline gaps)
Core decisions	✅ PROMOTE: 0 · 🟡 EXTEND: 37 · ❌ REJECT: 0 · ⚪ INCONCLUSIVE: 14
Interaction safety holds	0 (no workflow currently runs 2+ concurrent experiments)

📋 Experiments by Decision Reason

🟡 EXTEND — missing native metric observations — 25 experiments

primary-metric per-run observations are not available for statistical comparison

Workflow	Experiment	Readiness	Variants (n)	min_samples	Issue
`agent-performance-analyzer.md`	`prompt_compression`	🟢 READY	`caveman`=62, `verbose`=51	20	`#33280`
`agent-persona-explorer.md`	`sub_agent_strategy`	🟢 READY	`batch`=59, `per_scenario`=54	20	—
`daily-agentrx-trace-optimizer.md`	`sub_agent_strategy`	🟢 READY	`single_agent`=37, `sub_agents`=37	20	—
`daily-astrostylelite-markdown-spellcheck.md`	`prompt_style`	🟢 READY	`concise`=67, `detailed`=64	20	—
`daily-community-attribution.md`	`prompt_style`	🟢 READY	`concise`=64, `verbose`=65	20	—
`daily-function-namer.md`	`model_size`	🟡 COLLECTING	n/a	20	—
`daily-news.md`	`prompt_style`	🟢 READY	`concise`=54, `detailed`=47	20	`#31190`
`daily-rendering-scripts-verifier.md`	`remove_redundant_context_v1`	🟡 COLLECTING	n/a	20	—
`daily-safe-output-optimizer.md`	`log_fetch_strategy`	🟡 COLLECTING	n/a	20	`#38094`
`gpclean.md`	`tool_verbosity`	🟢 READY	`full_bash`=52, `minimal_toolset`=58	20	—
`issue-arborist.md`	`prompt_style`	🟢 READY	`concise`=37, `detailed`=32	20	`#30015`
`pr-sous-chef.md`	`remove_redundant_context_v1`	🟡 COLLECTING	n/a	20	—
`smokeantigravity (file removed)`	`sub_agent_strategy`	🟢 READY	`single_agent`=99, `sub_agents`=109	20	—
`smoke-copilot.md`	`caveman`	🟢 READY	`no`=178, `yes`=178	20	—
`smoke-copilot.md`	`subagent_model`	🟢 READY	`large`=168, `small`=168	20	—
`smoke-copilot-aoai-apikey.md`	`caveman`	🟢 READY	`no`=91, `yes`=90	20	—
`smoke-copilot-aoai-apikey.md`	`subagent_model`	🟢 READY	`large`=90, `small`=91	20	—
`smoke-copilot-aoai-entra.md`	`caveman`	🟢 READY	`no`=77, `yes`=77	20	—
`smoke-copilot-aoai-entra.md`	`subagent_model`	🟢 READY	`large`=77, `small`=77	20	—
`smoke-gemini.md`	`sub_agent_strategy`	🟢 READY	`single_agent`=124, `sub_agents`=140	20	—
`smoke-pi.md`	`sub_agent_decomposition`	🟢 READY	`parallel_sub_agents`=42, `single_agent`=34	20	—
`smoke-project.md`	`prompt_style_test`	🟡 COLLECTING	n/a	20	`#37302`
`smoke-temporary-id.md`	`sub_agent_strategy`	🟡 COLLECTING	n/a	20	—
`test-quality-sentinel.md`	`model_size`	🟡 COLLECTING	n/a	20	`#43530`
`typist.md`	`tone_style`	🟢 READY	`conversational`=43, `formal`=35	20	`#34032`

⚪ INCONCLUSIVE — 3+ variant test unsupported — 14 experiments

core analysis does not yet support statistical comparison across 3+ variants

Workflow	Experiment	Readiness	Variants (n)	min_samples	Issue
`aw-failure-investigator.md`	`tone_variant`	🟢 READY	`assertive`=140, `clinical`=135, `narrative`=131	20	`#36105`
`copilot-agent-analysis.md`	`output_format`	🟡 COLLECTING	`prose`=34, `ste`=3, `structured`=43	30	—
`daily-caveman-optimizer.md`	`model_size`	🟡 COLLECTING	`agent`=1, `claude-haiku-4.5`=57, `claude-sonnet-4.6`=34, `claude-sonnet-5`=5, `small-agent`=2	20	—
`daily-code-metrics.md`	`output_format`	🟡 COLLECTING	`executive_summary`=50, `full_detail`=55, `ste`=12	20	`#1`
`daily-compiler-quality.md`	`output_format`	🟡 COLLECTING	`concise`=50, `detailed`=56, `ste`=12	20	`#32390`
`daily-doc-healer.md`	`model_size`	🟡 COLLECTING	`agent`=2, `claude-haiku-4.5`=50, `claude-sonnet-4.6`=43, `claude-sonnet-5`=3, `small-agent`=1	20	—
`daily-doc-updater.md`	`model_size`	🟡 COLLECTING	`agent`=1, `claude-haiku-4.5`=42, `claude-sonnet-4.6`=47, `claude-sonnet-5`=6, `small-agent`=2	20	—
`daily-issues-report.md`	`output_format`	🟡 COLLECTING	`collapsible`=55, `inline`=58, `ste`=16	20	`#30573`
`daily-semgrep-scan.md`	`semgrep_output_format`	🟡 COLLECTING	`bullet_list`=28, `prose`=27, `structured_sections`=22	30	`#32795`
`dailysubagentoptimizer (file removed)`	`timeout_setting`	🟡 COLLECTING	`default`=2, `relaxed`=1, `tight`=1	20	—
`deep-report.md`	`output_format`	🟡 COLLECTING	`annotated_brief`=43, `executive_brief`=64, `full_briefing`=51, `ste`=19	20	—
`dependabot-go-checker.md`	`prompt_style`	🟡 COLLECTING	`concise`=16, `detailed`=16, `step_by_step`=13	20	—
`plan.md`	`reasoning_depth`	🟡 COLLECTING	`baseline`=4, `deep`=2, `shallow`=1	20	`#42941`
`smoke-copilot-sub-agents.md`	`sub_agent_strategy`	🟡 COLLECTING	`delegated_sequential`=14, `inline_strict`=12, `single_agent_control`=15	30	`#47551`

🟡 EXTEND — below min_samples — 9 experiments

one or more variants have not yet reached the configured min_samples threshold

Workflow	Experiment	Readiness	Variants (n)	min_samples	Issue
`architecture-guardian.md`	`sub_agent_strategy`	🟡 COLLECTING	`single_agent`=17, `sub_agents`=21	20	—
`audit-workflows.md`	`audit_decomposition`	🟡 COLLECTING	`phased_sub_agents`=38, `single_agent`=30	264	`#43177`
`blog-auditor.md`	`prompt_style`	🟡 COLLECTING	`concise`=3, `detailed`=6	20	`#32603`
`breaking-change-checker.md`	`tone_variant`	🟡 COLLECTING	`neutral`=22, `urgent`=18	20	`#42467`
`daily-architecture-diagram.md`	`detail_level`	🟡 COLLECTING	`brief`=10, `comprehensive`=6	20	`#31926`
`daily-cache-strategy-analyzer.md`	`model_size`	🟡 COLLECTING	`gpt-5.3-codex`=2, `gpt-5.3-codex-spark`=3	20	—
`dataflow-pr-discussion-dataset.md`	`caveman_mode`	🟡 COLLECTING	`no`=5, `yes`=9	20	`#37102`
`dependabotcampaign (file removed)`	`summary_detail`	🟡 COLLECTING	`brief`=6, `detailed`=6	20	—
`weekly-blog-post-writer.md`	`prefetch_strategy`	🟡 COLLECTING	`eager`=5, `lazy`=8	20	`#38590`

🟡 EXTEND — guardrail metric unsupported — 3 experiments

a required guardrail metric has no native observation pipeline yet

Workflow	Experiment	Readiness	Variants (n)	min_samples	Issue
`ci-coach.md`	`prompt_style`	🟢 READY	`concise`=37, `detailed`=48	20	`#32335`
`daily-fact.md`	`reasoning_depth`	🟢 READY	`multi_candidate`=34, `single_pass`=52	30	`#31324`
`daily-security-red-team.md`	`reasoning_depth`	🟢 READY	`iterative`=74, `single_pass`=48	30	`#31673`

📈 Full Sample-Size Progress Table (all 51 experiments)

Workflow	Experiment	Progress toward min_samples
`agent-performance-analyzer.md`	`prompt_compression`	caveman:██████████62/20 verbose:██████████51/20
`agent-persona-explorer.md`	`sub_agent_strategy`	batch:██████████59/20 per_scenario:██████████54/20
`architecture-guardian.md`	`sub_agent_strategy`	single_agent:████████░░17/20 sub_agents:██████████21/20
`audit-workflows.md`	`audit_decomposition`	phased_sub_agents:█░░░░░░░░░38/264 single_agent:█░░░░░░░░░30/264
`aw-failure-investigator.md`	`tone_variant`	assertive:██████████140/20 clinical:██████████135/20 narrative:██████████131/20
`blog-auditor.md`	`prompt_style`	concise:██░░░░░░░░3/20 detailed:███░░░░░░░6/20
`breaking-change-checker.md`	`tone_variant`	neutral:██████████22/20 urgent:█████████░18/20
`ci-coach.md`	`prompt_style`	concise:██████████37/20 detailed:██████████48/20
`copilot-agent-analysis.md`	`output_format`	prose:██████████34/30 ste:█░░░░░░░░░3/30 structured:██████████43/30
`daily-agentrx-trace-optimizer.md`	`sub_agent_strategy`	single_agent:██████████37/20 sub_agents:██████████37/20
`daily-architecture-diagram.md`	`detail_level`	brief:█████░░░░░10/20 comprehensive:███░░░░░░░6/20
`daily-astrostylelite-markdown-spellcheck.md`	`prompt_style`	concise:██████████67/20 detailed:██████████64/20
`daily-cache-strategy-analyzer.md`	`model_size`	gpt-5.3-codex:█░░░░░░░░░2/20 gpt-5.3-codex-spark:██░░░░░░░░3/20
`daily-caveman-optimizer.md`	`model_size`	agent:░░░░░░░░░░1/20 claude-haiku-4.5:██████████57/20 claude-sonnet-4.6:██████████34/20 claude-sonnet-5:██░░░░░░░░5/20 small-agent:█░░░░░░░░░2/20
`daily-code-metrics.md`	`output_format`	executive_summary:██████████50/20 full_detail:██████████55/20 ste:██████░░░░12/20
`daily-community-attribution.md`	`prompt_style`	concise:██████████64/20 verbose:██████████65/20
`daily-compiler-quality.md`	`output_format`	concise:██████████50/20 detailed:██████████56/20 ste:██████░░░░12/20
`daily-doc-healer.md`	`model_size`	agent:█░░░░░░░░░2/20 claude-haiku-4.5:██████████50/20 claude-sonnet-4.6:██████████43/20 claude-sonnet-5:██░░░░░░░░3/20 small-agent:░░░░░░░░░░1/20
`daily-doc-updater.md`	`model_size`	agent:░░░░░░░░░░1/20 claude-haiku-4.5:██████████42/20 claude-sonnet-4.6:██████████47/20 claude-sonnet-5:███░░░░░░░6/20 small-agent:█░░░░░░░░░2/20
`daily-fact.md`	`reasoning_depth`	multi_candidate:██████████34/30 single_pass:██████████52/30
`daily-function-namer.md`	`model_size`	N/A
`daily-issues-report.md`	`output_format`	collapsible:██████████55/20 inline:██████████58/20 ste:████████░░16/20
`daily-news.md`	`prompt_style`	concise:██████████54/20 detailed:██████████47/20
`daily-rendering-scripts-verifier.md`	`remove_redundant_context_v1`	N/A
`daily-safe-output-optimizer.md`	`log_fetch_strategy`	N/A
`daily-security-red-team.md`	`reasoning_depth`	iterative:██████████74/30 single_pass:██████████48/30
`daily-semgrep-scan.md`	`semgrep_output_format`	bullet_list:█████████░28/30 prose:█████████░27/30 structured_sections:███████░░░22/30
`dailysubagentoptimizer (file removed)`	`timeout_setting`	default:█░░░░░░░░░2/20 relaxed:░░░░░░░░░░1/20 tight:░░░░░░░░░░1/20
`dataflow-pr-discussion-dataset.md`	`caveman_mode`	no:██░░░░░░░░5/20 yes:████░░░░░░9/20
`deep-report.md`	`output_format`	annotated_brief:██████████43/20 executive_brief:██████████64/20 full_briefing:██████████51/20 ste:██████████19/20
`dependabotcampaign (file removed)`	`summary_detail`	brief:███░░░░░░░6/20 detailed:███░░░░░░░6/20
`dependabot-go-checker.md`	`prompt_style`	concise:████████░░16/20 detailed:████████░░16/20 step_by_step:██████░░░░13/20
`gpclean.md`	`tool_verbosity`	full_bash:██████████52/20 minimal_toolset:██████████58/20
`issue-arborist.md`	`prompt_style`	concise:██████████37/20 detailed:██████████32/20
`plan.md`	`reasoning_depth`	baseline:██░░░░░░░░4/20 deep:█░░░░░░░░░2/20 shallow:░░░░░░░░░░1/20
`pr-sous-chef.md`	`remove_redundant_context_v1`	N/A
`smokeantigravity (file removed)`	`sub_agent_strategy`	single_agent:██████████99/20 sub_agents:██████████109/20
`smoke-copilot.md`	`caveman`	no:██████████178/20 yes:██████████178/20
`smoke-copilot.md`	`subagent_model`	large:██████████168/20 small:██████████168/20
`smoke-copilot-aoai-apikey.md`	`caveman`	no:██████████91/20 yes:██████████90/20
`smoke-copilot-aoai-apikey.md`	`subagent_model`	large:██████████90/20 small:██████████91/20
`smoke-copilot-aoai-entra.md`	`caveman`	no:██████████77/20 yes:██████████77/20
`smoke-copilot-aoai-entra.md`	`subagent_model`	large:██████████77/20 small:██████████77/20
`smoke-copilot-sub-agents.md`	`sub_agent_strategy`	delegated_sequential:█████░░░░░14/30 inline_strict:████░░░░░░12/30 single_agent_control:█████░░░░░15/30
`smoke-gemini.md`	`sub_agent_strategy`	single_agent:██████████124/20 sub_agents:██████████140/20
`smoke-pi.md`	`sub_agent_decomposition`	parallel_sub_agents:██████████42/20 single_agent:██████████34/20
`smoke-project.md`	`prompt_style_test`	N/A
`smoke-temporary-id.md`	`sub_agent_strategy`	N/A
`test-quality-sentinel.md`	`model_size`	N/A
`typist.md`	`tone_style`	conversational:██████████43/20 formal:██████████35/20
`weekly-blog-post-writer.md`	`prefetch_strategy`	eager:██░░░░░░░░5/20 lazy:████░░░░░░8/20

🔬 Detailed ASCII Comparison — READY experiments

Experiment : prompt_compression
Workflow   : agent-performance-analyzer.md
Hypothesis : H0: no change in effective_tokens. H1: caveman reduces tokens by ≥20% while maintaining quality ≥90%
min_samples: 20 per variant | total_runs: 113
  caveman              n=62
  verbose              n=51
Balance test  : chi_square p=0.301 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : sub_agent_strategy
Workflow   : agent-persona-explorer.md
Hypothesis : H0: no change in effective_tokens or duration. H1: batch reduces tokens by ≥20% and duration by ≥15% without quality loss
min_samples: 20 per variant | total_runs: 113
  batch                n=59
  per_scenario         n=54
Balance test  : chi_square p=0.643 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : tone_variant
Workflow   : aw-failure-investigator.md
Hypothesis : H0: no change in output_length_chars across tone variants. H1: assertive tone produces shorter, more actionable outputs than clinical or narrative, with equivalent or better sub-issue quality.
min_samples: 20 per variant | total_runs: 406
  assertive            n=140
  clinical             n=135
  narrative            n=131
Balance test  : chi_square p=0.858 (balanced)
Core decision : INCONCLUSIVE
Reason code   : unsupported_multi_variant
Rationale     : automatic decisions currently require exactly two variants
----------------------------------------------------------------------
Experiment : prompt_style
Workflow   : ci-coach.md
Hypothesis : H0: no change in PR merge rate or proposal quality. H1: concise prompt reduces token usage ≥25% without degrading proposal quality
min_samples: 20 per variant | total_runs: 85
  concise              n=37
  detailed             n=48
Balance test  : chi_square p=0.000 (balanced)
Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
----------------------------------------------------------------------
Experiment : sub_agent_strategy
Workflow   : daily-agentrx-trace-optimizer.md
Hypothesis : H0: no change in issue quality or run success rate. H1: sub_agents variant yields higher evidence completeness score with equal or lower token cost
min_samples: 20 per variant | total_runs: 74
  single_agent         n=37
  sub_agents           n=37
Balance test  : chi_square p=1.000 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : prompt_style
Workflow   : daily-astrostylelite-markdown-spellcheck.md
Hypothesis : Concise prompt reduces token consumption ≥20% without degrading fix precision. H0: no difference in fix rate.
min_samples: 20 per variant | total_runs: 131
  concise              n=67
  detailed             n=64
Balance test  : chi_square p=0.783 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : prompt_style
Workflow   : daily-community-attribution.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 129
  concise              n=64
  verbose              n=65
Balance test  : chi_square p=0.891 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : reasoning_depth
Workflow   : daily-fact.md
Hypothesis : H0: no change in discussion engagement rate. H1: multi_candidate produces more novel verses with higher reaction counts (expected +20% reactions).
min_samples: 30 per variant | total_runs: 86
  multi_candidate      n=34
  single_pass          n=52
Balance test  : chi_square p=0.049 (imbalanced)
Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "empty_output_rate" is not backed by a supported metric observation
----------------------------------------------------------------------
Experiment : prompt_style
Workflow   : daily-news.md
Hypothesis : H0: no change in output quality. H1: concise prompt reduces token usage by ≥20% with no significant drop in output completeness score
min_samples: 20 per variant | total_runs: 101
  concise              n=54
  detailed             n=47
Balance test  : chi_square p=0.493 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : reasoning_depth
Workflow   : daily-security-red-team.md
Hypothesis : H0: no change in finding quality. H1: iterative reduces false-positive rate by >=20% at <=30% token overhead
min_samples: 30 per variant | total_runs: 122
  iterative            n=74
  single_pass          n=48
Balance test  : chi_square p=0.018 (imbalanced)
Core decision : EXTEND
Reason code   : guardrail_unsupported
Rationale     : mandatory guardrail "run_success_rate" is not backed by a supported metric observation
----------------------------------------------------------------------
Experiment : tool_verbosity
Workflow   : gpclean.md
Hypothesis : H0: no change in token consumption. H1: minimal toolset reduces tokens by 10-15% while maintaining issue quality (detection accuracy + alternative research depth)
min_samples: 20 per variant | total_runs: 110
  full_bash            n=52
  minimal_toolset      n=58
Balance test  : chi_square p=0.575 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : prompt_style
Workflow   : issue-arborist.md
Hypothesis : H0: no change in links_created. H1: detailed instructions produce ≥15% more correct links per run
min_samples: 20 per variant | total_runs: 69
  concise              n=37
  detailed             n=32
Balance test  : chi_square p=0.555 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : sub_agent_strategy
Workflow   : smokeantigravity (file removed)
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 208
  single_agent         n=99
  sub_agents           n=109
Balance test  : chi_square p=0.495 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : caveman
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 356
  no                   n=178
  yes                  n=178
Balance test  : chi_square p=1.000 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : subagent_model
Workflow   : smoke-copilot.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 336
  large                n=168
  small                n=168
Balance test  : chi_square p=1.000 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : caveman
Workflow   : smoke-copilot-aoai-apikey.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 181
  no                   n=91
  yes                  n=90
Balance test  : chi_square p=0.899 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : subagent_model
Workflow   : smoke-copilot-aoai-apikey.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 181
  large                n=90
  small                n=91
Balance test  : chi_square p=0.899 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : caveman
Workflow   : smoke-copilot-aoai-entra.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 154
  no                   n=77
  yes                  n=77
Balance test  : chi_square p=1.000 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : subagent_model
Workflow   : smoke-copilot-aoai-entra.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 154
  large                n=77
  small                n=77
Balance test  : chi_square p=1.000 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : sub_agent_strategy
Workflow   : smoke-gemini.md
Hypothesis : H0: no change in effective_tokens. H1: sub_agents reduces tokens by >=20%
min_samples: 20 per variant | total_runs: 264
  single_agent         n=124
  sub_agents           n=140
Balance test  : chi_square p=0.326 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : sub_agent_decomposition
Workflow   : smoke-pi.md
Hypothesis : (not specified)
min_samples: 20 per variant | total_runs: 76
  parallel_sub_agents  n=42
  single_agent         n=34
Balance test  : chi_square p=0.362 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------
Experiment : tone_style
Workflow   : typist.md
Hypothesis : H0: no change in engagement. H1: conversational tone increases discussion views+reactions+comments by 20%+ while maintaining analysis quality
min_samples: 20 per variant | total_runs: 78
  conversational       n=43
  formal               n=35
Balance test  : chi_square p=0.368 (balanced)
Core decision : EXTEND
Reason code   : insufficient_observations
Rationale     : primary metric observations are not available for statistical comparison
----------------------------------------------------------------------

📊 Summary

View Full Experiments Table

Experiment	Workflow	Control	Candidate	Evidence	Guardrails	Core decision	Report action
`prompt_compression`	`agent-performance-analyzer.md`	`caveman`	`verbose`	p=0.301	—	EXTEND	EXTEND
`sub_agent_strategy`	`agent-persona-explorer.md`	`batch`	`per_scenario`	p=0.643	—	EXTEND	EXTEND
`sub_agent_strategy`	`architecture-guardian.md`	`single_agent`	`sub_agents`	p=0.524	—	EXTEND	EXTEND
`audit_decomposition`	`audit-workflows.md`	`single_agent`	`phased_sub_agents`	p=0.334	—	EXTEND	EXTEND
`tone_variant`	`aw-failure-investigator.md`	`—`	`—`	p=0.858	—	INCONCLUSIVE	INCONCLUSIVE
`prompt_style`	`blog-auditor.md`	`concise`	`detailed`	p=0.319	—	EXTEND	EXTEND
`tone_variant`	`breaking-change-checker.md`	`neutral`	`urgent`	p=0.535	—	EXTEND	EXTEND
`prompt_style`	`ci-coach.md`	`detailed`	`concise`	p=0.000	n/a (unsupported)	EXTEND	EXTEND
`output_format`	`copilot-agent-analysis.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`sub_agent_strategy`	`daily-agentrx-trace-optimizer.md`	`single_agent`	`sub_agents`	p=1.000	—	EXTEND	EXTEND
`detail_level`	`daily-architecture-diagram.md`	`brief`	`comprehensive`	p=0.319	—	EXTEND	EXTEND
`prompt_style`	`daily-astrostylelite-markdown-spellcheck.md`	`concise`	`detailed`	p=0.783	—	EXTEND	EXTEND
`model_size`	`daily-cache-strategy-analyzer.md`	`gpt-5.3-codex`	`gpt-5.3-codex-spark`	p=0.659	—	EXTEND	EXTEND
`model_size`	`daily-caveman-optimizer.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`output_format`	`daily-code-metrics.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`prompt_style`	`daily-community-attribution.md`	`concise`	`verbose`	p=0.891	—	EXTEND	EXTEND
`output_format`	`daily-compiler-quality.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`model_size`	`daily-doc-healer.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`model_size`	`daily-doc-updater.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`reasoning_depth`	`daily-fact.md`	`single_pass`	`multi_candidate`	p=0.049	n/a (unsupported)	EXTEND	EXTEND
`model_size`	`daily-function-namer.md`	`—`	`—`	p=0.000	—	EXTEND	EXTEND
`output_format`	`daily-issues-report.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`prompt_style`	`daily-news.md`	`concise`	`detailed`	p=0.493	—	EXTEND	EXTEND
`remove_redundant_context_v1`	`daily-rendering-scripts-verifier.md`	`—`	`—`	p=0.000	—	EXTEND	EXTEND
`log_fetch_strategy`	`daily-safe-output-optimizer.md`	`eager`	`lazy`	p=0.000	—	EXTEND	EXTEND
`reasoning_depth`	`daily-security-red-team.md`	`single_pass`	`iterative`	p=0.018	n/a (unsupported)	EXTEND	EXTEND
`semgrep_output_format`	`daily-semgrep-scan.md`	`—`	`—`	p=0.023	—	INCONCLUSIVE	INCONCLUSIVE
`timeout_setting`	`dailysubagentoptimizer (removed)`	`—`	`—`	p=0.781	—	INCONCLUSIVE	INCONCLUSIVE
`caveman_mode`	`dataflow-pr-discussion-dataset.md`	`no`	`yes`	p=0.285	—	EXTEND	EXTEND
`output_format`	`deep-report.md`	`—`	`—`	p=0.000	—	INCONCLUSIVE	INCONCLUSIVE
`summary_detail`	`dependabotcampaign (removed)`	`brief`	`detailed`	p=1.000	—	EXTEND	EXTEND
`prompt_style`	`dependabot-go-checker.md`	`—`	`—`	p=0.819	—	INCONCLUSIVE	INCONCLUSIVE
`tool_verbosity`	`gpclean.md`	`full_bash`	`minimal_toolset`	p=0.575	—	EXTEND	EXTEND
`prompt_style`	`issue-arborist.md`	`concise`	`detailed`	p=0.555	—	EXTEND	EXTEND
`reasoning_depth`	`plan.md`	`—`	`—`	p=0.369	—	INCONCLUSIVE	INCONCLUSIVE
`remove_redundant_context_v1`	`pr-sous-chef.md`	`—`	`—`	p=0.000	—	EXTEND	EXTEND
`sub_agent_strategy`	`smokeantigravity (removed)`	`single_agent`	`sub_agents`	p=0.495	—	EXTEND	EXTEND
`caveman`	`smoke-copilot.md`	`no`	`yes`	p=1.000	—	EXTEND	EXTEND
`subagent_model`	`smoke-copilot.md`	`large`	`small`	p=1.000	—	EXTEND	EXTEND
`caveman`	`smoke-copilot-aoai-apikey.md`	`no`	`yes`	p=0.899	—	EXTEND	EXTEND
`subagent_model`	`smoke-copilot-aoai-apikey.md`	`large`	`small`	p=0.899	—	EXTEND	EXTEND
`caveman`	`smoke-copilot-aoai-entra.md`	`no`	`yes`	p=1.000	—	EXTEND	EXTEND
`subagent_model`	`smoke-copilot-aoai-entra.md`	`large`	`small`	p=1.000	—	EXTEND	EXTEND
`sub_agent_strategy`	`smoke-copilot-sub-agents.md`	`—`	`—`	p=0.802	—	INCONCLUSIVE	INCONCLUSIVE
`sub_agent_strategy`	`smoke-gemini.md`	`single_agent`	`sub_agents`	p=0.326	—	EXTEND	EXTEND
`sub_agent_decomposition`	`smoke-pi.md`	`parallel_sub_agents`	`single_agent`	p=0.362	—	EXTEND	EXTEND
`prompt_style_test`	`smoke-project.md`	`—`	`—`	p=0.000	—	EXTEND	EXTEND
`sub_agent_strategy`	`smoke-temporary-id.md`	`single_agent`	`sub_agents`	p=0.000	—	EXTEND	EXTEND
`model_size`	`test-quality-sentinel.md`	`—`	`—`	p=0.000	—	EXTEND	EXTEND
`tone_style`	`typist.md`	`conversational`	`formal`	p=0.368	—	EXTEND	EXTEND
`prefetch_strategy`	`weekly-blog-post-writer.md`	`eager`	`lazy`	p=0.410	—	EXTEND	EXTEND

Descriptive window: last 30 runs per workflow (where available) · Decision thresholds: from decision_policy (95% confidence, min_samples per experiment config)
Run: 34576316456

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

awfailureinvestigator / tone_variant and copilotagentanalysis, dailycodemetrics, dailycompilerquality, dailyissuesreport, dailysemgrepscan, dailydochealer, dailydocupdater, dailycavemanoptimizer, dependabotgochecker, plan, smokecopilotsubagents, dailysubagentoptimizer, deepreport are all INCONCLUSIVE due to unsupported_multi_variant — prioritize adding 3+ variant statistical support (e.g. ANOVA/Kruskal-Wallis) to the core analyzer so these 14 experiments (largest single blocker) can progress.
cicoach, dailyfact, dailysecurityredteam are READY but blocked by guardrail_unsupported — wire native observation collection for their guardrail metrics (run_success_rate, quality scores) so guardrail pass/fail can be evaluated and these experiments can reach a real decision.
The largest group (25 experiments, insufficient_observations) is READY or actively accumulating samples but has no primary-metric per-run observation pipeline at all — this is the single highest-leverage fix; prioritize effective_tokens, run_duration_ms, and run_success_rate observation collection since they are the most common primary metrics declared in frontmatter.

Top 3 eval actions

Workflows testing output_format variants (copilotagentanalysis, dailycodemetrics, dailycompilerquality, dailyissuesreport, deepreport, dailysemgrepscan) declare eval:output_format_adherence-style secondary metrics but lack graders — add an eval question per workflow asking "Does the report match the declared output_format (prose/ste/structured)?" to produce the observation the core analyzer needs.
dailycavemanoptimizer, dailydochealer, dailydocupdater, testqualitysentinel, dailycachestrategyanalyzer test model_size — add an eval question scoring output quality (e.g. "Did the smaller model produce an equally actionable recommendation?") to give the guardrail pipeline a quality signal alongside cost.
agentperformanceanalyzer, agentpersonaexplorer, dailyagentrxtraceoptimizer, smokeantigravity, smokegemini, smokepi test sub_agent_strategy — tighten the existing output_quality_score / scenarios_analyzed eval questions to be binary pass/fail so they map cleanly onto guardrail_metrics thresholds instead of free-text scores.

Decision-pipeline gaps for next PR

insufficient_observations (25 experiments) and guardrail_unsupported (3 experiments) both stem from the same root cause: no per-run primary/guardrail metric write-back into experiments/* branch state. This is the top priority gap.
unsupported_multi_variant (14 experiments) blocks every 3+ variant test regardless of sample size — core analysis needs an omnibus test (ANOVA / Kruskal-Wallis / chi-square-based) with pairwise post-hoc comparison and Bonferroni correction (the bonferroni_alpha field already exists in the schema but is unused while this gap persists).
No workflow currently runs 2+ concurrent experiments, so the factorial-interaction diagnostic (buildFactorialInteractionCells) was not exercised this run. Once dailycavemanoptimizer-style multi-experiment workflows appear, promote the interaction check from a reporting-only safety hold into a normalized core analysis signal.

Descriptive window: last 30 runs per workflow · Decision thresholds: from decision_policy
Run: 34576316456

Generated by 🧪 daily-experiment-report · copilot · auto · 153 AIC · ⌖ 18.4 AIC · ⊞ 16.7K · ◷

expires on Sep 14, 2026, 12:05 AM UTC-08:00

Navigation Menu

Uh oh!

[experiments] Daily Experiment Report — 2026-09-11 #60223

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

Uh oh!

[experiments] Daily Experiment Report — 2026-09-11 #60223

Uh oh!

github-actions[bot] Bot Sep 11, 2026

🧪 Daily Experiment Report — 2026-09-11

⚡ Quick Stats

📋 Experiments by Decision Reason

📊 Summary

🔧 Self-Tuning Continuation Plan

Replies: 0 comments

github-actions[bot]
Bot Sep 11, 2026