[experiments] Daily Experiment Report — 2026-09-04

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-09-04

48 experiment workflows declare experiments: in this repository; 47 have written state to their experiments/<id> branch, spanning 51 individual A/B experiments across model, prompt, output-format, and sub-agent-strategy variants. 25 experiments have reached min_samples for every variant (🟢 READY); 22 are still accumulating runs (🟡 COLLECTING); 4 declared-experiment workflows (dailyrenderingscriptsverifier, prsouschef, smokeproject, smoketemporaryid) have no state file yet — likely brand-new or not yet executed on their experiment branch.

⚠️ Data-pipeline gap: gh aw experiments analyze could not run in this environment (outbound git access to github.com is proxy-blocked), so this report was built directly from each experiments/<id> branch's state.json (assignment counts only) fetched via the GitHub MCP server, plus each workflow's frontmatter (min_samples, issue, metric, hypothesis). No core decision/reason_code/readiness field, no per-run outcome (success rate, duration), and no guardrail pass/fail could be computed — those require the gh aw experiments analyze binary's decision engine and 30-run outcome correlation, which needs authenticated gh/git access not available here. All statuses below are descriptive readiness (n ≥ min_samples) and chi-square balance only, not statistical decisions. Treat this as a partial/interim report.

⚡ Quick Stats

Metric	Value
Workflows declaring experiments:	48
Workflows with state data	47
Total individual experiments	51
Reached min_samples (🟢 READY)	25
Still collecting (🟡 COLLECTING)	22
Missing state file	4
Core decisions computed	0 (blocked — see gap above)

📈 View Full Experiment Table (n, min_samples, balance p-value, status)

Balance p-value is a chi-square goodness-of-fit test against equal assignment across variants (higher = more balanced random assignment). It is not an outcome-significance test.

Workflow                                  Experiment              Variants (n)                                                      min_samples  Balance p  Status
agent-performance-analyzer                prompt_compression      caveman=41, verbose=33                                           14           0.352      READY
agent-persona-explorer                    sub_agent_strategy      batch=47, per_scenario=26                                        14           0.014      READY
aw-failure-investigator                   tone_variant            clinical=93, narrative=75, assertive=80                          50           0.352      READY
ci-coach                                  prompt_style            concise=25, detailed=31                                          20           0.423      READY
daily-agentrx-trace-optimizer             sub_agent_strategy      sub_agents=23, single_agent=21                                   20           0.763      READY
daily-astrostylelite-markdown-spellcheck  prompt_style            detailed=41, concise=50                                          30           0.345      READY
daily-code-metrics                        output_format           executive_summary=36, full_detail=41                             20           0.569      READY
daily-community-attribution               prompt_style            concise=44, verbose=45                                           20           0.916      READY
daily-compiler-quality                    output_format           detailed=40, concise=38                                          20           0.821      READY
daily-issues-report                       output_format           collapsible=43, inline=45                                        30           0.831      READY
daily-news                                prompt_style            detailed=31, concise=40                                          30           0.285      READY
daily-security-red-team                   reasoning_depth         iterative=48, single_pass=34                                     30           0.122      READY
deep-report                                output_format           executive_brief=27, full_briefing=22, annotated_brief=15         15           0.182      READY
gpclean                                    tool_verbosity          full_bash=32, minimal_toolset=38                                 30           0.473      READY
smoke-antigravity                          sub_agent_strategy      single_agent=99, sub_agents=109                                  20           0.488      READY
smoke-copilot                              caveman                 no=155, yes=154                                                  20           0.955      READY
smoke-copilot                              subagent_model          large=145, small=144                                            20           0.953      READY
smoke-copilot-aoai-apikey                  caveman                 yes=78, no=78                                                    20           1.000      READY
smoke-copilot-aoai-apikey                  subagent_model          small=78, large=78                                               20           1.000      READY
smoke-copilot-aoai-entra                   caveman                 no=64, yes=65                                                    20           0.930      READY
smoke-copilot-aoai-entra                   subagent_model          large=65, small=64                                               20           0.930      READY
smoke-gemini                               sub_agent_strategy      sub_agents=130, single_agent=109                                 30           0.174      READY
smoke-pi                                   sub_agent_decomposition single_agent=34, parallel_sub_agents=42                          20           0.359      READY
test-quality-sentinel                      model_size              claude-sonnet-4.6=260, claude-haiku-4.5=269                     70           0.696      READY
typist                                     tone_style              conversational=25, formal=24                                    10           0.886      READY
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
architecture-guardian                      sub_agent_strategy      sub_agents=18, single_agent=15                                  20           0.602      COLLECTING
audit-workflows                            audit_decomposition     phased_sub_agents=15, single_agent=13                            264          0.705      COLLECTING
blog-auditor                                prompt_style            detailed=2, concise=1                                           20           0.564      COLLECTING
breaking-change-checker                    tone_variant            urgent=8, neutral=11                                             194          0.491      COLLECTING
copilot-agent-analysis                     output_format           structured=30, prose=25                                          30           0.500      COLLECTING
daily-architecture-diagram                 detail_level            brief=6, comprehensive=3                                         10           0.317      COLLECTING
daily-cache-strategy-analyzer              model_size              small-agent=1, agent=1, gpt-5.4-mini=18, gpt-5-mini=2, gpt-5-codex=8, gpt-5.4=28   20   0.000  COLLECTING
daily-caveman-optimizer                    model_size              small-agent=2, agent=1, claude-sonnet-4.6=23, claude-haiku-4.5=33 20           0.000      COLLECTING
daily-doc-healer                           model_size              small-agent=1, agent=2, claude-haiku-4.5=31, claude-sonnet-4.6=25 20           0.000      COLLECTING
daily-doc-updater                          model_size              small-agent=2, agent=1, claude-haiku-4.5=26, claude-sonnet-4.6=31 20           0.000      COLLECTING
daily-fact                                  reasoning_depth         single_pass=33, multi_candidate=24                              30           0.233      COLLECTING
daily-function-namer                       model_size              claude-haiku-4-5-20251001=3                                     20           n/a        COLLECTING
daily-safe-output-optimizer                 log_fetch_strategy      lazy=19, eager=18                                               20           0.869      COLLECTING
daily-semgrep-scan                          semgrep_output_format   structured_sections=22, prose=27, bullet_list=28                 30           0.669      COLLECTING
daily-subagent-optimizer                    timeout_setting         default=2, relaxed=1, tight=1                                   20           0.779      COLLECTING
dataflow-pr-discussion-dataset              caveman_mode            yes=5, no=3                                                     10           0.480      COLLECTING
dependabot-campaign                         summary_detail          brief=6, detailed=6                                             20           1.000      COLLECTING
dependabot-go-checker                       prompt_style            step_by_step=10, concise=9, detailed=9                          30           0.965      COLLECTING
issue-arborist                              prompt_style            concise=14, detailed=10                                        30           0.414      COLLECTING
plan                                         reasoning_depth         shallow=1, baseline=4, deep=2                                   137          0.368      COLLECTING
smoke-copilot-sub-agents                    sub_agent_strategy      delegated_sequential=3, single_agent_control=5, inline_strict=2  30           0.497      COLLECTING
weekly-blog-post-writer                     prefetch_strategy       lazy=5, eager=2                                                 10           0.257      COLLECTING

Note: daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-doc-healer, daily-doc-updater show model_size variant sets that include stray legacy labels (small-agent, agent, single-digit counts) alongside the current model names — these look like leftover assignments from before a variant-set change and should be reviewed/pruned so balance tests aren't skewed.

🟢 Ready for Analysis (min_samples reached on all variants)

View 25 ready experiments with tracking issues

Experiment	Workflow	Tracking issue
prompt_compression	agent-performance-analyzer	#33280
tone_variant	aw-failure-investigator	#36105
prompt_style	ci-coach	#32335
output_format	daily-code-metrics	#1
output_format	daily-compiler-quality	#32390
output_format	daily-issues-report	#30573
prompt_style	daily-news	#31190
reasoning_depth	daily-security-red-team	#31673
model_size	test-quality-sentinel	#43530
tone_style	typist	#34032

The remaining 15 ready experiments (agent-persona-explorer, daily-agentrx-trace-optimizer, daily-astrostylelite-markdown-spellcheck, daily-community-attribution, deep-report, gpclean, smoke-antigravity, smoke-copilot ×2, smoke-copilot-aoai-apikey ×2, smoke-copilot-aoai-entra ×2, smoke-gemini, smoke-pi) have no issue: field configured, so no tracking-issue notification applies per Step 8 rules.

No core decisions (PROMOTE/EXTEND/REJECT/INCONCLUSIVE) are reported this run — the decision engine (gh aw experiments analyze) requires outbound git/API access that this environment's network policy blocks. Readiness above reflects sample-size and balance only.

🔧 Self-Tuning Continuation Plan

Top 3 experiment actions

1. Restore gh aw experiments analyze execution in the reporting environment (grant outbound git/gh access to github.com, or add an MCP-based equivalent) so core decision/reason_code/readiness fields can be computed instead of this descriptive-only fallback.
2. Investigate the 4 workflows with experiments: declared but no state.json on their branch (daily-rendering-scripts-verifier, pr-sous-chef, smoke-project, smoke-temporary-id) — confirm push_experiments_state job is running and succeeding for these.
3. Clean up stray legacy variant labels in model_size experiments (daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-doc-healer, daily-doc-updater) so future balance/decision tests aren't computed over obsolete single-digit variant buckets.

Top 3 eval actions

1. Once decision-engine access is restored, prioritize evals for the 10 issue:-tracked ready experiments above (ci-coach, daily-security-red-team, test-quality-sentinel, etc.) since they already have a stakeholder-facing tracking issue awaiting a result.
2. Add native outcome-metric collection (success rate, duration, guardrail pass/fail) directly into state.json at the workflow level, rather than requiring a secondary 30-run GitHub Actions API correlation — this is the same "insufficient_observations" gap seen across nearly all experiments here.
3. Tighten grading for model_size experiments with >2 variants and imbalanced legacy buckets (daily-cache-strategy-analyzer chi-square p≈0.000) so pruning of stale variants happens automatically rather than via manual audit.

Decision-pipeline gaps for next PR

• No experiment in this run produced evidence/decision_guardrails — this indicates guardrail and outcome-metric collection is not yet wired to a queryable source usable outside a full gh-authenticated environment; consider exposing state.json outcome fields (success/duration) directly so descriptive-only sandboxes can still compute meaningful guardrail pass/fail.
• No interaction diagnostics were computed since no workflow currently runs 2+ simultaneous experiments observed together in this dataset; smoke-copilot and its AOAI variants run two experiments (caveman, subagent_model) per workflow and would be the first good candidates for factorial interaction-cell analysis once outcome correlation is available.

Descriptive window: full experiments/<id> branch state (assignment counts only) · Decision thresholds: not evaluated (see data-pipeline gap)
Run: 33850615177

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by 🧪 daily-experiment-report · copilot · auto · 143.7 AIC · ⌖ 18.1 AIC · ⊞ 16.9K · ◷

• expires on Sep 7, 2026, 12:10 AM UTC-08:00