You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Most iteration-heavy: Cluster 3 (engine, threat, copilot) — avg 7.7 comments, 4.2 reviews per PR
Data note: workflow-run logs (aw_info.json) were not available in this run's cache, so turn counts/duration/cost per PR could not be joined in. Analysis relies on PR body/title text plus comment, review, and diff-size signals.
Data cleaning note: ~33% of PR bodies carried a bot-generated attribution footer ("Generated by PR Sous Chef ...") that initially formed a spurious cluster. This was stripped before vectorization so clusters reflect actual task content, not footer boilerplate.
Cluster Analysis
Cluster 1: mcp, github, docs — general workflow/docs fixes
Size: 406 tasks (40.6% of total)
Success rate: 77.8%
Avg comments/reviews: 5.7 / 2.8
Avg diff: 18.0 files changed (+257/-2879 lines)
Top keywords: mcp, github, docs, agent, workflows, md, fix, workflow
Representative titles: "Optimize make test-unit for sub-30s feedback via impacted-first execution", "Close linter doc-sync gap by bridging registry, prose docs, and spec labels", "[actionpins] Add missing internal coverage and table-drive action pin mapping tests"
Lowest success rate among the "core engineering" clusters (69.7%) despite the smallest diffs — suggests test/CLI PRs draw more scrutiny per line changed.
Representative titles: "Add undocumented flag to BaseEngine to stop antigravity drift-detection churn", "fix: add pi.dev to Pi engine firewall allowlist", "fix: resolve 16 CLI consistency issues from 2026-07-22 inspection"
Most iteration-heavy cluster — engine/version-compatibility and threat-detection changes draw the most back-and-forth review.
Representative titles: "feat: add container_pins to aw.json and document action/container substitutions", "Add container image substitutions for private registries", "Refresh gh-aw-node packages and multi-platform publishing"
Clear outlier: nearly 1-in-2 of these PRs close unmerged. Container/image digest pinning and version-bump PRs likely hit external CI/registry flakiness or supply-chain gating that isn't reflected in diff size alone.
Cluster 0: model, pricing, models — model config, pricing, AI credits
Size: 53 tasks (5.3% of total)
Success rate: 79.2% — highest of all clusters
Avg comments/reviews: 7.0 / 3.3
Avg diff: 44.8 files changed (+283/-125 lines)
Top keywords: model, pricing, models, claude, copilot, ai credits, credits, ai
Representative titles: "fix(claude): disable Anthropic SDK inner retries to prevent terminal-error storm", "chore: bump CLI versions — Claude Code 2.1.217, Codex 0.145.0, Pi 0.81.1", "fix: add · separator between model alias and AIC value in footer"
Smallest cluster but best success rate — model/pricing-catalog updates tend to be well-scoped and mechanical.
Refactor schedule heatmap styling to use centralized th...
5
merged
22
11
pkg, test, cli
48589
fix: move sink-visibility to env var to prevent zizmor ...
6
merged
41
4
workflow, safe, job
48169
Preserve HTTPS angle-bracket autolinks in safe-output s...
6
merged
38
5
workflow, safe, job
48363
Add sanitization-safe metadata channel for body-based...
6
merged
31
1
workflow, safe, job
49226
feat: per-output github-app override in safe-outputs
6
merged
21
4
workflow, safe, job
49477
Allow add-labels to opt out of pull-requests: write
6
merged
20
10
workflow, safe, job
Key Findings
Container/image pinning work has the weakest close rate (58.8%) — nearly 1 in 2 of these PRs don't merge, well below the 74.3% overall average. This cluster covers Docker digest bumps, action/container pin substitutions, and multi-platform publishing — likely subject to registry flakiness, version drift, or supply-chain gating that a pure text/diff analysis can't fully explain.
Model/pricing-catalog PRs are the most reliably mergeable (79.2%) despite being a small, narrow cluster — these tasks tend to be mechanical (version bumps, pricing table updates) and well-scoped.
Engine/threat-detection changes draw the most review back-and-forth (7.7 comments, 4.2 reviews per PR on average) — version-compatibility and security-detection logic apparently needs more scrutiny before merge.
General workflow/docs/MCP fixes dominate volume (40.6% of all tasks) — this is the largest and most heterogeneous cluster; a future pass could split it further (e.g., separating docs-only changes from MCP-specific fixes).
A third of PR bodies carry a "PR Sous Chef" attribution footer that had to be filtered out before clustering — worth keeping in mind for any other automated text analysis over this repo's PR corpus.
Recommendations
Investigate the container/image cluster's close rate. Since diff size and comment/review counts for this cluster look unremarkable, the low merge rate likely stems from external factors (CI flakiness, registry auth, timing of upstream digest changes) rather than PR quality — worth a manual look at a sample of the ~35 closed-but-not-merged PRs in this group.
Consider tighter task scoping for engine/threat-detection PRs, given they carry the highest review overhead; splitting version-compatibility fixes from threat-detection logic changes into separate, smaller tasks may reduce iteration count.
Re-run this analysis with workflow-run logs joined in (turn counts, duration, AI credit cost) once gh-aw logs cache is populated, to correlate cluster membership with actual agent effort, not just PR-level signals.
Strip bot-attribution footers repo-wide in any future NLP/text-analysis tooling over this PR corpus, since ~1/3 of bodies carry the "PR Sous Chef" signature block, which otherwise forms a spurious cluster.
Generated by Prompt Clustering Analysis (Run: 31168402535)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Summary
Analysis Period: Last 30 days (2026-07-08 → 2026-08-07)
Total Tasks Analyzed: 1000 copilot-created PRs
Clusters Identified: 7
Overall Success Rate: 74.3% merged
Full Analysis Report
General Insights
aw_info.json) were not available in this run's cache, so turn counts/duration/cost per PR could not be joined in. Analysis relies on PR body/title text plus comment, review, and diff-size signals.Cluster Analysis
Cluster 1: mcp, github, docs — general workflow/docs fixes
copilot/branches #47199, feat(eslint): addno-setfailed-then-exit-zerorule #47232, chore: remove withinDeferredBoundary dead code in require-execsync-try-catch rule #47233, docs: document automatic Node.js setup for self-hosted/GPU runners #47234, Fix compiler emission bug: mount-mcp-clis step reference out of scope in generated lock files #47235copilot/branches", "feat(eslint): addno-setfailed-then-exit-zerorule", "chore: remove withinDeferredBoundary dead code in require-execsync-try-catch rule"This is the catch-all bucket for general lint rules, docs, and small workflow fixes — expected given the repo's focus on agentic workflow tooling.
Cluster 6: workflow, safe, job — safe-outputs / job wiring
engine.env.COPILOT_GITHUB_TOKEN#47270Cluster 5: pkg, test, cli — test suite / CLI package work
make test-unitfor sub-30s feedback via impacted-first execution #47200, Close linter doc-sync gap by bridging registry, prose docs, and spec labels #47204, [actionpins] Add missing internal coverage and table-drive action pin mapping tests #47206, Add Pi offline regression coverage to wasm golden engine test #47261, Refactor audit report largefunc slice inpkg/cli#47262make test-unitfor sub-30s feedback via impacted-first execution", "Close linter doc-sync gap by bridging registry, prose docs, and spec labels", "[actionpins] Add missing internal coverage and table-drive action pin mapping tests"Lowest success rate among the "core engineering" clusters (69.7%) despite the smallest diffs — suggests test/CLI PRs draw more scrutiny per line changed.
Cluster 3: engine, threat, copilot — engine/version/threat-detection logic
undocumentedflag to BaseEngine to stop antigravity drift-detection churn #47160, fix: add pi.dev to Pi engine firewall allowlist #47243, fix: resolve 16 CLI consistency issues from 2026-07-22 inspection #47337, Surface aw.yml config during add-wizard before generic engine setup #47462, Add add-wizard integration coverage for manifest bootstrap ordering #47517undocumentedflag to BaseEngine to stop antigravity drift-detection churn", "fix: add pi.dev to Pi engine firewall allowlist", "fix: resolve 16 CLI consistency issues from 2026-07-22 inspection"Most iteration-heavy cluster — engine/version-compatibility and threat-detection changes draw the most back-and-forth review.
Cluster 4: image, container, digest — container/image pinning & publishing
Clear outlier: nearly 1-in-2 of these PRs close unmerged. Container/image digest pinning and version-bump PRs likely hit external CI/registry flakiness or supply-chain gating that isn't reflected in diff size alone.
Cluster 0: model, pricing, models — model config, pricing, AI credits
Smallest cluster but best success rate — model/pricing-catalog updates tend to be well-scoped and mechanical.
Cluster 2: pull request, request, pull — PR/branch/safe-output plumbing
update_pull_request#47961, Handle workflow_dispatch PR targeting in deterministic safe-output sample replay #48398, fix: wrap fork push sequences in withGitHubHostToken to prevent duplicate Authorization (HTTP 400) #48595, fix(add_comment): remove expired AWI_MAINTENANCE_TOKEN, add 401 fallback to step-level token #48725update_pull_request", "Handle workflow_dispatch PR targeting in deterministic safe-output sample replay"Success Rate by Cluster
Representative Sample Table (top-5 most-discussed PRs per cluster, 35 of 1000)
largefuncfi...metadatachannel for body-based...add-labelsto opt out ofpull-requests: writeKey Findings
Recommendations
gh-aw logscache is populated, to correlate cluster membership with actual agent effort, not just PR-level signals.Generated by Prompt Clustering Analysis (Run: 31168402535)
All reactions