[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-24 (1000 PRs, 5 clusters)

[github-actions[bot]]

Summary

• Analysis window: 2026-07-03 → 2026-07-24 (Copilot-authored PRs in github/gh-aw)
• Tasks analyzed: 1000
• Overall merge (success) rate: 76.5% (765 merged / 1000)
• Clusters identified: 5 (chosen by silhouette across k=3–7)
• Method: pure-Python TF-IDF (uni+bigram) + spherical k-means — the run's scikit-learn/numpy stack was unavailable under PyPy, so an equivalent stdlib implementation was used.

Headline finding

[WIP]-prefixed PRs merge at 26% vs a 76% baseline — 56 tasks, only 15 merged. These are agent runs that stalled or were abandoned mid-work, and many are repeated retries of the same task (e.g. Fix failing GitHub Actions job js-integration-live-api recurs 3×; Add robots.txt to docs site recurs ~5×). This single pattern accounts for the bulk of wasted cycles.

Cluster analysis

#	Theme	Tasks	Share	Success	Top type	Top keywords
2	Core engineering — compiler / safe-outputs / workflow internals	490	49.0%	76.9%	bug	path, workflow, behavior, output, job, workflows
1	Dependency & version bumps (AWF, firewall, CLI pins)	189	18.9%	81.5%	chore	bump, version, cli, claude, package, smoke claude
4	Agent-generated tasks (sous-chef / AIC security & fixes)	139	13.9%	83.5%	bug	sous, chef, sous chef, sub, aic, chef run
0	Docs site & workshop content	137	13.7%	73.7%	docs	docs, workshop, content, page, body, site
3	Auto-generated CI / [WIP] fix attempts	45	4.5%	37.8%	bug	failing github, actions job, actions, github actions, job, failing

Representative PRs per cluster

Cluster 2 — Core engineering — compiler / safe-outputs / workflow internals (490 tasks, 76.9% merged)

• Guard setup JS sync fs/exec calls with contextual error handling #47425 [MERGED] Guard setup JS sync fs/exec calls with contextual error handling
• docs(spdd): tighten normative language, add safeguards, expand glossary (rotation 34–38) #46886 [CLOSED] docs(spdd): tighten normative language, add safeguards, expand glossary (rotation 34–38)
• Refactor safe-outputs config parsing into focused workflow modules #44313 [MERGED] Refactor safe-outputs config parsing into focused workflow modules
• [file-diet] Split activation job builder into focused compiler modules #44792 [MERGED] [file-diet] Split activation job builder into focused compiler modules

Cluster 1 — Dependency & version bumps (AWF, firewall, CLI pins) (189 tasks, 81.5% merged)

• Bump AWF to v0.27.27 and MCPG to v0.4.0 #44173 [MERGED] Bump AWF to v0.27.27 and MCPG to v0.4.0
• chore: upgrade gh-aw-firewall to v0.27.24 #43664 [MERGED] chore: upgrade gh-aw-firewall to v0.27.24
• chore: bump gh-aw-firewall to v0.27.28 #44467 [MERGED] chore: bump gh-aw-firewall to v0.27.28
• Bump gh-aw-firewall to v0.27.31 and refresh AWF pin/schema artifacts #45176 [MERGED] Bump gh-aw-firewall to v0.27.31 and refresh AWF pin/schema artifacts

Cluster 4 — Agent-generated tasks (sous-chef / AIC security & fixes) (139 tasks, 83.5% merged)

• fix(security): replace curl|sh installer pipes with download+verify (RGS-018) #43716 [MERGED] fix(security): replace curl|sh installer pipes with download+verify (RGS-018)
• fix: expose erl in AWF sandbox when using runtimes.elixir #44283 [MERGED] fix: expose erl in AWF sandbox when using runtimes.elixir
• feat: content redaction for public-facing safe-outputs #44501 [CLOSED] feat: content redaction for public-facing safe-outputs
• test(constants): table-driven assertions and extend octal-literal check to cmd/ #44114 [MERGED] test(constants): table-driven assertions and extend octal-literal check to cmd/

Cluster 0 — Docs site & workshop content (137 tasks, 73.7% merged)

• Add docs-site robots.txt allowlist and generate llms.txt from docs content #44364 [CLOSED] Add docs-site robots.txt allowlist and generate llms.txt from docs content
• Replace in-doc workshop experience with external workshop repository link #46827 [MERGED] Replace in-doc workshop experience with external workshop repository link
• Render workshop content through Astro-generated markdown sources #46406 [MERGED] Render workshop content through Astro-generated markdown sources
• Route workshop landing to org-scoped path (/workshop/hackathon-blue-bat-18/) #46579 [MERGED] Route workshop landing to org-scoped path (/workshop/hackathon-blue-bat-18/)

Cluster 3 — Auto-generated CI / [WIP] fix attempts (45 tasks, 37.8% merged)

• [WIP] Fix failing GitHub Actions job Integration: CMD Tests #45681 [MERGED] [WIP] Fix failing GitHub Actions job Integration: CMD Tests
• [WIP] Fix failing GitHub Actions job 'agent' #46766 [MERGED] [WIP] Fix failing GitHub Actions job 'agent'
• [WIP] Fix failing GitHub Actions job agent #45935 [MERGED] [WIP] Fix failing GitHub Actions job agent
• [WIP] Fix failing GitHub Actions job test #43295 [MERGED] [WIP] Fix failing GitHub Actions job test

Cross-cuts (from triage labels)

pr-type	Tasks	Success		pr-risk	Tasks	Success
bug	148	83%		low	217	78%
feature	81	81%		medium	180	80%
chore	77	74%		high	48	83%
refactor	60	78%
docs	45	82%
test	25	80%

Success is remarkably flat across type and risk (74–83%). Outcome is driven by execution completion, not task category or stated risk — reinforcing that the [WIP]/stall pattern, not task difficulty, is the lever.

Trend vs previous run (2026-07-23)

Metric	2026-07-23	2026-07-24	Δ
Overall success rate	76.3%	76.5%	+0.2pp (stable)
Tasks analyzed	~1000	1000	—
Clusters (k)	10	5	coarser, cleaner separation
Sous-chef / AIC cluster success	86.9%	83.5%	strongest cohort both days
Docs-site cluster success	23.8%	73.7%	recovered materially

Recommendations

1. Add a stall/abandon guard for [WIP] runs. 74% of [WIP] PRs never merge and many are identical re-attempts. Deduplicate against open/closed [WIP] titles before dispatching, and cap automatic retries of a task that has already failed twice.
2. Special-case recurring infra tasks (js-integration-live-api fixes, robots.txt/llms.txt docs-site edits). Their repeated failure suggests a missing capability or flaky dependency rather than a prompt problem — route these to a human or a purpose-built workflow instead of the general agent.
3. Keep leaning on the sous-chef/AIC path (83.5% success, highest cohort): its narrowly-scoped, well-specified prompts convert best. Mirror that specificity in the core-engineering cluster (the 490-task catch-all at 76.9%).
4. Success is category-independent — don't gate by pr-risk/pr-type; invest triage effort in detecting incomplete/stalled runs early.

Notes & limitations

• Per-PR turn counts / cost were not joined: the cached pr-full-data is stale (May, PRs fix(deps_security): propagate context.Context through security advisory HTTP call #30577–34390) and does not overlap the current window (Home page: move Public Preview notice into hero with warning callout styling #43081–47730); pulling fresh workflow logs for ~1000 runs was out of scope for one run. Cluster complexity is proxied by body length and outcome.
• Silhouette scores are low (0.04) — expected for sparse high-dimensional text; clusters were validated by manual inspection of top terms + representative PRs.

References: §30086219882

Generated by 📊 Copilot Agent Prompt Clustering Analysis · age00 · 192.9 AIC · ⌖ 34.3 AIC · ⊞ 13.1K · ◷

• expires on Jul 25, 2026, 3:02 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #47972.