[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-05 (1,000 PRs, 81.1% merge rate)

[github-actions[bot]]

Summary

Analysis period: 2026-06-13 → 2026-07-05 (rolling 30-day window)
Tasks analyzed: 1,000 Copilot-agent PRs (997 copilot-swe-agent, 3 github-actions)
Clusters identified: 10 (K-means over TF-IDF of title+body, k chosen by silhouette; sil = 0.048)
Overall merge rate: 81.1% (799 merged / 186 closed / 15 open) — up +1.0pp from 80.1% on 2026-07-03

The single strongest quality signal remains WIP PRs: 34 PRs opened as [WIP] merge at only 35.3% vs 81.1% overall. That's up sharply from 24.2% two days ago, but WIP tasks are still ~2.3× more likely to be abandoned than normal ones.

Full analysis report

Key findings

1. Meta / prompt-engineering work is the weakest cluster. "guidance / report / prompt" (126 PRs, 12.6% of volume) merges at just 72% and carries 7 of the 34 WIP flags. These are workflow-authoring tasks — refactoring shared prompt bundles, tweaking report formatting, rotating cache-memory. Much of it is exploratory and gets closed (e.g. [WIP] Refactor to create shared/guard-otlp.md bundle for workflows #42983, [WIP] Refactor to create shared/serena-otlp.md bundle for workflows #42944 shared-bundle refactors; [WIP] Expand personas and add systematic cache-memory rotation #42090 persona expansion — all WIP/closed).

2. Mechanical / dependency work is nearly flawless. "smoke claude / domains / claude" (45 PRs) merges at 98% — version bumps, firewall/smoke-domain config, and MCP-container wiring. Long bodies (avg 3,554 chars, mostly smoke-test output) but low ambiguity. Similarly, "rule / cases / false" ESLint/analyzer rules (46 PRs) merge at 91%.

3. The CI-fixer cluster is a WIP factory. The smallest cluster, "actions / github actions / job" (16 PRs), is 15/16 WIP with the shortest bodies in the dataset (avg 412 chars — auto-generated "Fix failing GitHub Actions job for X" titles). It still lands at 75% merge, so the pattern works, but nearly every one starts as a draft/WIP retry loop.

4. Volume is concentrated in config & error-fixing. The two largest clusters — "schema / added / workflows" (239 PRs, 75%) and "error / failure / runs" (178 PRs, 81%) — together are 42% of all tasks. Schema/config work under-performs the mean (75%), suggesting spec/validation changes are where the agent most often gets a rejection.

Cluster analysis

Cluster	PRs	Merge rate	WIP	Avg body	Top terms
schema / added / workflows	239 (23.9%)	75%	2	1366	schema, added, workflows, updated, docs, github
error / failure / runs	178 (17.8%)	81%	4	1497	error, failure, runs, fix, new, agent
helper / existing / path	151 (15.1%)	85%	4	1324	helper, existing, path, behavior, added, change
guidance / report / prompt	126 (12.6%)	72%	7	1394	guidance, report, prompt, section, workflows
sous chef / chef	86 (8.6%)	87%	0	2113	sous chef, chef, id, aic, run
tool / denial / agent	68 (6.8%)	87%	2	1196	tool, denial, agent, daily, output, safe
rule / cases / false	46 (4.6%)	91%	0	1365	rule, cases, false, eslint, flagged
sub agent / model	45 (4.5%)	84%	0	1217	sub agent, sub, agent, model, models
smoke claude / domains	45 (4.5%)	98%	0	3554	smoke claude, domains, firewall, details
actions / github actions / job	16 (1.6%)	75%	15	412	actions, github actions, job, failing, fix

Representative PRs per cluster

• schema / workflows (75%): Add replace-label safe-outputs type (experimental) #40423 Add replace-label safe-outputs type · Implement mcp-scripts.dependencies end-to-end with runtime-manager install flow and pinned release-tag validation #39739 mcp-scripts.dependencies end-to-end · [WIP] Fix safe outputs MCP container path for consumer repos #39695 [WIP/closed] MCP container path for consumer repos
• error / failure (81%): fix: enforce minLength JSON schema constraints at MCP call time and extend '.' stdin sentinel to per-field CLI arguments #40497 enforce minLength JSON-schema at MCP call time · [WIP] Fix ARC/DinD topology path redirections in gh-aw compiler #42808 [WIP/closed] ARC/DinD topology path redirections
• helper / path (85%): Harden go-gh remote fetch callsites with escaped contents paths/refs and bounded REST clients #41036 harden go-gh remote fetch callsites · Align CLI help/grouping and flag semantics for command consistency #41727 align CLI help/grouping · Refactor non-workflow large functions into focused helpers #42179 [closed] refactor large functions into helpers
• guidance / prompt (72%): [WIP] Refactor to create shared/guard-otlp.md bundle for workflows #42983 / [WIP] Refactor to create shared/serena-otlp.md bundle for workflows #42944 [WIP/closed] shared-bundle refactors · [WIP] Expand personas and add systematic cache-memory rotation #42090 [WIP/closed] persona + cache-memory rotation
• sous chef (87%): Add model policy frontmatter + import unioning + env policy overrides #41824 model-policy frontmatter + import unioning · fix(eslint-factory): require-async-entrypoint-catch — track async arrow/fn-expr entrypoints and flag .then() without .catch() #43356 eslint-factory async-entrypoint catch
• tool / denial (87%): Add skillet skill-routed PR reviewer workflow #40399 skillet skill-routed PR reviewer · fix: prevent stdout/stderr interleaving in mcp_cli_bridge for large outputs #40037 fix stdout/stderr interleaving in mcp_cli_bridge
• rule / cases (91%): feat(linters): add deferinloop analyzer — flags defer inside for-loop bodies #40679 deferinloop analyzer · ctxbackground: fix false negative and unsafe autofix for closures #41187 ctxbackground false-negative fix
• sub agent / model (84%): feat(sous-chef): eliminate per-PR REST calls in pending-checks pre-filter #43471 eliminate per-PR REST calls in pending-checks pre-filter · Support per-model Copilot SDK wireApi via provider/model registry metadata #42497 per-model Copilot SDK wireApi
• smoke claude / domains (98%): Run safe-outputs MCP in the gh-aw node container #39100 run safe-outputs MCP in gh-aw node container · Generalize early wildcard-target validation across safe-outputs MCP tools #39300 wildcard-target validation
• actions / job (75%): [WIP] Fix failing GitHub Actions job for integration add #40239 / [WIP] Fix failing GitHub Actions job for integration add with dispatch-workflow #40109 / [WIP] Fix failing GitHub Actions job for integration unauthenticated add #43304 [WIP] "Fix failing GitHub Actions job for integration ..."

Trend vs previous run (2026-07-03)

Metric	2026-07-03	2026-07-05	Δ
Overall merge rate	80.1%	81.1%	+1.0pp
WIP merge rate	24.2%	35.3%	+11.1pp
WIP PRs	33	34	+1
Clusters (k)	12	10	−2

Merge rate has held in a tight 79–82% band across the whole recorded history (since 2026-06-17), so day-to-day moves are noise rather than regime change. The WIP merge-rate jump is the only material shift.

Methodology & limitations

• Features: TF-IDF (1–3 grams, min_df=3, max_df=0.55, sublinear, 800 features) over cleaned title+body; code fences, URLs, and #refs stripped.
• k selection: silhouette scan over k∈[8,14]; k=10 won (0.048). Silhouettes are low across the board (~0.03–0.10 historically) — prompts overlap heavily in vocabulary, so clusters are thematic tendencies, not hard partitions.
• "Merge rate" = success proxy: merged / (merged + closed), excluding the 15 still-open PRs.
• No workflow/turn metrics this run. The pre-fetched pr-full-data/ cache is stale (PRs #30k, dated May 24) and does not intersect the current window (#38k–43k), so turn counts, duration, and cost could not be joined. Analysis is prompt-text + outcome only — consistent with prior runs, which also lacked turn data.

Recommendations

1. Add a scoping/plan gate for "guidance / report / prompt" tasks. This 126-PR cluster (72% merge, 7 WIP) is where agent effort is most often abandoned. Requiring a short design note or splitting shared-bundle refactors into smaller PRs should recover several merges/day.
2. Treat [WIP] as a retry smell, not a status. WIP PRs merge at 35% and the CI-fixer cluster is 15/16 WIP. Consider having the agent iterate to green before opening, or auto-converting stale WIP drafts — a lot of the gap is retry churn on failing CI.
3. Lean into the high-confidence lanes. Dependency bumps, firewall/smoke config, and ESLint/analyzer rules merge at 91–98%. These are safe to keep fully autonomous; reserve human review budget for schema/config changes (75%), the largest and lowest-yield lane.

References: §28737657305

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 165.9 AIC · ⌖ 18.9 AIC · ⊞ 13.2K · ◷

• expires on Jul 6, 2026, 2:50 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #43773.