[prompt-clustering] Copilot Prompt Clustering — 2026-07-21: merge rate 76.5% (▼), WIP prompts the top drag

[github-actions[bot]]

Summary

Analysis window: 2026-06-30 → 2026-07-21 (latest 1000 Copilot PRs in github/gh-aw)
Copilot PRs analyzed: 1000
Clusters identified: 9 (k-means, cosine; cohesion 0.4178)
Overall merge rate: 76.5% (753 merged · 231 closed · 16 open)
Trend (last 8 runs): ▇▆█▅▃▁▁▁ ▼ -4.3% (from 80.8%)

Headline: the wip / progress / started cluster (37 PRs) merges at only 37.8% — the biggest drag on overall throughput. These read as work-in-progress / abandoned task descriptions.

Full analysis report

Merge-rate trend

Date	Merge rate	k
2026-07-02	80.8%	11
2026-07-03	80.1%	12
2026-07-05	81.1%	10
2026-07-07	79.4%	12
2026-07-10	—	10
2026-07-15	78.0%	12
2026-07-18	76.9%	12
2026-07-19	77.1%	9
2026-07-21 ← today	76.5%	9

Cluster analysis

K-means over TF-IDF (unigrams+bigrams, 400-term vocab) grouped the 1000 prompts into 9 themes. Clusters ordered by size.

Cluster 1: before / step / job

• Size: 176 PRs (18%) · dominant intent fix
• Merge rate: 78.9% (138 merged / 37 closed)
• Top terms: before, step, job, after, because, feat
• Examples: #45827 fix: fast-fail copilot harness when LLM invocation cap is sa (merged); #42585 _safe-outputs: pre-flight workflow scope check + full-branch _ (merged); #46058 feat(observability): pre-bundle Process Safe Outputs step lo (closed)

Cluster 2: sous / chef / sous chef

• Size: 141 PRs (14%) · dominant intent fix
• Merge rate: 87.1% (121 merged / 18 closed)
• Top terms: sous, chef, sous chef, workflow sous, chef engine, aic
• Examples: #43850 _fix(pr-code-quality-reviewer): add Go ecosystem to firewall _ (merged); #43716 fix(security): replace curl|sh installer pipes with download (merged); #44320 [review] docs: document sink-visibility field in MCP Gateway (closed)

Cluster 3: prompt / guidance / report

• Size: 139 PRs (14%) · dominant intent docs
• Merge rate: 75.5% (105 merged / 34 closed)
• Top terms: prompt, guidance, report, docs, body, issue
• Examples: #44960 Normalize progressive-disclosure guidance across reporting w (merged); #44314 Normalize report body formatting rules across 11 reporting w (merged); #45703 Normalize report header guidance across non-compliant report (closed)

Cluster 4: regression / coverage / regression coverage

• Size: 130 PRs (13%) · dominant intent other
• Merge rate: 76.7% (99 merged / 30 closed)
• Top terms: regression, coverage, regression coverage, behavior, path, docs
• Examples: #45554 Compiler: fix docker-sbx runtime output for TTY, CLI staging (merged); #46378 Fix workshop progress rail state coupling and responsive ove (merged); #45471 Fix Crush workflow model forwarding for non-interactive runs (closed)

Cluster 5: shared / refactor / helpers

• Size: 118 PRs (12%) · dominant intent refactor
• Merge rate: 83.8% (98 merged / 19 closed)
• Top terms: shared, refactor, helpers, helper, import, call
• Examples: #43649 refactor(linters): consolidate duplicated AST/context helper (merged); #42868 Consolidate duplicate log parsers and replace local helper r (merged); #44439 Refactor semantic helper clusters in workflow/cli (tools set (closed)

Cluster 6: copilot coding / coding agent / start copilot

• Size: 116 PRs (12%) · dominant intent fix
• Merge rate: 76.4% (81 merged / 25 closed)
• Top terms: copilot coding, coding agent, start copilot, agent suffix, coding, suffix
• Examples: #46800 chore(deps): update charmbracelet/x/exp/golden to v0.0.0-202 (merged); #46555 fix: pin SHA-256 checksum for sentrux binary install (RGS-01 (merged); #46928 Switch AI Moderator engine from codex to copilot (closed)

Cluster 7: bump / version / golden

• Size: 73 PRs (7%) · dominant intent chore
• Merge rate: 62.5% (45 merged / 27 closed)
• Top terms: bump, version, golden, pinned, review, claude
• Examples: #45215 Bump pinned agentic CLI/MCP tool versions for 2026-07-13 (merged); #45670 chore: bump default Claude/Codex/Pi CLI versions and regener (merged); #46158 Update pinned CLI tool versions for 2026-07-17 (closed)

Cluster 8: spec / missing / table

• Size: 70 PRs (7%) · dominant intent docs
• Merge rate: 74.3% (52 merged / 18 closed)
• Top terms: spec, missing, table, docs, gaps, sync
• Examples: #43594 SPDD 2026-07-05: outcome evaluators, policy compiler, and sp (merged); #46283 docs(spdd): add safeguards, edge cases, ops order, norms, an (merged); #45526 Add spec-conformance guardrails for AI Credits catalog/versi (closed)

Cluster 9: wip / progress / started

• Size: 37 PRs (4%) · dominant intent fix
• Merge rate: 37.8% (14 merged / 23 closed)
• Top terms: wip, progress, started, date, plan, description
• Examples: #43296 [WIP] Fix failing GitHub Actions job 'JS Tests (shard 1/4)' (closed); #43305 [WIP] Fix failing GitHub Actions job lint-go (closed); #46766 [WIP] Fix failing GitHub Actions job 'agent' (merged)

Intent breakdown (conventional-commit heuristic)

Intent	PRs	Merged	Closed	Merge rate
fix	279	223	46	82.9%
feat	149	108	40	73.0%
ci	130	95	33	74.2%
other	125	100	25	80.0%
docs	120	86	34	71.7%
test	80	60	19	75.9%
refactor	73	54	18	75.0%
chore	34	21	12	63.6%
perf	10	6	4	60.0%

Key findings

1. WIP prompts are the dominant failure mode. The wip / progress / started cluster (37 PRs) merges at just 37.8%. Prompts here describe partial/started work rather than a crisp, verifiable task.
2. Bug-fix tasks lead on completion. fix intent PRs (279) merge at 82.9% — well-scoped, testable asks succeed most.
3. Refactor/shared-helper work is reliable. The shared / refactor / helpers cluster merges at 83.8%, showing mechanical, well-bounded changes complete cleanly.
4. Chore/version bumps lag. chore intent merges at 63.6% — often superseded or duplicated.
5. Merge rate is drifting down -4.3% across the tracked window (now 76.5%), suggesting rising task complexity or more speculative WIP PRs.

Recommendations

1. Triage WIP-style prompts before dispatch. ~37 PRs this window came from prompts that read as unfinished ("wip", "progress", "started"). Require a concrete acceptance criterion / definition-of-done in the task before handing off to the agent.
2. Favor fix/refactor framing. The highest-merging clusters are bug fixes and shared-helper refactors. When feasible, decompose large features into scoped, testable units.
3. Batch or gate chore/version bumps. Their low merge rate suggests duplication; consolidate dependency updates so the agent isn't racing itself.
4. Track the trend. Merge rate has slipped; watch whether it stabilizes or keeps declining over the next runs and correlate with task-type mix.

Methodology & limitations

• Clustering ran in pure-Python TF-IDF + cosine k-means; the scikit-learn/numpy NLP stack was not installable in this run (no build network), so the pipeline was reimplemented with the standard library. Results are deterministic.
• Prompt proxy: PR bodies (the agent's task summary) were used as the prompt signal; the literal dispatch prompt is not stored in this dataset.
• No turn/duration enrichment: gh-aw was not on PATH this run, so per-PR turn counts, duration, and cost could not be joined. Merge/close outcome is the success proxy.
• Firewall-warning and code-fence boilerplate were stripped before vectorizing.

---

References: §29822514185
[history already has today — skipped append]

[cache updated]

Generated by 📊 Copilot Agent Prompt Clustering Analysis · age00 250.2 AIC · ⌖ 42.2 AIC · ⊞ 13.1K · ◷

• expires on Jul 22, 2026, 3:03 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-22T11:03:01.691Z.

Closed by Workflow