[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-23 (1,000 PRs, 76.3% merge rate)

[github-actions[bot]]

Summary

Analysis window: Jul 2 → Jul 23, 2026 (last 30 days) · PRs analyzed: 1,000 (989 by copilot-swe-agent)
Outcomes: 763 merged · 234 closed · 3 open · Merge (success) rate: 76.3%
Clusters identified: 10 (silhouette-selected k) · Cohesion: 0.043 · Avg PR body: ~1,670 chars

Merge rate is flat vs yesterday (76.4% → 76.3%, −0.1pp) and sits right on the 16-day mean. The prompt population is stable and healthy — the actionable signal is concentrated in two under-performing niches (release/version-bump chores and the docs-site audit tail), not in the broad workload.

Key findings

1. Bug-fix prompts remain the reliable core. The fix task type is 42% of all work (416 PRs) and merges at 81% — well above the 76.3% baseline. Sharply-scoped rule/lint fixes (lines / import / rule cluster) hit 84.7%, and firewall/domain-allowlist fixes (blocked / firewall / aic) top the board at 91.7%. Precise, acceptance-criteria-driven prompts convert.
2. Two clusters drag the average and deserve prompt attention. version / release / changes (chore/dependency bumps, 42 PRs) merges at only 54.8%, and the site / docs / docs-site audit tail (21 PRs) at just 23.8% — the clear outlier. Together they are ~6% of volume but account for a disproportionate share of closures.
3. Broad "workflow/agent" prompts are the biggest below-average bucket. At 192 PRs and 66.7%, this cluster is large and soft — the vaguer the surface area ("update the workflow / agent output"), the lower the conversion, consistent with the fix-vs-open-ended gap seen across the whole dataset.

Cluster analysis (10 clusters, ranked by size)

#	Cluster (top terms)	Size	Merge rate	Dominant type	Example PRs
1	engine / config / step	238	75.6%	fix	#47517, #47471, #47462
2	workflow / agent / workflows	192	66.7%	fix	#47508, #47506, #47429
3	added / spec / test	138	81.2%	fix	#47516, #47482, #47474
4	lines / import / rule	137	84.7%	fix	#47455, #47333, #47330
5	aic / sous-chef	99	86.9%	fix	#47515, #47507, #47419
6	workshop / step / token	66	83.3%	docs	#47480, #47335, #47328
7	blocked / firewall / domains	48	91.7%	fix	#47237, #47232, #47149
8	version / release / changes	42	54.8%	chore	#47469, #47430, #47405
9	site / docs / docs-site	21	23.8%	docs	#45399, #45281, #44803
10	actions / fix / job	19	73.7%	fix	#47279, #47266, #46859

Silhouette by k: {4: .024, 5: .033, 6: .035, 7: .038, 8: .039, 9: .042, 10: .043}. Cohesion is intrinsically low because TF-IDF prompt space is high-dimensional; k=10 maximized separation. Themes are stable vs the 07-22 run (fix core strong, docs-site + release chores weak).

Success rate by task type (conventional-commit classification)

Type	PRs	Merged	Closed	Merge rate
fix	416	337	76	81.0%
test	65	51	14	78.5%
feat	120	92	28	76.7%
refactor	85	65	20	76.5%
ci	106	78	28	73.6%
docs	154	107	47	69.5%
perf	13	9	4	69.2%
chore	29	17	12	58.6%
other	12	7	5	58.3%

fix is both the largest and the highest-converting type; chore (dependency/version work) is the weakest sizeable type, reinforcing the cluster-level finding. Type classification is heuristic (title + body verbs) and independent from the 07-22 classifier, so treat type counts as a snapshot, not a day-over-day delta — the tracked, comparable trend is the overall merge rate above.

Methodology & limitations

• Source: 1,000 most-recent Copilot PRs (pr-data/copilot-prs.json, snapshot 2026-07-23). Prompt text extracted from the <issue_title>/<issue_description> blocks in each PR body, then stripped of code fences, HTML/tracker comments, URLs, and memory IDs.
• Vectorization: TF-IDF (1–2 grams, max 500 features, min_df=3, max_df=0.6). Clustering: K-Means, k∈[4,10] chosen by silhouette. Success = merged.
• Trend: appended to cache-memory/clustering/history.jsonl (16 comparable merge-rate points); per-day cluster snapshot cached; analyzed-PR list deduped to avoid rework.
• Limitations: Workflow-run enrichment (turns/duration/cost) was not joinable — the cached full-PR dataset predates this window (May PRs, no number overlap), so no gh-aw logs matching was possible. Report is prompt- and outcome-based only. Silhouette scores are low by construction for sparse text; clusters are directional, not hard partitions.

Recommendations

1. Tighten release/dependency-bump prompts (cluster Add workflow: githubnext/agentics/weekly-research #8, 54.8%). These chores fail nearly half the time. Give them explicit, verifiable acceptance criteria (exact version target, CVE/gate to clear, files to touch) rather than open-ended "refresh/update" phrasing — the firewall cluster shows how much precise constraints lift conversion.
2. Triage the docs-site audit tail (cluster Weekly Research Report: AI Workflow Automation Landscape and Strategic Opportunities - August 2025 #9, 23.8%). This 21-PR outlier is the single worst pocket. Audit whether these prompts are too broad ("audit the docs site") or duplicative; convert them into narrowly-scoped, single-file tasks like the high-performing workshop docs cluster (83.3%).
3. Add scope guards to broad "workflow/agent" prompts (cluster Add workflow: githubnext/agentics/weekly-research #2, 192 PRs, 66.7%). The largest below-baseline bucket. Front-load these with a concrete target file/function and a done-condition to pull them toward the 81% fix norm.

References: §29999578819

Generated by 📊 Copilot Agent Prompt Clustering Analysis · age00 160.6 AIC · ⌖ 37.9 AIC · ⊞ 13.1K · ◷

• expires on Jul 24, 2026, 3:02 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #47770.