[prompt-clustering] Copilot Agent Prompt Clustering — 2026-06-22 (1,000 PRs, 79.2% merge)

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-06-02 → 2026-06-22) · Repository: github/gh-aw
Tasks Analyzed: 1,000 Copilot agent PRs · Clusters: 12 (k chosen by silhouette = 0.054)
Overall Merge Rate: 79.2% (792 merged · 201 closed · 7 open)

At a glance

• Overall merge rate is flat vs yesterday (79.2% → 79.2%) and holding inside its stable 78–81% band over the trailing month.
• WIP merge rate climbed to 41.0% (from 37.2% on 06-20/06-21) — the third straight uptick. Explicitly-WIP PRs still merge at roughly half the rate of normal ones, but the gap is narrowing.
• Highest-success themes are operational hygiene: domain/firewall + CLI version bumps (95.1%) and AIC usage/forecast reporting (87.3%).
• Lowest-success themes are safe-output handlers (72.9%) and the ambient-context optimizer (71.0%) — both touch generated runtime output and remain the most iteration-heavy areas.

Full Analysis Report

Methodology

• Source: pre-fetched copilot-prs.json (1,000 PRs authored by app/copilot-swe-agent, all in github/gh-aw).
• Text = PR title (weighted ×2) + body, with code blocks, URLs, and the boilerplate agent footer stripped. Domain-noise terms (workflow, gh, aw, fix, add, copilot, ...) added to the stop-list so clusters separate on subject rather than shared jargon.
• TF-IDF (uni+bi-grams, min_df=3, max_df=0.5, 400 features) → K-means, k selected by silhouette over k∈[6,12].
• Stale pr-full-data/ (early-May snapshot, ~30577 range) was excluded — it falls outside the 30-day window and has zero overlap with the current set, so per-PR turn/comment enrichment is unavailable for this period (see Limitations).

General Insights

• Most common theme: workflow/MCP/daily-automation plumbing (Cluster 0, 267 PRs / 27% of all work) — the bulk of agent activity is maintaining the agentic-workflow runtime itself.
• Highest success: domain/firewall allow-listing + routine CLI version bumps (95.1%) — small, well-scoped, mechanical changes.
• Lowest success: safe-output handler work (72.9%) and ambient-context optimizer hardening (71.0%) — changes to generated output are riskier and get closed more often.
• WIP signal: 39 PRs are explicitly titled WIP; they merge at 41.0% vs 80.8% for non-WIP. Cluster 10 ("failing GitHub Actions job") is almost entirely WIP (20 of 21) and is the clearest "in-flight / may not land" bucket.

Cluster Analysis

#	Theme (top terms)	Tasks	Merge rate	WIP
0	github / daily / mcp / step / prompt	267	81.7%	6
11	cjs / failure / test / js / artifact	121	76.9%	1
7	safe / safe outputs / output / target	96	72.9%	5
9	docs / org / schema / reference	93	76.3%	1
5	pkg / string / linters / package	90	76.7%	3
4	ai / credits / max-ai-credits	76	84.2%	1
1	sdk / driver / permission / tool	68	75.0%	1
6	aic / forecast / usage / report	55	87.3%	0
2	chef / sous-chef / aic / gpt-mini	41	78.1%	0
3	domains / firewall / smoke / cli-version	41	95.1%	0
8	ambient / ambient-context / optimizer	31	71.0%	1
10	job / github-actions / failing / wip	21	71.4%	20

Notable clusters

• Cluster 0 — Workflow/MCP plumbing (267, 81.7%): the engine room. Daily-automation steps, MCP wiring, prompt scaffolding. Large and healthy. e.g. #38097 (codemods for cross-repo compile failures, merged).
• Cluster 3 — Domains/firewall + CLI bumps (41, 95.1%): highest success. Mechanical, low-ambiguity changes (allow-list edits, version chores). e.g. #40618 (CLI version updates, merged).
• Cluster 6 — AIC usage/forecast reporting (55, 87.3%): well-specified reporting features land reliably. e.g. #40645 (detection-AIC parsing, merged).
• Cluster 7 — Safe-output handlers (96, 72.9%): lowest-success large cluster. Touches generated runtime output; specs are subtle and review-heavy. e.g. #38100 (standardized error codes across handlers, merged) vs several closed iterations.
• Cluster 8 — Ambient-context optimizer (31, 71.0%): persistent low performer across recent days; hardening against payload/size failures keeps reopening. e.g. #40508 (placeholder/oversize-payload hardening, merged).
• Cluster 10 — Failing-Actions-job fixes (21, 71.4%, 20 WIP): the "in-flight reactive fix" bucket. Almost all WIP-titled; success rate reflects that many are still open or get superseded. e.g. #38262 [WIP] Fix failing GitHub Actions job.

Trend (trailing month)

Date	Merge rate	WIP merge rate	k
06-14	80.7%	—	10
06-15	80.2%	—	10
06-17	—	36.0%	8
06-20	80.0%	37.2%	11
06-21	79.2%	37.2%	12
06-22	79.2%	41.0%	12

Overall merge rate is stable; the moving story is the steady recovery of the WIP merge rate.

Key Findings

1. Scope predicts success. Small mechanical clusters (firewall/CLI bumps 95.1%, AIC reporting 87.3%) consistently beat clusters that mutate generated output (safe-outputs 72.9%, ambient-context 71.0%). The ~24-point spread is the dominant pattern.
2. Generated-output surfaces are the chronic weak spot. Safe-output handlers and the ambient-context optimizer have stayed in the low-70s across multiple days of history — this is structural, not noise.
3. WIP is a reliable low-merge signal, but improving. WIP PRs merge ~2× less often than normal, yet the WIP rate rose 37%→41% over the last three measurements, suggesting better triage of in-flight work.

Recommendations

1. Add pre-submit checklists for safe-output and ambient-context changes. These two clusters are the durable low performers; targeted guidance (expected schema, payload-size guards, golden-output diffs) should lift their merge rate fastest.
2. Keep favoring tightly-scoped tasks. The data strongly rewards small, mechanical PRs — prefer splitting broad runtime changes into firewall/CLI/version-sized units where possible.
3. Treat WIP-titled, failing-Actions-job PRs as a separate queue. Cluster 10 is ~95% WIP; tracking it apart from finished work avoids depressing headline metrics and surfaces genuinely stuck reactive fixes.

References: §27949632098

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 151 AIC · ⌖ 9.34 AIC · ⊞ 13.8K · ◷

• expires on Jun 23, 2026, 3:57 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #41004.