[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 1,000 PRs, Last 30 Days

[github-actions[bot]]

Summary

Analysis Period: PRs created 2026-06-26 → 2026-07-16 (last 30 days)
Total Copilot PRs Analyzed: 1,000 (app/copilot-swe-agent)
Clusters Identified: 8 (TF-IDF + K-means, k selected by silhouette)
Outcome mix: 649 merged · 194 closed · 157 still open
Overall merge rate: 64.9% (all PRs) — 77.0% among the 843 already-resolved PRs

The Copilot coding agent's work over the last month splits into eight recurring task families. The single largest is core agent-runtime & safe-outputs plumbing (41% of all PRs). The most reliably-merged work is small, targeted config tuning (Sous-Chef / sub-agent, 79% merged, ~2 files); the least reliable is version/dependency bumps (50% merged) — not because they're hard, but because they drag huge regenerated diffs (avg ~124 files touched).

Key Findings

1. Smaller PRs merge; big ones stall. Merged PRs average 1,319 lines of churn vs 3,063 lines for closed PRs — a 2.3× gap. Median PR touches just 4 files, but the mean is 24, so a long tail of sprawling PRs drives most of the closures.
2. Dependency bumps are the weak spot (50% merged). They carry the highest scope of any cluster (~124 files avg) because compiled/regenerated workflow files balloon the diff. This is the clearest optimization target.
3. Tightly-scoped tuning wins (79% merged). The Sous-Chef & sub-agent cluster has the fewest commits (2.23) and smallest scope, and the highest merge rate — well-bounded prompts convert best.
4. Review is the gate, not a formality. PRs that received a human review merged 80% of the time (n=808); the 192 with no review recorded merged only 1.6% (these are the open/abandoned tail).
5. 56 [WIP] PRs remain in flight — concentrated in CI-failure-fixing and the agent-runtime cluster.

Full Analysis Report

Methodology

• Corpus: 1,000 PRs authored by app/copilot-swe-agent, full metadata (body, comments, reviews, commits, files) pre-fetched per PR.
• Text: PR title + body, stripped of code fences, HTML, URLs and domain-noise stopwords (gh, aw, workflow, fix, add, ...), then TF-IDF (uni+bigrams, min_df=3, max_df=0.6, sublinear).
• Clustering: K-means over k=3..8; k=8 chosen by best silhouette. Caveat: silhouette peaked at only 0.031 — PR text is topically overlapping, so clusters are thematic tendencies, not hard-separated groups. Labels below are human-assigned from top terms + sampled titles.
• Complexity proxy: gh-aw logs turn-counts do not apply here (Copilot's coding agent is a distinct system from gh-aw engines), so commit count and files changed stand in for iteration/scope. This is noted as a limitation.

General Insights

• Most common task type: Agent runtime & safe-outputs (41.1% of PRs).
• Highest merge rate: Sous-Chef & sub-agent tuning (79.1%).
• Lowest merge rate: Version / dependency bumps (50.0%).
• Highest iteration/scope: CLI & engine features (7.6 commits, 28.9 files avg).
• Outliers: version bumps regenerating compiled workflows push file counts into the hundreds (feat: upgrade GitHub MCP server to v1.6.0, enable fields_param by default #45832, Add experimental Auggie shared engine as a built-in declarative engine #45905 touched 279–303 files).

Cluster Summary

Cluster (theme)	Size	Merge rate	Avg commits	Avg files	Top terms
Agent runtime & safe-outputs	411 (41.1%)	63.7%	3.67	20.8	agent, run, output, path, step
Refactor into shared helpers	171 (17.1%)	68.4%	3.89	12.6	package, helper, helpers, spec, shared
Docs, prompts & instructions	146 (14.6%)	63.7%	2.82	16.4	docs, guidance, prompt, reference, section
CLI & engine features	90 (9.0%)	71.1%	7.61	28.9	engine, sous-chef, aic, bootstrap
ESLint rule factory	62 (6.2%)	61.3%	3.29	4.8	rule, eslint, suggestion, factory, catch
Version / dependency bumps	60 (6.0%)	50.0%	3.77	123.6	bump, version, release, pinned
Sous-Chef & sub-agent tuning	43 (4.3%)	79.1%	2.23	17.9	sub-agent, model, nudge, agents
CI job failure fixes (WIP)	17 (1.7%)	64.7%	2.29	4.5	failing actions job

Cluster Detail

Agent runtime & safe-outputs — 411 PRs, 63.7% merged. Core engine plumbing: safe-output handlers, job/step wiring, token minting, harness fail-fast. The default bucket where most day-to-day fixes land. Example: #45827 fast-fail copilot harness on LLM cap.

Refactor into shared helpers — 171 PRs, 68.4% merged. Splitting large functions, extracting packages, adding focused tests. Low scope (12.6 files), healthy merge rate. Example: #45635 split a 430-line function.

Docs, prompts & instructions — 146 PRs, 63.7% merged. Documentation, frontmatter, prompt/instruction-file edits. Fewest commits (2.82). Example: #45826 maxLength exemption in schema docs.

CLI & engine features — 90 PRs, 71.1% merged. New CLI flags, engine bootstrap flows, feature frontmatter. Highest iteration (7.6 commits) — feature work needs the most back-and-forth. Example: #45831 --create bootstrap flow.

ESLint rule factory — 62 PRs, 61.3% merged. Surgical, single-purpose lint-rule work; smallest scope (4.8 files). Example: #44706.

Version / dependency bumps — 60 PRs, 50.0% merged (lowest). Bumping CLI/MCP/action versions and regenerating compiled workflows → enormous diffs (avg 124 files). The churn, not the intent, drives closures. Example: #45832 touched 279 files.

Sous-Chef & sub-agent tuning — 43 PRs, 79.1% merged (highest). Small nudge-heuristic and sub-agent model/config tweaks. Tight scope + clear intent = best conversion. Example: #43888.

CI job failure fixes — 17 PRs, 64.7% merged. Auto-generated [WIP] Fix failing GitHub Actions job ... PRs; tiny diffs (4.5 files). Example: #45935.

Representative PRs

PR	Title	Cluster	Outcome	Commits	Files
#45827	fast-fail copilot harness when LLM cap saturated	Agent runtime & safe-outputs	🟡 open	28	255
#45908	Honor repositories: ["*"] in token minting	Agent runtime & safe-outputs	🟡 open	21	8
#45635	split downloadRunArtifactsConcurrent (430 lines)	Refactor into shared helpers	✅ merged	13	2
#44981	Formalize replace-label compliance model	Refactor into shared helpers	🟡 open	13	3
#45826	maxLength exemption for body/content fields	Docs, prompts & instructions	✅ merged	23	14
#44373	document sink visibility field	Docs, prompts & instructions	✅ merged	10	1
#44465	declarative custom-engine frontmatter	CLI & engine features	🟡 open	27	27
#45831	--create repository bootstrap flow	CLI & engine features	🟡 open	23	18
#44706	scope-aware Octokit client detection	ESLint rule factory	✅ merged	8	3
#45964	consolidate fs-sync try/catch rules	ESLint rule factory	🟡 open	6	4
#45832	upgrade GitHub MCP server to v1.6.0	Version / dependency bumps	🟡 open	19	279
#45905	experimental Auggie shared engine	Version / dependency bumps	🟡 open	16	303
#43888	relax pr-sous-chef checks-pending guard	Sous-Chef & sub-agent tuning	✅ merged	8	3
#45459	harden Sous-Chef pre-agent queue fetch	Sous-Chef & sub-agent tuning	✅ merged	4	4
#45935	[WIP] Fix failing GitHub Actions job agent	CI job failure fixes	✅ merged	5	2
#45927	[WIP] Fix failing Integration: Workflow Misc	CI job failure fixes	✅ merged	5	7

Recommendations

1. Split dependency bumps from regenerated output. The 50% merge rate for version bumps is a diff-size problem: separate the version change from the compiled-workflow regeneration (or auto-commit regeneration on merge) so reviewers see a small, intentional diff.
2. Keep prompts tightly scoped. The 79%-merging Sous-Chef cluster and the sub-2-file ESLint/CI clusters show that one-file, single-intent tasks convert best. Prefer decomposing broad asks into narrow PRs over one sprawling PR (mean 24 files, but the winners sit near the median of 4).
3. Triage the 157 open + 56 WIP backlog. Open PRs cluster in agent-runtime and CLI-features — the highest-iteration areas. A weekly sweep to land or close these would lift the headline 64.9% closer to the 77% resolved-rate.
4. Treat "no review" as an early-abandon signal. Zero-review PRs merge 1.6% of the time; surfacing un-reviewed Copilot PRs older than N days would recover otherwise-lost work.

---

Generated by Prompt Clustering Analysis · methodology: TF-IDF + K-means (k=8, silhouette 0.031, thematic) · complexity via commit/file proxies (gh-aw turn-logs N/A for Copilot agent) · Run: §29490568873

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 196.8 AIC · ⌖ 42.3 AIC · ⊞ 7K · ◷

• expires on Jul 17, 2026, 2:51 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #46203.