[prompt-clustering] Copilot Agent Prompt Clustering — 994 PRs, 7 themes (last 30d)

[github-actions[bot]]

Summary

Analysis period: last 30 days (2026-05-30 → 2026-06-19)
Copilot agent PRs analyzed: 994 (of 1,000 fetched; 6 had no usable task text)
Clusters identified: 7 (k chosen over 5–7 by silhouette)
Overall merge-success rate: 80.4% (794 merged / 199 closed / 7 open)
Avg commits per PR: 3.82 · Avg files changed: 39.7

All PRs were authored by app/copilot-swe-agent in github/gh-aw. Task prompts were extracted from PR bodies (copilot's "Thanks for asking..." boilerplate stripped; the original task usually survives as a blockquote), vectorized with TF-IDF (1–2 grams) and clustered with K-means.

Key findings

1. One dominant catch-all (46%). Cluster C3 — schema / validation / docs / behavior — holds 453 PRs. This is general-maintenance work and is too broad to optimize as a single category; the TF-IDF separation is genuinely weak here (see caveat).
2. Firewall/network work is the safest and biggest-change bet. C2 (firewall, blocked domains, changeset) has the highest success (94.7%) despite the largest diffs (avg +2,693/−2,208, ~194 files) — these are mostly mechanical version-bump + regenerated-artifact PRs that merge cleanly.
3. "Fix the failing Actions job" is the weakest category. C5 (root-cause / logs / check-run) has the lowest success (68.4%) and fewest commits (2.53) — these reactive [WIP] debugging PRs are frequently abandoned/closed rather than iterated to a merge.
4. AI-budget tuning is a recurring theme. C0 (ai credits / budget / guardrail, ~10%) reflects steady tuning of max-ai-credits / AIC budgets across workflows.

Full cluster analysis & data

Methodology & caveat

• Source: PR bodies → boilerplate stripped, code blocks / URLs / issue-refs removed → TF-IDF (max 800 feat, 1–2 grams, min_df=3) → K-means.
• k selection: silhouette over k∈{5,6,7}: {5: 0.022, 6: 0.025, 7: 0.028}. Separation is weak (silhouette ≈ 0.03) — expected for short, jargon-dense technical prompts that share vocabulary ("workflow", "agent", "schema"). Treat clusters as themes, not hard partitions.
• Iteration proxy: these are GitHub Copilot coding-agent PRs, not gh-aw workflow runs, so per-task turn/duration/cost from gh-aw logs does not apply. commits_count is used as the iteration proxy instead.

Clusters (largest first)

C3 — Schema / validation / docs (catch-all)

• Size: 453 (45.6%) · Success: 80.7% · Commits: 3.62 · Files: 35.9
• Keywords: schema, workflow, behavior, validation, coverage, existing, path, docs
• Examples: #40249 expand network-mapping tables · #40248 extend brittle error-string detection · #40188 seed memory branches via API

C4 — Prompt & safe-output / engine guidance

• Size: 201 (20.2%) · Success: 79.9% · Commits: 3.23 · Files: 14.3 (smallest diffs)
• Keywords: prompt, workflow, safe, agent, output, safe output, guidance, mcp
• Examples: #40106 prevent tool-denial loops · #40105 fix update-pull-request + token efficiency · #40078 handle >300-file PRs

C1 — Workflow jobs (sous-chef, footer, forecast)

• Size: 117 (11.8%) · Success: 80.9% · Commits: 4.44 · Files: 51.9
• Keywords: aic, job, step, sous chef, footer, forecast, detection
• Examples: #40227 render token table · #40125 prevent step timeout · #40062 upload_assets path fix

C0 — AI credits / budget guardrails

• Size: 96 (9.7%) · Success: 80.2% · Commits: 3.85 · Files: 48.3
• Keywords: ai credits, guardrail, budget, credit, failure
• Examples: #39118 raise default max-patch-size · #39061 raise daily AIC budget · #39058 raise max-ai-credits

C6 — Copilot SDK driver / harness

• Size: 70 (7.0%) · Success: 75.7% · Commits: 4.69 · Files: 27.0
• Keywords: sdk, driver, copilot sdk, permission, harness, mode, auth
• Examples: #40116 tool-output previews · #40115 JSONL session events · #40052 Copilot 403 auth guidance

C2 — Firewall / network domains ⭐ highest success

• Size: 38 (3.8%) · Success: 94.7% · Commits: 6.42 · Files: 193.9 (+2,693/−2,208)
• Keywords: domains, firewall, blocked, changeset, smoke, network
• Examples: #40208 bump firewall v0.27.7 · #40157 require RUNNER_TOOL_CACHE · #40132 bump firewall v0.27.6

C5 — Fix failing Actions jobs ⚠️ lowest success

• Size: 19 (1.9%) · Success: 68.4% · Commits: 2.53 (fewest) · Files: 18.2
• Keywords: actions, job, identify, github actions, logs, root cause, check
• Examples: #40239 integration add · #40109 integration add w/ dispatch · #39995 Workflow Features

Success rate by cluster

Cluster	Theme	PRs	%	Success	Avg commits	Avg files
C2	Firewall / network domains	38	3.8%	94.7%	6.42	193.9
C1	Workflow jobs (sous-chef/footer)	117	11.8%	80.9%	4.44	51.9
C3	Schema / validation / docs	453	45.6%	80.7%	3.62	35.9
C0	AI credits / budget guardrails	96	9.7%	80.2%	3.85	48.3
C4	Prompt & safe-output guidance	201	20.2%	79.9%	3.23	14.3
C6	Copilot SDK driver / harness	70	7.0%	75.7%	4.69	27.0
C5	Fix failing Actions jobs	19	1.9%	68.4%	2.53	18.2

Recommendations

1. Treat "fix the failing Actions job" prompts as a distinct, lower-yield class (C5). At 68% success and only ~2.5 commits, these reactive [WIP] PRs often stall. Give them richer up-front context (failing log excerpt + suspected root cause in the prompt, not just a check-run URL) before dispatch, and consider auto-closing stale ones.
2. Keep batching mechanical firewall/version-bump work (C2). It's the highest-yield category (95%) even at large diff size — a good fit for fully-automated dispatch.
3. Sub-segment the C3 catch-all (46%). It's too broad to act on; a follow-up pass restricted to C3 (or topic modeling instead of K-means) would surface the real sub-themes inside general schema/docs maintenance.
4. Interpret clusters as themes, not partitions — silhouette ≈ 0.03 means heavy vocabulary overlap. For sharper grouping, try sentence-embedding vectors instead of TF-IDF in the next iteration.

References: §27822078061

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 190.3 AIC · ⌖ 20.8 AIC · ⊞ 13.8K · ◷

• expires on Jun 20, 2026, 3:40 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #40457.