[prompt-clustering] Copilot Prompt Clustering — 2026-07-03: 12 clusters, 80.1% success, [WIP] merges at 24%

[github-actions[bot]]

Summary

Analysis period: last 30 days (2026‐06‐11 → 2026‐07‐03) · Repo: github/gh-aw
Copilot PRs analyzed: 1,000 (801 merged · 183 closed · 16 open)
Overall success rate: 80.1% · Clusters: 12 (silhouette 0.081)

Success rate is flat vs. the last two weeks (80.1% today vs. 80.8% yesterday, hovering 79–82% all month), but cluster separation is much cleaner this run (silhouette 0.081 vs. 0.029). The dominant story is not average success — it's the 28‐point spread between task types: routine dependency/firewall bumps merge at ~97%, while Copilot/MCP engine‐integration work merges at ~70%.

Key findings

1. [WIP] is the single strongest negative signal. The 33 PRs whose title starts with [WIP] merge at 24.2%, vs. 82.0% for everything else — a 3.4× gap. Today's WIP merge rate dropped sharply (24% vs. ~40% earlier in the week; small n=33, so noisy). Most WIPs concentrate in the "fix failing GitHub Actions job" cluster.
2. Reactive CI/agent‐integration work is the hard tail. The two lowest‐success clusters — copilot / tool / mcp / engine (70%) and docs / actions / failing job (74%, holding 9 of 33 WIPs) — are both reactive, environment‐coupled tasks (engine failures, DinD/ARC topology, CI job breakage). These are where iterations stall.
3. Bounded, mechanical changes are near‐guaranteed. claude / smoke / firewall / domains (dependency bumps + smoke tests, 97%), command / comment / status (CLI/slash‐command, 90%), and sous‐chef refactors (89%) top the table. High body length + low ambiguity ⇒ high merge rate.

Full cluster analysis (12 clusters)

Cluster breakdown

Cluster (top terms)	Size	Success	WIPs	Keywords
step / guidance / report	157 (15.7%)	76%	4	step, guidance, report, skill, agent
copilot / tool / mcp	113 (11.3%)	70%	5	copilot, tool, mcp, engine, failure
path / package / helper	104 (10.4%)	84%	2	path, package, helper, shared, behavior
schema / field / json	95 (9.5%)	75%	1	schema, field, json, runtime, config
safe / safe‐outputs	86 (8.6%)	78%	4	safe, safe outputs, output, agent, issue
command / comment / status	82 (8.2%)	90%	2	command, comment, status, cli, label
sous / chef	81 (8.1%)	89%	0	sous, chef, aic, run, engine
workflows / branch / frontmatter	73 (7.3%)	78%	5	workflows, branch, frontmatter, agentic
claude / smoke / firewall	61 (6.1%)	97%	0	claude, smoke, domains, firewall, blocked
docs / actions / failing job	61 (6.1%)	74%	9	docs, actions, github actions, job, failing
guard / access / error	46 (4.6%)	85%	1	guard, access, error, flagged, cases
model / sub‐agent	41 (4.1%)	76%	0	model, sub, agent, inline, codex

Representative PR per cluster

Theme	Example	Title
step/report	#42222	Populate audit job steps in structured audit output
copilot/mcp	#40116	Show tool output previews in Copilot CLI conversation rendering
path/package	#41036	Harden go‐gh remote fetch callsites with escaped paths/refs
schema/json	#41681	Add schema validation tests for sandbox.agent.sudo
safe‐outputs	#40575	Clarify direct safe‐output usage in Daily Compiler Quality Check
command/status	#41709	Reuse centralized status comments via the standard update path
sous‐chef	#41824	Add model‐policy frontmatter + import unioning + env overrides
workflows	#40477	Roll out gh-aw-detection to 20% of repository workflows
claude/firewall	#41129	Bump default gh-aw-firewall v0.27.9 and gh-aw-mcpg v0.3.29
docs/failing	#42379	[WIP] Fix failing GitHub Actions job for CLI Completion
guard/access	#42223	eslint-factory: flag computed string-literal access
model/sub	#41184	Pin Daily Sub-Agent Model Resolution Audit to a valid Codex model

Trend (cached history)

Date	PRs	Success	k	Silhouette	WIP merge
06‐21	~1000	79.2%	–	–	~37%
06‐25	~1000	80.1%	–	–	~41%
07‐01	1000	82.4%	–	–	~38%
07‐02	1000	80.8%	11	0.029	~42%
07‐03	1000	80.1%	12	0.081	24%

Method notes / limitations

• Corpus = cleaned PR title + body (code fences, inline code, URLs, [WIP] markers stripped). No START COPILOT markers exist in these bodies — the body is the agent's task/solution description.
• TF‐IDF (uni+bigrams, min_df=5, 400 features) + cosine K‐means; k chosen by silhouette over k∈[6,12] with a min‐cluster‐size guard. scikit‐learn/pandas are unavailable on this PyPy runtime, so both were reimplemented in NumPy.
• Turn/duration/cost enrichment is absent this run: the cached full‐PR data (pr-full-data/) covers PR #30xxx from May 24, which does not overlap the current #40xxx–43xxx window, and no aw_info.json workflow logs were fetched. Success = merged/total is the only outcome signal.

Recommendations

1. Treat [WIP] as a triage flag, not a status. A 24% merge rate means most [WIP] PRs are stuck reactive fixes ("fix failing Actions job") that churn without landing. Consider auto‐nudging or re‐scoping [WIP] PRs older than N days rather than leaving them open.
2. Add environment context up front for engine/MCP tasks. The lowest‐success clusters (copilot/mcp/engine at 70%, failing‐CI at 74%) fail on environment coupling — DinD/ARC topology, HOME paths, transport regressions. Prompts for these should pin the exact runtime topology and reproduction steps rather than "fix the failing job."
3. Keep routing bounded mechanical work to the agent. Dependency bumps, smoke/firewall config, and slash‐command/status‐comment changes merge at 89–97%. These are reliable, high‐throughput wins worth continuing to automate.

References: §28654835856

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 177.7 AIC · ⌖ 18.3 AIC · ⊞ 13.2K · ◷

• expires on Jul 4, 2026, 2:59 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #43367.