[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-07-04

[github-actions[bot]]

Summary

Analysis Period: 2026-06-11 → 2026-07-04 (last 30 days) · Repository: github/gh-aw
Tasks Analyzed: 1,000 copilot-swe-agent PRs · Clusters: 7 (K-means / TF-IDF)
Overall Merge Rate: 81.3% (813 merged · 180 closed · 7 open) · Median merge time: ~3h

The copilot coding agent's work in gh-aw is dominated by infrastructure/compiler plumbing (step/command/path generation, ~40% of all tasks) and safe-outputs + schema/engine config work. The safest, highest-merge work is firewall/domain smoke fixes (97%) and error-handling/refactor cleanups (90%); the riskiest is schema/model/engine config (74%) and the small pool of "fix failing Actions job" retry PRs (75%, and heavily [WIP]).

Full Analysis Report

General Insights

• Most common task type — Cluster 4 "compiler / step & path generation": 401 PRs (40%). This is core gh-aw code-generation plumbing.
• Highest merge rate — Cluster 1 "firewall / blocked-domains / smoke": 97% (small, well-scoped, verifiable-by-smoke-test tasks).
• Lowest merge rate — Cluster 2 "schema / model / engine config": 74% (cross-cutting config changes are the hardest to land).
• Struggling signal — 33 PRs still carry a [WIP] title; only 12/33 (36%) merged vs 81% overall. [WIP] in a final title is a strong predictor of a non-merge.
• Task framing — Titles are highly conventional-commit shaped: ~177 fix, ~50 feat, ~28 refactor, ~27 docs. Bug fixes are the single largest intent.

Cluster Analysis

Cluster 4 — Compiler: step / command / path generation

• Size: 401 tasks (40.1%) · Merge rate: 79% (near baseline)
• Keywords: step, command, runs, path, run, repo
• Characteristics: The workhorse cluster — generating/refactoring GitHub Actions steps, run commands, and path handling in the compiler. Broad and heterogeneous, which pulls its silhouette down.
• Examples: [WIP] Fix ARC/DinD topology path redirections in gh-aw compiler #42808 (ARC/DinD topology path redirections), [WIP] Refactor to create shared/guard-otlp.md bundle for workflows #42983 (shared guard-otlp.md bundle), [WIP] Refactor to create shared/serena-otlp.md bundle for workflows #42944 (shared serena-otlp.md bundle)

Cluster 6 — Safe-outputs tooling & prompts

• Size: 179 (17.9%) · Merge rate: 78%
• Keywords: safe, tool, issue, output, safe output, prompt
• Characteristics: Work on the safe-outputs MCP surface (tool wiring, issue/comment outputs, prompt text). Steady but slightly below baseline.
• Examples: Generalize early wildcard-target validation across safe-outputs MCP tools #39300 (wildcard-target validation across safe-outputs MCP), [WIP] Fix safe outputs MCP container path for consumer repos #39695 (safe-outputs MCP container path for consumer repos), [WIP] Expand personas and add systematic cache-memory rotation #42090 (personas + cache-memory rotation)

Cluster 2 — Schema / model / engine config ⚠️ lowest merge

• Size: 153 (15.3%) · Merge rate: 74%
• Keywords: schema, model, engine, sub, field, config
• Characteristics: Cross-cutting configuration — engine selection, model policy, schema fields, sub-workflow config. The most likely to be closed unmerged; these changes touch many call-sites and are easy to get subtly wrong.
• Examples: fix(copilot): support arbitrary HOME directory, not just /home/runner #38725 (arbitrary HOME dir support), fix(claude): inject COPILOT_GITHUB_TOKEN, scope ANTHROPIC_BASE_URL to Anthropic provider, fix provider-aware api-proxy reflect check, convert Copilot env vars to Anthropic vars in harness, fix smoke model for reasoning effort, and mark model-provider: ... #40930 (inject COPILOT_GITHUB_TOKEN / scope ANTHROPIC_BASE_URL), Implement mcp-scripts.dependencies end-to-end with runtime-manager install flow and pinned release-tag validation #39739 (mcp-scripts.dependencies end-to-end)

Cluster 3 — Error handling & refactors ✅ high merge

• Size: 123 (12.3%) · Merge rate: 90%
• Keywords: error, string, analyzer, function, package, cases
• Characteristics: Localized cleanups — error messages, analyzer rules, splitting large functions. Well-bounded and easy to review; lands reliably.
• Examples: Refactor non-workflow large functions into focused helpers #42179 (refactor large functions into helpers), Refactor workflow env access in github_cli/features to injected lookup #43104 (injected env lookup in github_cli/features), spdd batch 4: promote guard-policies spec, add safeguards/norms to manifest and alias specs, create MCP access-control compliance fixtures #43245 (guard-policies spec promotion)

Cluster 5 — Generated "sous-chef" / AIC workflows ✅ high merge

• Size: 83 (8.3%) · Merge rate: 90%
• Keywords: chef, sous, sous chef, aic, generated sous
• Characteristics: Auto-generated agentic workflow bundles (the "sous-chef"/AIC family). Formulaic and regenerable, so they merge cleanly.
• Examples: Add model policy frontmatter + import unioning + env policy overrides #41824 (model policy frontmatter + import unioning), safe_outputs: set git safe.directory via process-scoped env vars before bundle branch pinning #41830 (git safe.directory via process env), [ARC/DinD] Emit chroot.binariesSourcePath and chroot.identity in AWF stdin-config #38911 (ARC/DinD chroot fields)

Cluster 1 — Firewall / blocked-domains / smoke ✅ highest merge

• Size: 45 (4.5%) · Merge rate: 97%
• Keywords: domains, firewall, blocked, aic, claude, smoke claude
• Characteristics: Small, sharply-scoped fixes validated directly by smoke tests. The gold standard for agent success.
• Examples: Fix AIC usage cache always empty in activation job #39130 (AIC usage cache always empty), Run safe-outputs MCP in the gh-aw node container #39100 (safe-outputs MCP in node container), Fix external detection job setup for codex and pi workflows #40954 (external detection for codex/pi)

Cluster 0 — "Fix failing Actions job" retries ⚠️ struggling

• Size: 16 (1.6%) · Merge rate: 75%
• Keywords: failing actions, actions job, job, failing, asking
• Characteristics: Reactive PRs opened to fix a failing CI job; nearly all titled [WIP]. Small but disproportionately unfinished — a distinct failure mode rather than a task theme.
• Examples: [WIP] Fix failing GitHub Actions job for integration add #40239, [WIP] Fix failing GitHub Actions job for integration add with dispatch-workflow #40109, [WIP] Fix failing GitHub Actions job for integration unauthenticated add #43304 (all "[WIP] Fix failing GitHub Actions job for integration ...")

Merge Rate by Cluster

Cluster	Theme	Tasks	Merge Rate	Top Keywords
1	Firewall / smoke	45	97%	domains, firewall, blocked
3	Error / refactor	123	90%	error, analyzer, function
5	Sous-chef / AIC gen	83	90%	chef, sous, aic
4	Compiler step/path	401	79%	step, command, path
6	Safe-outputs tooling	179	78%	safe, output, prompt
0	Fix failing Actions	16	75%	failing, actions, job
2	Schema / model / engine	153	74%	schema, model, engine

Outcome by Triage Label (orthogonal signal)

Label	Tasks	Merge Rate
pr-type:feature	27	85%
pr-type:bug	55	82%
pr-type:test	6	83%
pr-type:chore	27	78%
pr-type:refactor	18	72%
pr-type:docs	7	57%
pr-priority:high	32	84%
pr-priority:low	40	60%
pr-risk:high	21	81%
pr-risk:medium	52	77%

Notably, pr-priority:low tasks merge at only 60% — low-priority work is more often abandoned than blocked on difficulty. Risk level barely predicts merge outcome.

Key Findings

1. Scope predicts success far more than risk. Tightly-scoped, smoke-verifiable clusters (firewall 97%, refactor 90%) beat broad cross-cutting config (schema/engine 74%) by ~23 points, while the pr-risk label spans only 74–81%.
2. [WIP] is a merge-killer. 33 PRs shipped a [WIP] title and merged at 36% vs 81% baseline — the agent's own "not done" marker is the strongest single negative signal in the dataset.
3. Config/schema work is the improvement frontier. The two lowest clusters (schema/model/engine, safe-outputs tooling) are also two of the three largest — 332 PRs, ~33% of volume, sitting at 74–78%.

Recommendations

1. Decompose schema/engine/config tasks. For Cluster 2 work, prompt the agent to change one call-site family per PR and enumerate every affected config path up front, rather than a single sweeping change.
2. Treat [WIP] as a blocker, not a status. Add a check (or reviewer convention) that a [WIP]-titled agent PR is auto-deferred and re-prompted with the specific failing step, since only ~1 in 3 currently recovers.
3. Reuse the smoke-test framing. The 97% firewall cluster succeeds because success is machine-checkable. Where possible, give the agent a concrete verification command in the prompt so more tasks look like Cluster 1.
4. Route low-priority tasks deliberately. pr-priority:low merges at 60%; either invest reviewer attention or explicitly close, rather than leaving them to churn.

Methodology & Limitations

• TF-IDF (1–2 grams, domain stopwords removed) over PR title+body, K-means with k=7 chosen by silhouette across k=3–7 (all scores low, ~0.015–0.023, as expected for short heterogeneous PR text — clusters are thematically coherent but not geometrically crisp).
• Turn-count / cost enrichment was unavailable: the pre-fetched full-PR cache (pr-full-data, from an earlier run) covers PR #30xxx and has zero overlap with this period's Track Pi CLI (@earendil-works/pi-coding-agent) in cli-version-checker #38724–[ca] CLI Version Updates: Claude Code 2.1.201, MCP Gateway v0.3.33 #43358, and gh-aw logs was not on PATH. Merge outcome is used as the success proxy.
• Prompts analyzed are the agent-authored PR descriptions (best available proxy for the underlying task), not the raw originating prompts.
• Cache refreshed: 1,000 analyzed PR numbers written to analyzed-prs.txt for next-run dedup.

References: §28703236869

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 157.2 AIC · ⌖ 19.8 AIC · ⊞ 13.2K · ◷

• expires on Jul 5, 2026, 2:48 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #43529.