[prompt-clustering] Prompt Clustering Analysis - 2026-08-23

[github-actions[bot]]

Summary

Analysis Period: Last ~18 days (2026-08-06 to 2026-08-23)
Total Tasks Analyzed: 989 copilot-agent PRs (11 non-agent-authored PRs excluded from an initial pull of 1,000)
Clusters Identified: 6
Overall Merge Rate: 80.3% (803 merged / 989 total; 8 still open, treated as not-yet-merged)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 2 — workflow/agent infrastructure work (285 tasks, 29% of total)
• Highest Success Rate: Cluster 4 — general engineering hardening/fixes (90.5%) and Cluster 1 — CI auto-fix tasks (90.0%)
• Most Complex Tasks: Cluster 4 also has by far the heaviest review load (avg 13.6 comments, 4.6 reviews per PR) despite its high merge rate — these are substantive, well-scrutinized changes rather than easy ones
• Outliers: 2 clusters flagged (size ≥ 15 PRs, merge rate ≥ 10 points below the 80.3% baseline): Cluster 0 (46.6%) and Cluster 5 (62.3%)

Methodology: PR titles + bodies were cleaned (code blocks, HTML comments, URLs, issue refs stripped) and vectorized with TF-IDF (1–2 grams, top 300 terms), then clustered with K-means (k chosen via elbow method, capped 3–7 → k=6). Workflow run logs (turn counts, duration, cost) were unavailable in this run's environment, so the analysis relies on PR metadata, comments, and reviews rather than agent-turn counts.

Cluster Analysis

Cluster 4: General engineering hardening & fixes

• Size: 262 tasks (26.5% of total)
• Success Rate: 90.5%
• Avg Comments / Reviews: 13.6 / 4.6 — highest engagement of any cluster
• Top Keywords: workflow-adjacent hardening terms (noisy TF-IDF due to heavy hyphenation in titles, e.g. "sous-chef", "sub-agent" splitting into fragments); title inspection shows concurrency hardening, path handling, timeout bounding, structured validation errors
• Characteristics: Broad bucket of targeted, well-defined bug fixes and hardening PRs. High comment/review volume but still merges most of the time — reviewers engage heavily but tasks are scoped well enough to land.
• Example PRs: Add safe output for approving fork pull request workflow runs #52541 (Add safe output for approving fork PR workflow runs, 85 comments, merged), Add coverage-aware perf gating for allocation-focused custom linters #51573 (coverage-aware perf gating for linters, merged), Harden assign_to_agent concurrency: isolate handler state and serialize MCP stdin dispatch #52034 (harden assign_to_agent concurrency, merged)

---

Cluster 2: Workflow / agent infrastructure

• Size: 285 tasks (28.8% of total)
• Success Rate: 81.1%
• Avg Comments / Reviews: 3.9 / 2.0
• Top Keywords: workflow, workflows, step, agent, job, prompt, run, daily
• Characteristics: The largest single cluster — day-to-day feature and maintenance work on the gh-aw workflow system itself (allowlisting domains, MCP gateway hardening, lockfile analyzers, PR triage cooldowns). Near-baseline success rate.
• Example PRs: Detect and replace heredocs in generated workflow YAML #53183 (heredoc detection/replacement in generated YAML, merged), Render workflow prompts with JavaScript instead of shell heredocs #51850 (render workflow prompts via JS instead of shell heredocs, merged), Harden MCP gateway startup failure diagnostics #54956 (harden MCP gateway startup diagnostics, merged)

---

Cluster 3: Lint rules, conformance & error handling

• Size: 273 tasks (27.6% of total)
• Success Rate: 83.9%
• Avg Comments / Reviews: 4.0 / 2.3
• Top Keywords: coverage, test, added, rule, error, validation
• Characteristics: Tooling-quality work — custom lint rules, conformance-checker fixes, error-code standardization, test coverage additions. Steady, slightly-above-baseline success.
• Example PRs: Add Working-Set Rebuild Factor observability #54416 (Working-Set Rebuild Factor observability, merged), Harden cloud-hypervisor privileged runtime path and add explicit human-review warning #52757 (harden cloud-hypervisor privileged runtime path, merged), Support runner groups for custom safe jobs #53977 (support runner groups for custom safe jobs, merged)

---

Cluster 0: Copilot/agent-platform meta-tasks (OUTLIER)

• Size: 88 tasks (8.9% of total)
• Success Rate: 46.6% — 33.7 points below the 80.3% overall baseline
• Avg Comments / Reviews: 2.9 / 1.3
• Top Keywords: copilot, issue, comments, model, retry, harness, aw, issue_title
• Characteristics: Self-referential tasks about the agent platform itself — engine migrations (Copilot→Claude/Codex), model-alias resolution, orphan-escalation detection, and a large share of open-ended [WIP] investigation/escalation drafts that were never intended to close in one shot.
• Example PRs: Bump Claude Code CLI to 2.1.227 and Copilot CLI to 1.0.79 #51977 (bump CLI versions, merged), Enable GitHub-hosted inference for Codex #54606 (enable GitHub-hosted Codex inference, still open), Resolve the auto model alias to priced models instead of the copilot/auto passthrough #54364 (resolve auto model alias, closed unmerged), Harden Copilot api-proxy startup: verify listener accept readiness and absorb first-request ECONNREFUSED #52619 (harden Copilot api-proxy startup, closed unmerged)

---

Cluster 5: Dependency / image pinning & compliance (OUTLIER)

• Size: 61 tasks (6.2% of total)
• Success Rate: 62.3% — 18.0 points below baseline
• Avg Comments / Reviews: 3.5 / 1.4
• Avg Files Changed: 54.3 — by far the widest-reaching PRs (version bumps touch many workflow lockfiles at once)
• Top Keywords: image, container, pin, digest, license, mcp, upstream, pinned
• Characteristics: Container/CLI version bumps, digest pinning, and license/CVE-scan remediation. Wide blast radius (many files) plus security/compliance review requirements slow these down.
• Example PRs: Bump CLI, MCP SDK, and scanner image defaults #52855 (bump CLI/MCP SDK/scanner defaults, merged), Add ignore-images to .grant.yaml and exclude the Serena MCP image from license scanning #53369 (exclude Serena MCP image from license scanning, merged), Add repository-level grype ignore policy for no-fix libc6 CVEs #52924 (grype ignore policy for no-fix libc6 CVEs, merged)

---

Cluster 1: CI auto-fix ("Fix failing GitHub Actions job")

• Size: 20 tasks (2.0% of total)
• Success Rate: 90.0%
• Avg Comments / Reviews: 1.2 / 0.9 — lightest review load of any cluster
• Top Keywords: actions, job, github actions, fix, failing
• Characteristics: Small, templated, single-purpose CI-repair PRs (all titled [WIP] Fix failing GitHub Actions job ...). Despite the [WIP] prefix — which correlates with failure everywhere else — this specific auto-triggered pattern merges reliably, likely because the task is narrowly scoped to one failing check.
• Example PRs: [WIP] Fix failing GitHub Actions job Integration: Workflow Infra #54032, [WIP] Fix failing GitHub Actions job for unit tests (D-L) #51151, [WIP] Fix failing GitHub Actions job js-integration-live-api #54650, [WIP] Fix failing GitHub Actions job Integration: Workflow Features #53443 (all merged)

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Comments	Avg Files Changed	Top Keywords
4	General hardening/fixes	262	90.5%	13.6	31.2	sub-agent, concurrency, hardening
1	CI auto-fix	20	90.0%	1.2	1.6	fix failing GitHub Actions job
3	Lint/conformance/errors	273	83.9%	4.0	10.6	coverage, test, rule, validation
2	Workflow/agent infra	285	81.1%	3.9	39.5	workflow, step, agent, prompt
5	Image pinning/compliance	61	62.3%	3.5	54.3	image, container, pin, digest, license
0	Copilot/platform meta-tasks	88	46.6%	2.9	17.0	copilot, issue, model, retry, harness

Full Data Table (sample — top 4 by comment volume per cluster)

24-row sample across all 6 clusters

PR #	Title	Cluster	Outcome	Comments
51977	Bump Claude Code CLI to 2.1.227 and Copilot CLI to 1.0.79	0	Merged	29
54606	Enable GitHub-hosted inference for Codex	0	Open	16
54364	Resolve the auto model alias to priced models	0	Closed	16
52619	Harden Copilot api-proxy startup readiness	0	Closed	14
54032	[WIP] Fix failing GitHub Actions job: Integration Infra	1	Merged	3
51151	[WIP] Fix failing GitHub Actions job: unit tests (D-L)	1	Merged	3
54650	[WIP] Fix failing GitHub Actions job: js-integration-live-api	1	Merged	2
53443	[WIP] Fix failing GitHub Actions job: Workflow Features	1	Merged	2
53183	Detect and replace heredocs in generated workflow YAML	2	Merged	36
51242	fix(pydantic): install pydantic-ai CLI, match pai interface	2	Merged	33
51850	Render workflow prompts with JS instead of shell heredocs	2	Merged	26
54956	Harden MCP gateway startup failure diagnostics	2	Merged	25
54416	Add Working-Set Rebuild Factor observability	3	Merged	20
52757	Harden cloud-hypervisor privileged runtime path	3	Merged	17
50765	Stop generating MCP gateway stderr logs	3	Merged	17
53977	Support runner groups for custom safe jobs	3	Merged	16
52541	Add safe output for approving fork PR workflow runs	4	Merged	85
51573	Add coverage-aware perf gating for allocation-focused linters	4	Merged	63
52034	Harden assign_to_agent concurrency handling	4	Merged	62
52053	Add custom validation hooks for repo and cache memory	4	Merged	55
52855	Bump CLI, MCP SDK, and scanner image defaults	5	Merged	16
53369	Exclude Serena MCP image from license scanning	5	Merged	14
52924	Add repository-level grype ignore policy for libc6 CVEs	5	Merged	13
51752	Mitigate mcp/memory scan findings by removing container	5	Merged	13

(Full 989-row dataset was clustered and analyzed; this table shows the highest-engagement example per cluster to keep the report scannable.)

Key Findings

1. [WIP]-prefixed PRs are the single strongest predictor of non-merge — but only in certain contexts. Across all 989 PRs, titles starting with [WIP] merge at just 31.0% (22/71) vs. the 80.3% baseline. Yet inside Cluster 1 (narrowly-scoped CI auto-fixes), [WIP]-prefixed PRs merge at 90%. The signal isn't the prefix itself — it's whether the underlying task is a bounded, single-check fix (merges fine) or an open-ended investigation/escalation (rarely converges).
2. Wide blast radius correlates with lower merge rate. The two outlier clusters (0 and 5) don't have unusually low comment counts — Cluster 5 in particular touches an average of 54 files per PR (version bumps across many workflow lockfiles), which increases the chance a downstream check or reviewer objects to at least one changed file.
3. Meta-tasks about the agent platform itself are the highest-risk category. Cluster 0 (engine migrations, model-alias resolution, harness/orphan-escalation handling) has the lowest success rate of any cluster and the fewest reviews per PR — these tasks appear to get abandoned or superseded rather than actively rejected.

Lowest-Merge Cluster Root Cause

• Cluster: 0 — Copilot/agent-platform meta-tasks
• Merge Rate Gap: 46.6% vs 80.3% overall (gap: 33.7 points)
• Primary Blocker: Scope/complexity mismatch — specifically, unfinished [WIP] self-escalation/investigation drafts
• Confidence: High
• Evidence PRs: [WIP] Escalate cgo workflow regression to improve success rate #54991, [WIP] Copilot Request #54706, [WIP] Tone down ci-doctor.md status messages #54649, [WIP] Redesign Design Decision Gate to reduce failure rate and cost #54619, [WIP] Review 5 spec files for daily SPDD work #54286 (all [WIP], all closed unmerged)

Of the 47 not-merged PRs in Cluster 0, 32 (68%) are [WIP]-titled investigation/escalation tasks — compare to zero [WIP] PRs among the 41 merged PRs in this same cluster. These are tasks where the agent opened a draft to "escalate," "investigate," or "redesign" a system-level issue (e.g. "[WIP] Escalate cgo workflow regression", "[WIP] Redesign Design Decision Gate to reduce failure rate and cost") without a bounded fix in scope — they stall and get closed rather than converging. The remaining non-merge share splits across review friction (9%, e.g. engine-migration PRs like #54917 that draw detailed reviewer pushback on architecture) and CI failures (9%, e.g. #54364 blocked on failing lint-go).

Blocker Category	Count	Share	Notes
Scope/complexity mismatch (unfinished [WIP] drafts)	33	70%	Self-initiated investigation/escalation tasks with no bounded exit criteria
Other/unknown	6	13%	Superseded, stale, or context-specific closures (e.g. auth restoration abandoned)
Review friction	4	9%	Reviewer pushback on migration/config-swap approach
CI/test failure	4	9%	Failing lint/build checks left unresolved

Recommendations

1. Give escalation/investigation tasks a bounded exit condition. Cluster 0's [WIP] investigation drafts (e.g. "Escalate X regression", "Redesign Y to reduce cost") should specify a concrete, checkable deliverable up front, rather than leaving the PR open-ended — this is the fix most directly targeted at the 33.7-point gap.
2. Batch or stage wide-reaching version-bump PRs (Cluster 5). Since these touch ~54 files on average, splitting bumps by subsystem or pre-validating against known CVE/license exceptions before opening the PR would reduce the chance a single unrelated file blocks the whole change.
3. Keep replicating the Cluster 1 / Cluster 4 pattern: both high-success clusters share one trait — the task, however it's phrased, resolves to a single well-defined check or fix. Prompt engineering for other categories should aim for the same narrow scoping.
4. Outlier Cluster Recommendation: For Cluster 0 specifically, route model/engine-migration and harness-escalation tasks through a lighter-weight triage step (confirm scope and success criteria) before the agent opens a PR, since these currently convert at little more than half the platform's baseline rate.

---

Generated by Prompt Clustering Analysis (Run: 32632036972)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 115.5 AIC · ⌖ 13.5 AIC · ⊞ 15.5K · ◷

• expires on Aug 24, 2026, 2:16 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-24T10:16:18.682Z.

Closed by Workflow