[prompt-clustering] Prompt Clustering Analysis - 2026-08-29

[github-actions[bot]]

Summary

Analysis Period: 2026-08-11 to 2026-08-29 (last ~18 days of PR activity, requested window: 30 days)
Total Tasks Analyzed: 1,140 copilot-authored PRs with usable prompt text (out of 1,140 fetched; 1,000 base search results + additional full-data cache entries)
Clusters Identified: 7
Overall Success Rate: 79.9% merged

Full Analysis Report

Data & Methodology Notes

• Workflow-log enrichment (turn counts, duration, cost) was not available for this run: agenticworkflows logs hits a hard ~60s per-call timeout on this repo's copilot engine, and a single call only covers about 1 hour of run history before its count budget is exhausted (this repo produces a very high volume of workflow runs). Spanning the full 30-day window would require dozens of sequential paginated calls, which isn't practical in one pass. This analysis is therefore based on PR metadata and prompt text only — no turn/duration/cost columns below.
• One clustering artifact worth flagging: Cluster 1 below is driven almost entirely by a repeated bot-footer signature ("PR Sous Chef ... Comment /souschef to run again") that appears in many PR bodies and wasn't fully stripped by the text cleaner. It is not a coherent task theme — treat its keywords as noise, not a real category. Future runs should strip <!-- gh-aw-agentic-workflow: ... --> / bot-footer blocks before vectorizing.

General Insights

• Most Common Task Type: Cluster 5 — safe-outputs / lint-rule / error-code standardization work (386 PRs, 34% of all tasks)
• Highest Success Rate (excluding the noise cluster): Cluster 6 — CI job failure fixes (87.0%)
• Most Complex Tasks: Cluster 4 — engine/harness/model config work, largest average diff size (40 files changed on average) and second-lowest success rate
• Outliers: 1 outlier cluster flagged (size ≥ 15 PRs, merge rate ≥ 10 points below the 79.9% overall baseline)

Cluster Analysis

Cluster 0: Container/MCP image vulnerability remediation

• Size: 60 tasks (5.3% of total)
• Success Rate: 66.7% ⚠️ (outlier — 13.2 points below overall)
• Avg reviews/comments: 1.3 / 4.4 — Avg files changed: 68
• Top Keywords: image, container, mcp, aw, gh aw, version
• Characteristics: Container/dependency CVE burn-down tasks (bumping gh-aw-mcpg, replacing vulnerable MCP images, digest-pin refreshes, license-policy exceptions). Many are auto-generated by a security-scan-driven assignment pipeline rather than a human request.
• Example PRs: Fix recurring gh-aw-firewall digest-pin loss on DefaultFirewallVersion bumps #51423, Replace vulnerable ast-grep MCP container #51718, Patch vulnerable ip-address dependency in gh-aw-node #51719, Replace vulnerable mcp/context7 container with the hosted Context7 HTTP MCP server #51727, Use Alpine-based grafana/mcp-grafana image to clear container scan findings #51728

---

Cluster 1: ⚠️ Artifact cluster (bot-footer noise, not a real theme)

• Size: 63 tasks (5.5% of total)
• Success Rate: 87.3%
• Top Keywords: "sous", "sous chef", "chef", "pr sous", "pr" — these come from a recurring "PR Sous Chef" bot-comment footer embedded in PR bodies, not from actual task content
• Characteristics: Underlying tasks are actually diverse (pydantic CLI packaging, threat-detection logging, sandbox defaults, skill-marker syntax). Ignore the keyword list; do not use this cluster for thematic conclusions.
• Example PRs: fix(pydantic): install the pydantic-ai CLI and match the real pai interface #51242, feat(threat-detection): render detection.log in detection job with group/mask macros #51255, feat: sandbox.agent id:awf is now default — remove redundant explicit declarations #51282, feat(sandbox): add runtime-install field to control sbx/gVisor install step generation #51413, Add explicit end marker syntax for inline skills and sub-agents #51446

---

Cluster 2: Package/CLI test & quality improvements

• Size: 158 tasks (13.9% of total)
• Success Rate: 81.0%
• Avg reviews/comments: 3.3 / 7.8 — Avg files changed: 12
• Top Keywords: pkg, test, cli, package, workflow
• Characteristics: Focused, small-diff fixes — error handling, test coverage, internal package cleanups.
• Example PRs: Handle discarded filepath.Glob errors in CLI #51296, Add mobile interaction regression test for the home page quick-start CTA #51323, Document missing internal dependencies in parser, cli, and workflow specs #51380, Improve test quality for pkg/console/confirm_test.go #51434, Deduplicate commit-ref SHA resolution in action_resolver #51472

---

Cluster 3: Workflow authoring / .md workflow & prompt fixes

• Size: 263 tasks (23.1% of total) — second largest cluster
• Success Rate: 77.6%
• Avg reviews/comments: 1.6 / 4 — Avg files changed: 22
• Top Keywords: workflow, daily, workflows, md, github, issue
• Characteristics: Edits to agentic workflow prompt files (daily/insight workflows, workflow-authoring fixes, issue-driven prompt updates).
• Example PRs: Fix silent pagination truncation in gh aw mcp inspect, dedupe connect logic #51193, Use inline threat detection for Code Scanning Fixer #51277, [WIP] Add OPENROUTER_API_KEY configuration for skill scoring #51287, Remove Dashboard Language specification and review workflow #56674

---

Cluster 4: Engine/harness/model configuration

• Size: 187 tasks (16.4% of total)
• Success Rate: 73.3% (second lowest, though not a flagged outlier — falls just under the 10-point/15-PR threshold combination is met on size but the rate gap is 6.6 points, below the 10-point bar)
• Avg reviews/comments: 2.6 / 7 — Avg files changed: 40 (largest average diff footprint)
• Top Keywords: copilot, engine, claude, harness, model, agent
• Characteristics: Engine-version bumps, sandbox defaults, secret-validation refactors — structurally complex, cross-cutting changes.
• Example PRs: Fix silent conversation-transcript fetch failures in Copilot session data fetcher #51195, Refactor engine secret validation wrappers #51216, Default GitHub-hosted Ubuntu runners to Docker Sandbox #51254, Pi engine: route OpenAI/Codex backend through the Responses API #56692

---

Cluster 5: Safe-outputs / lint rules / error-code standardization

• Size: 386 tasks (33.9% of total) — largest cluster
• Success Rate: 84.7% — highest success rate among large clusters
• Avg reviews/comments: 3.1 / 7 — Avg files changed: 19
• Top Keywords: coverage, safe, error, add, schema, output
• Characteristics: The bulk of day-to-day feature/bug-fix work on safe-outputs handlers, lint rule false-positive fixes, and error-code conventions. High volume, high consistency, high success rate.
• Example PRs: Detect typed map[string]bool set aliases in short declarations #51303, Fix require-error-code-in-thrown-error false positives on Identifier message args #51324, Standardize USE-001 error codes in add_labels and dismiss_pull_request_review #51338, Allow conclusion in on.workflow_run frontmatter schema #56743

---

Cluster 6: CI job failure fixes

• Size: 23 tasks (2.0% of total) — smallest cluster
• Success Rate: 87.0% — highest overall
• Avg reviews/comments: 0.9 / 1 — Avg files changed: 2 (smallest diffs)
• Top Keywords: job, actions, fix, github, description, logs
• Characteristics: Narrowly-scoped "[WIP] Fix failing GitHub Actions job X" tasks — small, mechanical, well-specified. This pattern (narrow CI-failure prompt → tiny diff) correlates with the best outcomes in the dataset.
• Example PRs: [WIP] Fix failing GitHub Actions job 'Integration: CLI Completion & Other' #52927, [WIP] Fix failing GitHub Actions job Integration: CLI Compile & Poutine #53274, [WIP] Fix failing GitHub Actions job Integration: Workflow Infra #56688

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
0	Container/MCP CVE remediation	60	66.7% ⚠️	68	image, container, mcp
1	(bot-footer artifact, ignore)	63	87.3%	59	sous chef (noise)
2	Package/CLI test & quality	158	81.0%	12	pkg, test, cli
3	Workflow .md authoring/fixes	263	77.6%	22	workflow, daily, md
4	Engine/harness/model config	187	73.3%	40	copilot, engine, claude
5	Safe-outputs/lint/error-codes	386	84.7%	19	coverage, safe, error
6	CI job failure fixes	23	87.0%	2	job, actions, fix

Representative Data Sample

Sample of 28 PRs across clusters (full dataset: 1,140 PRs, see cache for complete table)

PR #	Title	Cluster	Outcome	Comments	Reviews
56329	Bump default CLI and MCP versions	0	Merged	2	1
51333	[WIP] Fix vulnerabilities in api-proxy container image	0	Closed	0	0
51423	Fix recurring gh-aw-firewall digest-pin loss	0	Merged	10	4
51718	Replace vulnerable ast-grep MCP container	0	Merged	1	1
56800	Fix Playwright MCP TOML sandbox configuration	1	Open	4	2
51255	Render detection.log in detection job	1	Merged	19	14
51282	sandbox.agent id:awf is now default	1	Closed	34	9
51413	Add runtime-install field for sbx/gVisor	1	Merged	10	9
56691	Avoid CodeQL bad-redirect-check false positives	2	Open	22	7
51323	Add mobile interaction regression test	2	Merged	1	1
51380	Document missing internal dependencies	2	Merged	1	1
51434	Improve test quality for confirm_test.go	2	Merged	7	4
56674	Remove Dashboard Language spec/review workflow	3	Merged	1	1
51193	Fix silent pagination truncation in mcp inspect	3	Merged	14	6
51277	Use inline threat detection for Code Scanning Fixer	3	Merged	9	3
51287	[WIP] Add OPENROUTER_API_KEY config for skill scoring	3	Closed	0	0
56692	Pi engine: route OpenAI/Codex via Responses API	4	Merged	8	5
51195	Fix silent conversation-transcript fetch failures	4	Merged	1	1
51216	Refactor engine secret validation wrappers	4	Merged	11	4
51254	Default GitHub-hosted Ubuntu runners to Docker Sandbox	4	Closed	1	0
56743	Allow conclusion in on.workflow_run frontmatter	5	Merged	17	9
51303	Detect typed map[string]bool set aliases	5	Merged	2	1
51324	Fix require-error-code false positives on Identifier	5	Merged	21	6
51338	Standardize USE-001 error codes	5	Merged	9	5
56688	[WIP] Fix failing Integration: Workflow Infra job	6	Merged	0	2
51571	[WIP] Fix fuzz workflow failure on main	6	Closed	0	0
52927	[WIP] Fix failing Integration: CLI Completion job	6	Merged	1	1
53274	[WIP] Fix failing Integration: CLI Compile job	6	Merged	1	1

Key Findings

1. Container/MCP CVE-remediation tasks are the weakest-performing cluster. At 66.7% merged vs. 79.9% overall, these security-scan-driven assignments fail roughly 1-in-3 times, mostly because the agent gets handed a task with no actual code fix available (see root-cause analysis below).
2. Narrow, single-purpose prompts win. Cluster 6 (fix one named failing CI job) and Cluster 5 (add one error code / fix one lint false positive) both post the smallest diffs and the highest success rates. Cluster 4 (broad engine/harness refactors, largest average diff at 40 files) sits near the bottom.
3. A recurring bot signature contaminated one cluster. Cluster 1's "sous chef" keywords are a PR-bot footer artifact, not a task theme — a reminder to strip generated boilerplate/signatures before vectorizing prompt text in future runs.

Lowest-Merge Cluster Root Cause

• Cluster: 0 — Container/MCP image vulnerability remediation
• Merge Rate Gap: 66.7% vs 79.9% overall (gap: 13.2 points)
• Primary Blocker: Scope/complexity mismatch — the agent is frequently assigned an investigation-only or upstream-blocked CVE tracking task with no code change actually possible in this repo
• Confidence: Medium-high (10 of 20 closed PRs in this cluster have a literal 0-line diff — direct, unambiguous evidence)
• Evidence PRs: [WIP] Fix vulnerabilities in container image ghcr.io/github/github-mcp-server:v1.8.0 #51332, Document blocker for gh-aw-firewall/agent scan remediation #51981, Document upstream blocker for cli-proxy vulnerability remediation #52705, Track upstream remediation for api-proxy 0.28.1 findings #53080, Verify CLI proxy container remediation availability #53131 (all 0-diff "investigate/document blocker" PRs), plus Bump gh-aw-mcpg image to v0.4.9 #51770/Bump gh-aw-mcpg to v0.4.9 #51796 (duplicate concurrent PRs for the same gh-aw-mcpg version bump, one superseding the other)

Of the 20 closed (non-merged) PRs in this cluster:

Blocker Category	Count	Share	Notes
Scope/complexity mismatch	10	50%	Zero-diff "investigate/verify/document blocker" PRs — the underlying CVE has no available upstream fix, so there was no code change for the agent to make (#51332, #51333, #51981, #52049, #52630, #52705, #53080, #53131, #53137, #55035)
Review friction	5	25%	Draft/incomplete fixes sent back for scope clarification or revision, e.g. "clarify scope before proceeding" (#53201), draft not ready (#52629), stub diffs (#52939, #53140), superseded fix approach (#51769)
CI/test failure	2	10%	Blocked on unresolved merge conflicts requiring a follow-up make merge-main (#52935, #54347)
Other/unknown	3	15%	Two duplicate PRs targeting the same dependency bump as an already-merged sibling PR (#51770, #51796, duplicates of #51754/#51828); one large-diff PR with no review/comment activity at all before closing (#51980)

For comparison, the 40 merged PRs in this cluster are overwhelmingly small, targeted, single-image replacements or version bumps with real diffs (e.g. #51718 "Replace vulnerable ast-grep MCP container", #51719 "Patch vulnerable ip-address dependency") — the pattern that succeeds is "one concrete image/dependency swap per PR," not "investigate whether a fix is even possible."

Recommendations

1. Pre-filter upstream-blocked CVE findings before agent assignment. Half of this cluster's failures are 0-diff investigation PRs for CVEs with no available fix. A triage step that checks for an existing upstream fix/advisory before dispatching a coding-agent task would avoid assigning unsolvable tickets (the repo has already started this — see the merged Pre-filter upstream-blocked container/image CVE findings before Copilot assignment #53709 "Pre-filter upstream-blocked container/image CVE findings before Copilot assignment" — expand and enforce it more broadly).
2. Deduplicate concurrent version-bump tasks. Bump gh-aw-mcpg image to v0.4.9 #51770/Bump gh-aw-mcpg to v0.4.9 #51796/Bump MCP Gateway (gh-aw-mcpg) from v0.4.8 to v0.4.9 #51754/Bump gh-aw-mcpg to v0.4.9 and github-mcp-server to v1.9.0 #51828 were four near-simultaneous PRs bumping the same gh-aw-mcpg dependency. A lock/claim mechanism (or checking for an open PR against the same dependency before spawning a new task) would cut duplicate work.
3. Favor narrow, single-target prompts. Clusters 5 and 6 (single lint rule / single CI job) both outperform the broader, multi-file clusters (0 and 4). When scoping new automated tasks, splitting broad refactors into single-target prompts appears to correlate with materially higher merge rates.
4. Outlier cluster mitigation: route CVE-remediation tasks through an upstream-fix-availability check first; only dispatch to the coding agent once a concrete code change (image bump, digest pin, dependency patch) is confirmed possible — this alone would have avoided at least 10 of the 20 closed PRs in this cluster.

---

Generated by Prompt Clustering Analysis (Run: 33246131191). Workflow-log turn/cost enrichment was skipped this run due to per-call timeout constraints on this repo's very high run volume — see Data & Methodology Notes above.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 100.5 AIC · ⌖ 10.3 AIC · ⊞ 15.5K · ◷

• expires on Aug 30, 2026, 2:11 AM UTC-08:00