[prompt-clustering] Prompt Clustering Analysis - 2026-08-25

[github-actions[bot]]

Summary

Analysis Period: 2026-08-07 to 2026-08-25 (last ~18 days of activity, capped at 1,000 most recent copilot-authored PRs)
Total Tasks Analyzed: 1,000
Clusters Identified: 8 (k selected via silhouette score, range 4–8)
Overall Merge Rate: 81.5% (815 merged / 179 closed / 6 open)

Data limitation: No gh-aw logs / aw_info.json workflow-run data was available in this run (the gh-aw CLI was not present in the sandbox), so turn counts, duration, and token cost could not be joined to PRs. This analysis is based on PR body/comment/review/commit metadata only.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 5 — Feature Additions & Safe-Output/Coverage Engineering (214 tasks, 21%)
• Highest Success Rate: Cluster 4 — Internal Refactors & PR Sous Chef / Engine Internals (91.0% merged)
• Most Complex Tasks: Cluster 2 — Container Image & MCP Vulnerability Remediation (avg. 56.4 files changed per PR — by far the largest diffs, driven by lock-file regeneration across many workflows)
• Outliers: 1 outlier cluster flagged (size ≥15 PRs and merge rate ≥10 points below the 81.5% overall baseline)

Cluster Analysis

Cluster 1: Feature Additions & Safe-Output/Coverage Engineering

• Size: 214 tasks (21.4% of total)
• Merge Rate: 83.6%
• Top Keywords: add, coverage, workflow, safe, output, fixes
• Characteristics: New workflow features, safe-output handler additions, coverage/test scaffolding.
• Example PRs: #ensure-awf-install, "Add formal PR outcome evaluator coverage", "Increase post-result watchdog default to 120s"

---

Cluster 2: Workflow Config, Scheduling & Sandbox Rollout

• Size: 182 tasks (18.2%)
• Merge Rate: 76.4%
• Top Keywords: workflow, daily, workflows, md, github, lock
• Characteristics: Scheduled/daily workflow prompts, sandbox rollout percentage tuning (docker-sbx, gVisor), lock-file regeneration.
• Example PRs: "Default GitHub-hosted Ubuntu runners to Docker Sandbox", "Revert sbx to 10% of agentic workflows, set gvisor to 10%"

---

Cluster 3: Lint Rules, Error-Code Standardization & Tests

• Size: 139 tasks (13.9%)
• Merge Rate: 88.5%
• Top Keywords: error, rule, cjs, added, test, message
• Characteristics: Custom lint rule additions (USE-*, error-code audits), regression tests.
• Example PRs: "Standardize USE-001 error codes...", "Detect thrown custom Error subclasses in error-code lint rule"

---

Cluster 4: Internal Refactors & PR Sous Chef / Engine Internals

• Size: 133 tasks (13.3%)
• Merge Rate: 91.0% (highest of all clusters)
• Top Keywords: sous chef, refactor, run, aic
• Characteristics: Small, well-scoped refactors to internal tooling (safe-outputs assembly, secret validation wrappers) and the "PR Sous Chef" auto-merge bot. High merge rate suggests these are narrowly-scoped, low-risk changes.
• Example PRs: "Refactor engine secret validation wrappers", "Refactor safe-outputs handler assembly"

---

Cluster 5: CLI/pkg Internals & Parser/Workflow Specs

• Size: 101 tasks (10.1%)
• Merge Rate: 79.2%
• Top Keywords: pkg, pkg cli, cli, pkg workflow, file
• Characteristics: Go package-level fixes in pkg/cli, pkg/workflow, parser/spec documentation.
• Example PRs: "Handle discarded filepath.Glob errors in CLI", "Bump pinned Claude Code CLI to 2.1.226..."

---

Cluster 6: Agent Engine Integration (Copilot/Claude/Codex)

• Size: 95 tasks (9.5%)
• Merge Rate: 75.8%
• Top Keywords: copilot, engine, model, workflow, claude, smoke
• Characteristics: Engine-definition work for Copilot/Claude/Codex/Pydantic-AI/Lightpanda, smoke-test wiring.
• Example PRs: "feat: add Lightpanda browser agent engine definition and smoke test", "fix(pydantic): install the pydantic-ai CLI..."

---

Cluster 7: Container Image & MCP Vulnerability Remediation ⚠️ OUTLIER

• Size: 82 tasks (8.2%)
• Merge Rate: 63.4% (18.1 points below the 81.5% overall baseline)
• Top Keywords: image, mcp, container, aw, gh, cve
• Characteristics: CVE/container-scan-driven remediation — bumping or replacing vulnerable MCP/container images (github-mcp-server, gh-aw-mcpg, api-proxy, gh-aw-firewall). Largest average diffs (56.4 files changed) due to workflow lock-file regeneration.
• Example PRs: Bump Claude Code, OpenAI Codex, and Pi CLI pinned versions #51194, [WIP] Fix vulnerabilities in container image ghcr.io/github/github-mcp-server:v1.8.0 #51332, [WIP] Fix vulnerabilities in api-proxy container image #51333, Fix recurring gh-aw-firewall digest-pin loss on DefaultFirewallVersion bumps #51423, Grant agentic engines read/write access to /tmp/gh-aw in AWF sandbox #51608

---

Cluster 8: CI Job Failures & Threat-Detection Fixes

• Size: 54 tasks (5.4%)
• Merge Rate: 90.7%
• Top Keywords: job, actions, fix, detection, logs, check
• Characteristics: Fixes to failing Actions jobs, threat-detection/inline-detection logic.
• Example PRs: "Fix CGO/CJS workflow cache key collision causing widespread job failures", "Use inline threat detection for Code Scanning Fixer"

Success Rate by Cluster

Cluster	Theme	Tasks	Merge Rate	Avg Comments	Avg Files Changed	Top Keywords
4	Internal Refactors / PR Sous Chef	133	91.0%	—	—	sous chef, refactor
8	CI Job Failures / Threat Detection	54	90.7%	—	—	job, actions, detection
3	Lint Rules / Error Codes	139	88.5%	—	—	error, rule, cjs
1	Feature Additions / Coverage	214	83.6%	—	—	add, coverage, safe output
5	CLI/pkg Internals	101	79.2%	—	—	pkg, cli, workflow
2	Workflow Config / Sandbox Rollout	182	76.4%	—	—	workflow, daily, lock
6	Agent Engine Integration	95	75.8%	—	—	copilot, engine, claude
7	Container/MCP Vulnerability Remediation	82	63.4%	4.5	56.4	image, mcp, container, cve

Full Data Table (sample of 30 representative PRs across clusters)

PR #	Title	Cluster	Outcome
51194	Bump Claude Code, OpenAI Codex, and Pi CLI pinned versions	7	Merged
51332	[WIP] Fix vulnerabilities in container image github-mcp-server:v1.8.0	7	Closed
51333	[WIP] Fix vulnerabilities in api-proxy container image	7	Closed
51423	Fix recurring gh-aw-firewall digest-pin loss on DefaultFirewallVersion bumps	7	Merged
51608	Grant agentic engines read/write access to /tmp/gh-aw in AWF sandbox	7	Merged
51718	Replace vulnerable ast-grep MCP container	7	Merged
51754	Bump MCP Gateway (gh-aw-mcpg) from v0.4.8 to v0.4.9	7	Merged
51769	Replace vulnerable Node MCP runtime image	7	Closed
51770	Bump gh-aw-mcpg image to v0.4.9 (duplicate of #51754)	7	Closed
51796	Bump gh-aw-mcpg to v0.4.9 (duplicate)	7	Closed
51828	Bump gh-aw-mcpg to v0.4.9 and github-mcp-server to v1.9.0	7	Merged
51980	Mitigate github-mcp-server container findings by repinning image	7	Closed
51981	Document blocker for gh-aw-firewall/agent scan remediation	7	Closed
52049	[WIP] Fix vulnerabilities in container image gh-aw-mcpg	7	Closed
52629	Retry DIFC proxy image startup	7	Closed
52630	[WIP] Update container image to resolve vulnerabilities	7	Closed
52671	No gh-aw change: Anthropic proxy fix requires gh-aw-firewall	7	Closed
52705	Document upstream blocker for cli-proxy vulnerability remediation	7	Closed
53080	Track upstream remediation for api-proxy 0.28.1 findings	7	Closed
53119	[WIP] Update container scan findings for gh-aw-firewall/agent:0.28.1	7	Closed
53131	Verify CLI proxy container remediation availability	7	Closed
53137	[WIP] Update container findings for gh-aw-firewall/squid:0.28.1	7	Closed
55035	Investigate cap_net_raw sandbox hardening — fix belongs in gh-aw-firewall	7	Closed
55711	Fix AI Moderator Codex crash: seed CODEX_HOME into allow-list	7	Open
51942	Remediate targeted custom-linter findings	5	Merged
52076	docs: document --runtime flag for gh aw logs and gh aw audit	7	Merged
52412	Return partial MCP logs results before gateway timeout	5	Merged
52566	Fix coverage.findProfile path matching for perf-linter coverage gating	7	Merged
52580	Track container CVE burn-down Critical-first with a consolidated parent issue	7	Merged
53201	Harden AWF downloads and digest-pinned MCP startup	7	Closed

(Full 1,000-row dataset processed programmatically; only a representative sample is shown here for readability — see combined-data.json for the complete set.)

Key Findings

1. Vulnerability remediation is the highest-friction task category. Cluster 7 (container/MCP CVE fixes) merges at 63.4% vs. an 81.5% baseline — an 18-point gap — despite being a mid-size cluster (82 PRs, 8.2% of volume). It also produces the largest diffs (avg. 56.4 files changed, ~3-5x other clusters), because a single image/version bump cascades through every compiled .lock.yml workflow file.
2. Narrowly-scoped internal refactors succeed most often. Cluster 4 ("PR Sous Chef" / internal refactors) and Cluster 8 (CI job fixes) both merge above 90%, consistent with small, mechanical, single-purpose diffs.
3. A recurring "[WIP] Fix vulnerabilities in container image X" title pattern never converges. 9 of the 29 non-merged PRs in Cluster 7 are draft investigation PRs with this exact template that were closed without a completed fix — the agent opens an investigation but the remediation path (version bump vs. no-op vs. upstream fix) is never finalized in the same PR.

Lowest-Merge Cluster Root Cause

• Cluster: 7 — Container Image & MCP Vulnerability Remediation
• Merge Rate Gap: 63.4% vs. 81.5% overall (gap: 18.1 points)
• Primary Blocker: Scope/complexity mismatch — specifically, tasks where the vulnerability's actual fix lives outside the gh-aw repository (e.g., in gh-aw-firewall, cli-proxy, or an upstream base image), so the agent can only produce documentation/investigation output, not a mergeable code fix.
• Confidence: Medium — based on a manual read of comments/reviews on all 29 non-merged PRs in the cluster plus title-pattern analysis; no CI log data was available to independently confirm the "other" bucket.
• Evidence PRs: Document blocker for gh-aw-firewall/agent scan remediation #51981, No gh-aw change: Anthropic proxy fix requires gh-aw-firewall #52671, Document upstream blocker for cli-proxy vulnerability remediation #52705, Track upstream remediation for api-proxy 0.28.1 findings #53080, Verify CLI proxy container remediation availability #53131, Investigate cap_net_raw sandbox hardening request — fix belongs in gh-aw-firewall, not gh-aw #55035 (explicit "upstream blocker / no gh-aw change possible" PRs); [WIP] Fix vulnerabilities in container image ghcr.io/github/github-mcp-server:v1.8.0 #51332, [WIP] Fix vulnerabilities in api-proxy container image #51333, [WIP] Fix vulnerabilities in container image gh-aw-mcpg #52049, [WIP] Update container image to resolve vulnerabilities #52630, [WIP] Update container scan findings for ghcr.io/github/gh-aw-firewall/agent:0.28.1 #53119, [WIP] Update container findings for ghcr.io/github/gh-aw-firewall/squid:0.28.1 #53137, [WIP] Reinvestigate conversation-transcript fetch issue #53711, [WIP] Fix related runs link rendering issue in footer #53727, [WIP] Fix Delight workflow CLI-quality section access issue #55158 (abandoned [WIP] investigation drafts, never completed); Bump gh-aw-mcpg image to v0.4.9 #51770, Bump gh-aw-mcpg to v0.4.9 #51796 (explicitly flagged as duplicates of already-merged version bumps by the PR triage bot)

Blocker Category	Count	Share of 29 Closed	Notes
Upstream-owned fix (structural mismatch)	6	21%	Fix requires a change in gh-aw-firewall/cli-proxy/base image, not gh-aw itself
Abandoned [WIP] investigation draft	9	31%	Draft PR opened to investigate a CVE, never reaches a completed remediation
Duplicate/superseded by another merged PR	2 confirmed (+2 likely)	7–14%	Same version bump attempted by multiple parallel agent runs
CI/infra retry issue	1	3%	Registry-latency proxy startup flake, unrelated to the CVE itself
Other/unclassified	11	38%	Mix of scope changes, partial mitigations, and unclear closures — no CI log data to disambiguate further

(1 additional PR, #55711, is still OPEN as of this analysis and was excluded from the "closed" blocker breakdown.)

Recommendations

1. Split CVE-remediation tasks by fixability up front. Before dispatching a "fix container vulnerability X" task to the agent, have a triage step classify whether the fix is even possible within gh-aw (version bump available) vs. requires an upstream/base-image change — route the latter directly to a tracking issue instead of a draft PR, avoiding the 9 abandoned [WIP] PRs seen this period.
2. De-duplicate parallel CVE-scan-triggered tasks. At least 2 pairs of PRs in Cluster 7 targeted the identical version bump; a simple "is there already an open/merged PR touching this image pin" check before dispatch would reduce wasted agent runs.
3. Keep using narrow, single-purpose prompts for refactor/lint work — Clusters 4 and 8 (91.0% and 90.7% merge rates) show that small, mechanical, single-concern tasks are the agent's strongest category; continue favoring this shape where possible.
4. Container/MCP Vulnerability Remediation (Cluster 7) recommendation: Add an explicit "no-op / upstream-blocked" output path to the CVE remediation prompt template so investigation-only outcomes are captured in tracking issues rather than left as unmergeable draft PRs — this alone would address roughly half of this cluster's non-merges (upstream-blocked + abandoned WIP = 52%).

---

Generated by Prompt Clustering Analysis (Run: 32834315462). Workflow-run turn/cost metrics were unavailable in this run — recommend re-running with gh-aw logs access for a fuller picture of iteration counts per cluster.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 70.8 AIC · ⌖ 8.72 AIC · ⊞ 15.5K · ◷

• expires on Aug 26, 2026, 2:20 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #56003.