[prompt-clustering] Prompt Clustering Analysis - 2026-08-19

[github-actions[bot]]

Summary

Analysis Period: 2026-08-02 to 2026-08-19 (most recent 1000 copilot-authored PRs, search-capped)
Total Tasks Analyzed: 1000
Clusters Identified: 4
Overall Merge Rate: 78.1% (776 merged / 218 closed / 6 open)

The clearest signal in this run isn't really an NLP cluster — it's a PR-body attribution tag. 650 of 1000 PR bodies carry no Generated by [...] footer ("hand-authored" copilot tasks, 77.3% merge), while 305 are stamped Generated by 👨🍳 PR Sous Chef — an internal auto-fix workflow in this repo — at a notably higher 90.1% merge rate. A further ~30 PRs come from small daily reporting/audit workflows (Container Image Security Scan, Squad Plan, GEO Optimizer, Deep Report, Serena Go Expert) that merge at 0% across the board, though each has too few samples (3–11) to count as a formal outlier cluster.

The one cluster that does clear the outlier bar (≥15 PRs, ≥10-point merge-rate gap) is container image vulnerability remediation — 72 PRs at 55.6% merged vs. the 78.1% baseline, a 22.5-point gap. Root cause below: over half of its closed PRs never produced a single file change because the fix is blocked on an upstream vendor, not on the agent.

Full Analysis Report

General Insights

• Most common task type: general fixes / CI-reliability work (cluster 3, 571 tasks, 57% of volume) — shellcheck fixes, flaky-test repairs, permission-scoping regressions, template extraction.
• Highest success rate: Go pkg/CLI feature work (cluster 1) at 80.3%, closely followed by general fixes (79.9%) and docs/agentic-workflow authoring (79.3%) — these three are statistically indistinguishable from the 78.1% baseline.
• Most iteration-heavy: docs/agentic-workflow-authoring tasks average the most files changed per PR (34.4) and Go pkg/CLI tasks draw the most review/comment activity (7.6 comments, 3.1 reviews avg) — larger, more collaborative changes.
• Outlier: 1 formal outlier cluster (container image vulnerability remediation, n=72, gap −22.5 pts). Additionally, 6 small named-workflow sources (n=3–11 each, ~30 PRs total) show a 0% merge rate — flagged as a secondary, lower-confidence signal since none individually clears the 15-PR threshold.

Cluster Analysis

Cluster 1: Go package/CLI feature & linter work

• Size: 154 tasks (15.4% of total)
• Merge Rate: 80.3%
• Avg comments / reviews / files changed: 7.6 / 3.1 / 16.3
• Top Keywords: pkg, cli, pkg cli, package, added, test, string, changes
• Characteristics: New Go analyzers/linters, ADR-driven refactors, CLI subcommand additions — typically well-scoped, self-contained package changes.
• Example PRs: feat(linters): add uncheckedflushreturn analyzer + fix existing violations #49862 (uncheckedflushreturn analyzer), Fix on.needs being emitted into compiled workflow on: section #49864 (on.needs compiler fix), fix: support ready_for_review activity type in pull_request_target trigger #49865 (ready_for_review trigger support), fix: migrate all 10 filepath.Walk err-shadow sites (ADR-49633) and enforce in CI #49907 (filepath.Walk err-shadow migration, ADR-49633)

---

Cluster 2: Documentation & agentic-workflow/skill authoring

• Size: 203 tasks (20.3% of total)
• Merge Rate: 79.3%
• Avg comments / reviews / files changed: 5.6 / 2.3 / 34.4
• Top Keywords: md, engine, sub, docs, shared, agentic, skill
• Characteristics: Documentation updates, new/updated skill files, engine-abstraction refactors — the largest average file footprint of any cluster (many touch shared docs + multiple workflow definitions at once).
• Example PRs: Fix SC2015 shellcheck errors in generated "Collect usage artifact files" step #49845 (shellcheck SC2015 fixes in generated steps), Fix Codex fetch integration tests and restore timeout-minutes propagation fallback #49858 (Codex fetch integration test fixes), Add skill invocation visibility for APM-restored skills #49866 (skill invocation visibility for APM-restored skills)

---

Cluster 3: General fixes & CI/test reliability

• Size: 571 tasks (57.1% of total — the dominant bucket)
• Merge Rate: 79.9%
• Avg comments / reviews / files changed: 5.3 / 2.5 / 22.6
• Top Keywords: add, coverage, run, safe, job, test, error, fix
• Characteristics: The catch-all "bug fix / CI failure / add test coverage" bucket. Largest cluster by far — most copilot tasks in this repo are incremental fixes to workflows, safe-outputs handling, and test coverage gaps.
• Example PRs: Fix read-tool workspace permission scoping regression (daily workflows killed by denial threshold) #49840 (read-tool workspace permission regression), [WIP] Fix failing GitHub Actions job for unit tests #49844 (failing Actions job fix), Extract inline output templates from daily-cache-strategy-analyzer into shared file #49861 (template extraction refactor)

---

Cluster 4 (outlier): Container image vulnerability remediation

• Size: 72 tasks (7.2% of total)
• Merge Rate: 55.6% — 22.5 points below the 78.1% baseline
• Avg comments / reviews / files changed: 2.9 / 1.1 / 28.4
• Top Keywords: image, container, scan, findings, high, policy, proxy, json
• Characteristics: Tasks spawned from daily container-image vulnerability/license scans (CVE remediation, Grant license-policy updates, digest pinning). Lowest avg comments/reviews of any cluster — these tasks get the least human engagement, consistent with many silently stalling.
• Example PRs: [WIP] Update container image mcp/arxiv-mcp-server for vulnerabilities #49838 ([WIP] arxiv-mcp-server image), [WIP] Update grafana/mcp-grafana image to address vulnerabilities #49839 ([WIP] grafana/mcp-grafana image), [WIP] Update mcp/context7 container image to resolve vulnerabilities #49841 ([WIP] mcp/context7 image), [WIP] Fix container vulnerabilities for semgrep/semgrep #49843 ([WIP] semgrep/semgrep vulnerabilities)

Success Rate by Cluster

Cluster	Tasks	Merge Rate	Avg Comments	Avg Reviews	Avg Files Changed	Top Keywords
3 — General fixes & CI/test reliability	571	79.9%	5.3	2.5	22.6	add, coverage, test, fix, error
2 — Docs & agentic-workflow/skill authoring	203	79.3%	5.6	2.3	34.4	md, engine, docs, agentic, skill
1 — Go pkg/CLI feature work	154	80.3%	7.6	3.1	16.3	pkg, cli, package, test, string
4 — Container image vuln remediation ⚠️	72	55.6%	2.9	1.1	28.4	image, container, scan, findings, policy

Source-Workflow Breakdown (from PR-body attribution footer)

PRs grouped by the internal agentic workflow that generated them

Source	PRs	Merged	Closed	Open	Merge Rate
(unattributed / hand-triggered)	650	500	147	3	77.3%
PR Sous Chef	305	272	30	3	90.1%
Daily Container Image Security Scan	11	0	11	0	0%
Squad Plan	8	0	8	0	0%
GEO Optimizer Daily Audit	6	0	6	0	0%
Deep Report	3	0	3	0	0%
Sergo - Serena Go Expert	3	0	3	0	0%
Daily Go Test Parallelizer	3	3	0	0	100%
(10 other one-off sources, 1 PR each)	10	2	8	0	mixed

PR Sous Chef — an auto-fix/iteration bot — is both the second-largest source (30.5% of all PRs) and the highest performing at scale (90.1%), well above baseline. The five "Daily ... Audit/Report/Scan" sources are unanimous 0% at n=3–11 each; too small individually to call a statistical outlier, but consistent enough across five independent sources to flag as a pattern worth a maintainer look (see recommendations).

Sample Data Table

Representative PRs per cluster (sample of 24; full dataset is 1000 rows and omitted for size — available in the workflow's cached PR data)

PR #	Title	Cluster	Outcome
49862	feat(linters): add uncheckedflushreturn analyzer + fix existing violations	1	Merged
49864	Fix on.needs being emitted into compiled workflow on: section	1	Merged
49865	fix: support ready_for_review activity type in pull_request_target trigger	1	Merged
49907	fix: migrate all 10 filepath.Walk err-shadow sites (ADR-49633)	1	Merged
49845	Fix SC2015 shellcheck errors in generated "Collect usage artifact files" step	2	Merged
49858	Fix Codex fetch integration tests and restore timeout-minutes fallback	2	Merged
49866	Add skill invocation visibility for APM-restored skills	2	Merged
49840	Fix read-tool workspace permission scoping regression	3	Merged
49844	[WIP] Fix failing GitHub Actions job for unit tests	3	Merged
49861	Extract inline output templates from daily-cache-strategy-analyzer	3	Merged
49842	[WIP] Update container image to address vulnerabilities	3	Closed
49838	[WIP] Update container image mcp/arxiv-mcp-server for vulnerabilities	4	Closed
49839	[WIP] Update grafana/mcp-grafana image to address vulnerabilities	4	Closed
49841	[WIP] Update mcp/context7 container image to resolve vulnerabilities	4	Closed
49843	[WIP] Fix container vulnerabilities for semgrep/semgrep	4	Closed
51769	Replace vulnerable Node MCP runtime image	4	Closed
51770	Bump gh-aw-mcpg image to v0.4.9 (duplicate of #51754)	4	Closed
53674	Add task-preflight skill to stop zero-diff retry PRs on container-vuln tasks	4	Merged
52935	Reject digest-qualified container pin references	4	Closed
51981	[WIP] Update container image to resolve vulnerabilities	4	Closed

20 of 1000 rows shown — no cluster or generator was silently dropped from the aggregate stats above, only the row-level table is truncated for report size.

Key Findings

1. PR body attribution is a stronger success predictor than task text. PR Sous Chef-generated PRs merge at 90.1% vs. 77.3% for unattributed/hand-authored ones — a 12.8-point gap in the opposite direction of the outlier below, suggesting the internal auto-fix/retry loop this bot runs materially improves mergeability.
2. Container-vulnerability remediation is the one real outlier cluster (72 PRs, 55.6% vs. 78.1% baseline), and it's also the lowest-engagement cluster (fewest avg comments/reviews) — consistent with tasks quietly stalling rather than being actively debated and rejected.
3. A cluster of five small "Daily ... Audit/Report" workflows merge at literal 0% (n=3–11 each, ~30 PRs combined). Individually too small to be a formal outlier, but the unanimous failure across five independent sources is worth a maintainer look — these may be producing PRs nobody intends to merge (e.g., pure reporting workflows that shouldn't open PRs at all).
4. Interestingly, PR Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674 — "Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks" — shows the team has already identified and begun addressing the exact failure mode surfaced in finding Add workflow: githubnext/agentics/weekly-research #2 below.

Lowest-Merge Cluster Root Cause

• Cluster: 4 — Container image vulnerability remediation
• Merge Rate Gap: 55.6% vs. 78.1% overall (gap: 22.5 points)
• Primary Blocker: Scope/complexity mismatch — the task has no agent-actionable fix
• Confidence: High
• Evidence PRs: [WIP] Update container image mcp/arxiv-mcp-server for vulnerabilities #49838, [WIP] Update grafana/mcp-grafana image to address vulnerabilities #49839, [WIP] Update mcp/context7 container image to resolve vulnerabilities #49841, [WIP] Fix vulnerabilities in container image ghcr.io/github/github-mcp-server:v1.8.0 #51332, [WIP] Update container image to resolve vulnerabilities #52630, Document upstream blocker for cli-proxy vulnerability remediation #52705, Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674

Of the 32 closed PRs in this cluster, 18 (56%) never produced a single file change — they were opened as [WIP], the agent could not identify an actionable code change, and the PR was eventually closed stale. This pattern is distinct from every other cluster, where closed PRs almost always carry a real (if rejected) diff. Explicit titles in this set — "Document blocker for gh-aw-firewall/agent scan remediation", "Track upstream remediation for api-proxy 0.28.1 findings", "Verify CLI proxy container remediation availability" — confirm the root cause directly: these tasks ask the agent to patch a CVE in a third-party base image, but the fix requires the upstream vendor to publish a patched image first. The agent has no code change to make and no way to know that in advance, so it opens a WIP PR, stalls, and the PR ages out.

Reinforcing this: PR #53674, "Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks," merged during this same period — the team has already started building the mitigation this finding recommends.

Blocker Category	Count	Share	Notes
Scope mismatch — zero-diff, blocked on upstream vendor	18	56%	[WIP] PRs, 0 files changed, no available fix path
Real diff but still closed (superseded/consolidated)	9	28%	e.g. Grant license-policy PRs, duplicate image-bump PRs
Review friction (digest-pinning / security policy pushback)	3	9%	e.g. "Reject digest-qualified container pin references"
Explicit duplicate	2	6%	Two independent PRs bumping the same image to the same version

Recommendations

Based on clustering analysis:

1. Add a preflight/feasibility check to the container-vulnerability-scan workflow. Before opening a PR, have the agent check whether a patched upstream image actually exists; if not, post a single triage comment on the originating issue ("blocked on vendor, will retry when available") instead of opening a [WIP] PR that stalls. This directly extends the work already started in Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674.
2. Investigate the five 0%-merge "Daily ... Audit/Report" sources (Container Image Security Scan, Squad Plan, GEO Optimizer, Deep Report, Serena Go Expert). If these are meant to be reporting-only workflows, consider having them file issues or discussions instead of PRs, so their output doesn't show up as failed merges.
3. PR Sous Chef's 90.1% merge rate is worth studying as a template — whatever retry/refinement loop it runs before finalizing a PR is measurably outperforming both the hand-authored baseline and every content cluster. Consider extending its approach (or invoking it) for the container-vulnerability cluster specifically.
4. For the outlier cluster, route container-CVE tasks through a "waiting on upstream" label/state distinct from a normal open PR, so stalled zero-diff PRs don't compete with genuinely in-progress work in review queues.

Limitations

• No gh-aw CLI or cached workflow-run logs (aw_info.json) were available in this environment, so turn counts, duration, and cost per PR could not be joined into this analysis — clustering and outcome data are based on PR content and metadata only.
• Clustering used k=4 (elbow method on TF-IDF over cleaned titles+bodies, 150 features, 1–2 grams); cluster boundaries are approximate and some PRs (e.g. a handful of container-themed PRs landing in cluster 3) sit near the boundary.

---

Generated by Prompt Clustering Analysis (Run: §32239808421)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 130.8 AIC · ⌖ 18.3 AIC · ⊞ 14.9K · ◷

• expires on Aug 20, 2026, 2:23 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #54207.