[prompt-clustering] Prompt Clustering Analysis - 2026-08-26

[github-actions[bot]]

Summary

Analysis Period: Last ~30 days (PRs created 2026-08-08 → 2026-08-26; full-detail cache holds 1,043 copilot-authored PRs)
Total Tasks Analyzed: 1,043
Clusters Identified: 4
Overall Success Rate: 80.9% (844 merged / 1,043 total; 183 closed unmerged, 16 still open)

Note on scope: Workflow-run logs (gh-aw logs) were not available in the pre-fetched cache for this run, so turn counts and cost/duration metrics could not be joined to PRs. All metrics below (comments, reviews, commits, files changed) come from PR interaction data instead.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 0 — test coverage / internal refactors / bug fixes in pkg/ (508 PRs, 49% of all tasks)
• Highest Success Rate: Cluster 2 — sandbox/engine/agent-runtime feature work (88.9% merged)
• Most Complex Tasks: Cluster 2 also has the highest average footprint (61 files changed, 8.7 commits, 5.8 reviews per PR) — these are substantial feature PRs, not quick fixes
• Outliers: 1 outlier cluster flagged — Cluster 1 (infra/container/MCP tooling), 15.7 points below the overall merge rate with 181 PRs (well above the 15-PR / 10-point threshold)

Cluster Analysis

Cluster 1: Infra, Container & MCP Tooling Fixes

• Size: 181 tasks (17.4% of total)
• Success Rate: 65.2%
• Avg Reviews: 1.7 · Avg Commits: 3.1 · Avg Files Changed: 38.8
• Top Keywords: aw, gh, gh aw, image, mcp, github, container, workflow
• Characteristics: CLI/MCP fixes, dependency and container-image version bumps, and vulnerability-remediation tasks against github-mcp-server, api-proxy, and similar images. A large share of the non-merged PRs are opened with a [WIP] title prefix.
• Example PRs: Fix silent pagination truncation in gh aw mcp inspect, dedupe connect logic #51193, Bump Claude Code, OpenAI Codex, and Pi CLI pinned versions #51194, [WIP] Fix vulnerabilities in container image ghcr.io/github/github-mcp-server:v1.8.0 #51332, [WIP] Fix vulnerabilities in api-proxy container image #51333, [WIP] Fix Docker Hub secret check to soft-fail for scheduled workflows #51411

---

Cluster 0: Test Coverage & Internal Refactors

• Size: 508 tasks (48.7% of total)
• Success Rate: 84.6%
• Avg Reviews: 3.1 · Avg Commits: 4.3 · Avg Files Changed: 17.9
• Top Keywords: pkg, coverage, error, added, behavior, test, existing, tests
• Characteristics: The bulk of day-to-day work — adding unit test coverage, refactoring internal wrappers/handlers, and small behavior fixes inside pkg/. High volume, consistently above-baseline success.
• Example PRs: Refactor engine secret validation wrappers #51216, Refactor safe-outputs handler assembly #51229, Ensure behavior-defined engines always emit AWF install steps under firewall mode #51274, Increase post-result watchdog default to 120s and add frontmatter timeout override #51292, Refactor engine import defaults parameters #51295

---

Cluster 3: Workflow Prompt & Config Changes

• Size: 300 tasks (28.8% of total)
• Success Rate: 82.7%
• Avg Reviews: 1.9 · Avg Commits: 3.2 · Avg Files Changed: 30.2
• Top Keywords: workflow, workflows, copilot, daily, md, lock, agent, github
• Characteristics: Edits to .md workflow prompts and their generated .lock.yml, sandbox/engine defaults, and agentic-workflow behavior tuning.
• Example PRs: Add GitHub CLI fallback guidance to Daily Team Evolution Insights workflow prompt #51192, Fix silent conversation-transcript fetch failures in Copilot session data fetcher #51195, Default GitHub-hosted Ubuntu runners to Docker Sandbox #51254, feat: enable docker-sbx sandbox for ~50% of agentic workflows #51264, Install AWF binary for behavior-defined engines that declare no npm installation #51276

---

Cluster 2: Sandbox / Engine / Agent-Runtime Features

• Size: 54 tasks (5.2% of total)
• Success Rate: 88.9% (highest of all clusters)
• Avg Reviews: 5.8 · Avg Commits: 8.7 · Avg Files Changed: 61.1
• Top Keywords: sous, sous chef, chef, pr, aic (note: "sous chef"/"aic" are boilerplate from the "PR Sous Chef" bot footer present on many PR bodies, not a content theme — the real theme is visible in the titles)
• Characteristics: The largest, most heavily-reviewed feature PRs — sandbox runtime install controls, sandbox.agent defaults, skills/sub-agent frontmatter syntax. Despite being the most complex cluster by footprint, it has the best merge rate, suggesting these tasks get thorough review but land reliably once complete.
• Example PRs: fix(pydantic): install the pydantic-ai CLI and match the real pai interface #51242, feat(threat-detection): render detection.log in detection job with group/mask macros #51255, feat: sandbox.agent id:awf is now default — remove redundant explicit declarations #51282, feat(sandbox): add runtime-install field to control sbx/gVisor install step generation #51413, Add explicit end marker syntax for inline skills and sub-agents #51446

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Reviews	Avg Files Changed	Top Keywords
0	Test coverage & refactors	508	84.6%	3.1	17.9	pkg, coverage, error, test
1	Infra/container/MCP fixes	181	65.2% ⚠️	1.7	38.8	aw, gh, image, mcp, container
2	Sandbox/engine features	54	88.9%	5.8	61.1	sandbox, agent, skills, runtime
3	Workflow prompt/config	300	82.7%	1.9	30.2	workflow, copilot, md, lock

Overall: 1,043 tasks · 80.9% merged

Sample Data Table (20 representative PRs)

PR #	Title	Cluster	Outcome
51216	Refactor engine secret validation wrappers	0	Merged
51229	Refactor safe-outputs handler assembly	0	Merged
51274	Ensure behavior-defined engines always emit AWF install steps	0	Merged
51292	Increase post-result watchdog default to 120s	0	Merged
51295	Refactor engine import defaults parameters	0	Merged
51193	Fix silent pagination truncation in gh aw mcp inspect	1	Merged
51194	Bump Claude Code, OpenAI Codex, and Pi CLI pinned versions	1	Merged
51332	[WIP] Fix vulnerabilities in container image github-mcp-server:v1.8.0	1	Closed
51333	[WIP] Fix vulnerabilities in api-proxy container image	1	Closed
51411	[WIP] Fix Docker Hub secret check to soft-fail for scheduled workflows	1	Closed
51242	fix(pydantic): install the pydantic-ai CLI, match real pai interface	2	Merged
51255	feat(threat-detection): render detection.log with group/mask macros	2	Merged
51282	feat: sandbox.agent id:awf is now default	2	Merged
51413	feat(sandbox): add runtime-install field for sbx/gVisor step gen	2	Merged
51446	Add explicit end marker syntax for inline skills and sub-agents	2	Merged
51192	Add GitHub CLI fallback guidance to workflow prompt	3	Merged
51195	Fix silent conversation-transcript fetch failures	3	Merged
51254	Default GitHub-hosted Ubuntu runners to Docker Sandbox	3	Merged
51264	feat: enable docker-sbx sandbox for ~50% of agentic workflows	3	Merged
51276	Install AWF binary for behavior-defined engines with no npm install	3	Merged

(Full dataset covers all 1,043 PRs; this table shows the top rows per cluster for readability.)

Key Findings

1. Infra/container/MCP tasks underperform by ~16 points: Cluster 1 merges at 65.2% vs. an 80.9% overall rate — the only cluster large enough (181 PRs) and low enough to trigger the outlier threshold.
2. [WIP]-titled tasks rarely land: within Cluster 1's 63 non-merged PRs, 33 (52%) carry a [WIP] prefix, versus only 16 of 118 (14%) among the cluster's merged PRs — WIP framing correlates strongly with abandonment in this cluster.
3. Most non-merges in this cluster are silent, not rejected: 36 of 63 (57%) non-merged Cluster 1 PRs closed with zero comments and zero reviews — they weren't blocked by CI failures or review pushback, they simply never got picked up before being closed.
4. Complexity doesn't predict failure elsewhere: Cluster 2 has the largest average footprint (61 files, 8.7 commits) of any cluster yet the highest merge rate (88.9%), so size/complexity alone doesn't explain Cluster 1's gap — the task framing does.

Lowest-Merge Cluster Root Cause

• Cluster: 1 — Infra/Container/MCP tooling fixes (keywords: aw, gh, image, mcp, container, workflow)
• Merge Rate Gap: 65.2% vs. 80.9% overall (gap: 15.7 points)
• Primary Blocker: Silent abandonment of speculative/exploratory tasks (not CI failure, not review rejection)
• Confidence: Medium-high — based on interaction-count and title-pattern evidence across all 63 non-merged PRs in the cluster, not just automated keyword matching
• Evidence PRs: [WIP] Fix vulnerabilities in container image ghcr.io/github/github-mcp-server:v1.8.0 #51332, [WIP] Fix vulnerabilities in api-proxy container image #51333, [WIP] Fix Docker Hub secret check to soft-fail for scheduled workflows #51411, [WIP] Add robots.txt to docs site allowing AI crawlers #51419, Verify aw-compat audit findings (#51044) are already resolved — no code changes required #51424, [WIP] Add allowed-refs to Smoke Copilot's dispatch_workflow config #51451, [WIP] Create /llms.txt for the docs site #51616 (all closed with 0 comments / 0 reviews)

Blocker Category	Count (of 63 non-merged)	Share	Notes
Zero-interaction closure (no comments/reviews)	36	57%	Closed without any recorded human or bot feedback — task likely superseded, deprioritized, or the PR was a speculative draft never triaged
Some interaction, no clear CI/review signal	20	32%	Had 1+ comments/reviews but no explicit CI-failure or "changes requested" language
Review friction (CHANGES_REQUESTED / explicit concerns)	7	11%	e.g. #54351 "Harden custom MCP mount source validation" — flagged for validation-source risk

Automated keyword classification (CI-failure / review-friction / scope-mismatch language search over comments+reviews) found almost no explicit CI-failure or scope-complaint text in this cluster's non-merged PRs — the dominant signal instead came from the absence of any interaction at all, which the keyword classifier can't detect directly. That absence, combined with the [WIP] title pattern, points to tasks opened speculatively (container/dependency vulnerability sweeps, opportunistic hardening) that were never converted into reviewable, mergeable work.

Recommendations

Based on clustering analysis:

1. Give infra/container/vulnerability-fix tasks a clearer completion bar. Cluster 1's high [WIP] and zero-interaction rate suggests these tasks are being opened speculatively without a concrete acceptance criterion (e.g. "bump image to vX and pass the vuln scan"). Scope them with an explicit success condition before dispatch.
2. Auto-close-and-report stale WIP PRs with a summary comment instead of silent closure, so the reason for abandonment (blocked dependency, out of scope, superseded) is recorded and this cluster's true blocker becomes visible in future data.
3. Cluster 0 and 3 (test coverage, workflow-prompt edits) are the safest task types to keep scaling — both sit at/above the overall merge rate with moderate footprint and normal review load.
4. Cluster 1 (outlier) recommendation: Route container-image/dependency-vulnerability tasks through a lighter, more mechanical flow (version bump + scan pass/fail) rather than open-ended [WIP] exploration, and require at least one triage comment before closing unmerged PRs in this category so future analyses can distinguish "fixed elsewhere" from "abandoned."

---

Generated by Prompt Clustering Analysis (Run: 32955555028)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 58.7 AIC · ⌖ 8.56 AIC · ⊞ 15.5K · ◷

• expires on Aug 27, 2026, 2:26 AM UTC-08:00