[prompt-clustering] Prompt Clustering Analysis - 2026-08-31

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-01 to 2026-08-31)
Total Tasks Analyzed: 1,013 copilot-authored PRs
Clusters Identified: 7
Overall Merge Rate: 80.8%

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 2 — general Go package fixes, test coverage, and error-message compliance (275 PRs, 27% of total)
• Second Most Common: Cluster 6 — refactoring & code consolidation, largely PR Sous Chef-assisted cleanup (244 PRs, 24% of total)
• Highest Success Rate: Cluster 3 — CI job failure fixes and Actions integration repair (92.9% merged, n=28)
• Most Complex Tasks: Cluster 6 (avg 33.7 files changed) and Cluster 5 (avg 38.1 files changed) — both involve broad, multi-file changes
• Persistent Outlier: Cluster 5 (gh/aw/mcp/image — container security & infra plumbing) has been the lowest-merge cluster for 3 consecutive daily runs (Aug 29, 30, 31), consistently 15-20 points below the fleet average

Cluster Analysis

Cluster 0: Engine/Model Routing (Copilot, Claude, Codex)

• Size: 109 tasks (10.8% of total)
• Success Rate: 73.4%
• Top Keywords: copilot, engine, model, claude, codex, smoke
• Characteristics: Tasks that patch model routing, engine-specific bugs (e.g. Linter Miner crashes on model: inherited), and model-inventory metadata. Below-average merge rate but not flagged as a statistical outlier (below the 15-PR / 10-point threshold on its own, though worth watching).
• Example PRs: Allow jq in Breaking Change Checker #52128, Fix Linter Miner crash: unsupported model: inherited on inline sub-agents #52133, [model-inventory] Add missing MAI-Code model entry and correct Copilot cache-read pricing #52191, Retry Claude connection failures as fresh runs #52199, Route PR code-quality reviews through the Copilot gateway #52377

---

Cluster 1: Shared Report Formatting & Workflow Standardization

• Size: 126 tasks (12.4% of total)
• Success Rate: 87.3%
• Top Keywords: daily, workflow, workflows, md, report, shared
• Characteristics: Bulk normalization tasks — migrating inline guidance to shared imports, standardizing report formatting across dozens of workflow files at once. High merge rate despite large diffs (avg 363 additions), suggesting mechanical/templated changes review well.
• Example PRs: Address Persona Explorer findings: trigger ambiguity, digest grouping, compliance example #52121, Normalize report formatting guidance for daily workflows #52413, Migrate inline report-formatting guidance to the shared/reporting.md import #52542, Extract shared smoke-test brevity prompt and import across 21 smoke workflows #52565, Normalize report formatting across 34 non-compliant workflows #52699

---

Cluster 2: Go Package Fixes, Coverage & Error-Message Compliance

• Size: 275 tasks (27.1% of total) — largest cluster
• Success Rate: 81.1%
• Top Keywords: pkg, coverage, error, test, rule, fixes
• Characteristics: The bread-and-butter category — targeted Go code fixes, test coverage additions, and compliance rule tightening. Smallest average diffs (6.5 files, 133 additions), consistent with focused bug fixes.
• Example PRs: Allow repository_dispatch user rate limits #52101, Add formal RS-05a checkout gate coverage #52103, Consolidate duplicate ordered dedupe/merge helpers into pkg/sliceutil #52159, Base full-mode patch/bundle on GITHUB_SHA when running from a non-default branch #52164, Raise error-message compliance in dispatch validation and CLI evaluation/logging paths #52173

---

Cluster 3: CI/Actions Job Failure Fixes

• Size: 28 tasks (2.8% of total)
• Success Rate: 92.9% — highest of any cluster
• Top Keywords: job, actions, fix, github, check, logs, failure
• Characteristics: Narrow, well-specified "fix this failing CI job" tasks. Smallest footprint (avg 1.8 files) and near-universal merge — the clearest case of task narrowness correlating with success.
• Example PRs: [WIP] Fix failing GitHub Actions job 'Integration: CLI Completion & Other' #52927, [WIP] Fix failing GitHub Actions job Integration: CLI Compile & Poutine #53274, Refactor custom job compiler into focused modules #53394, [WIP] Fix failing GitHub Actions job Integration: Workflow Features #53443, Add timing summaries to long-running validation workflows #53492

---

Cluster 4: Safe-Output & Agent Workflow Tooling

• Size: 80 tasks (7.9% of total)
• Success Rate: 82.5%
• Top Keywords: safe, safe output, output, outputs, workflow, agent
• Characteristics: Meta-tooling for the agentic-workflow system itself (safe-output fallbacks, health monitors, harness settings). Reasonably high merge rate for infra work.
• Example PRs: Add Agent Job Health Monitor to own fleet-wide agent-job failure rate #52108, Trim unused Matt Pocock skills from skills-reviewer workflow #52122, Add safe-output noop fallback to Daily Go Test Stubs (Aider) #52123, Document harness watchdog and retry runtime settings #52511, Update SPDD specification maintenance artifacts #52756

---

Cluster 5: Container/MCP/Infra Plumbing (gh aw) — ⚠️ Outlier

• Size: 151 tasks (14.9% of total)
• Success Rate: 62.9% — lowest of any cluster, 17.9 points below fleet average
• Top Keywords: gh, aw, gh aw, mcp, image, github, workflow, agent
• Characteristics: Container image CVE remediation, MCP server plumbing, digest pinning, and cross-repo infra work involving gh-aw-firewall. Largest average diffs (38.1 files, 497 additions) and highest volume of unmerged work (48 closed, 8 still open out of 151).
• Example PRs: [WIP] Add AI-crawler robots.txt allow rules and llms.txt for the docs site #52115, Add direct Ollama inference preflight and failure diagnostics #52197, Add regression tests preventing committed tool binaries #52540, Fix coverage.findProfile path matching so perf-linter coverage gating actually applies #52566, Track container CVE burn-down Critical-first with a consolidated parent issue #52580

---

Cluster 6: Refactoring & Code Consolidation

• Size: 244 tasks (24.1% of total)
• Success Rate: 89.8%
• Top Keywords: (noisy — post-cleaning residue from "PR Sous Chef" boilerplate); example titles show struct consolidation, file-splitting by lifecycle, SPDD compliance tightening
• Characteristics: PR-Sous-Chef-assisted cleanup tasks — deduplicating structs, splitting oversized files, tightening specs. Second-highest merge rate; heaviest average interaction (13.7 comments/PR) but still merges well.
• Example PRs: Give the four log-entry structs a shared LogEntry interface #52107, Split pkg/workflow/compiler_types.go by lifecycle: options, mutators, types #52109, Tighten SPDD compliance specs and fixtures #52111, Add preview Cloud Hypervisor runtime support for sandbox.agent.runtime with docker-sbx parity #52119, Consolidate JobStep/JobStepData identical structs in pkg/cli #52120

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Avg Comments
3	CI/Actions job fixes	28	92.9%	1.8	0.9
6	Refactoring/consolidation	244	89.8%	33.7	13.7
1	Report formatting/standardization	126	87.3%	17.9	2.2
4	Safe-output/agent tooling	80	82.5%	23.7	4.8
2	Go fixes/coverage/compliance	275	81.1%	6.5	4.4
0	Engine/model routing	109	73.4%	—	—
5	Container/MCP/infra (gh aw)	151	62.9% ⚠️	38.1	3.4

Full Data Table

Representative sample (6 PRs per cluster, 42 of 1013 total)

PR #	Title	Cluster	Outcome	Files Changed
52128	Allow jq in Breaking Change Checker	0	Merged	2
52133	Fix Linter Miner crash: unsupported model: inherited...	0	Merged	3
52191	[model-inventory] Add missing MAI-Code model entry...	0	Merged	3
52199	Retry Claude connection failures as fresh runs	0	Closed	2
52377	Route PR code-quality reviews through the Copilot gateway	0	Merged	3
52572	Extract reusable workflow read-permission components	0	Closed	22
52121	Address Persona Explorer findings: trigger ambiguity...	1	Merged	4
52413	Normalize report formatting guidance for daily workflows	1	Merged	20
52542	Migrate inline report-formatting guidance to shared import	1	Merged	172
52565	Extract shared smoke-test brevity prompt (21 workflows)	1	Merged	43
52573	Enforce Sentrux architecture constraints	1	Merged	3
52699	Normalize report formatting across 34 non-compliant workflows	1	Merged	69
52101	Allow repository_dispatch user rate limits	2	Merged	2
52103	Add formal RS-05a checkout gate coverage	2	Merged	1
52159	Consolidate duplicate ordered dedupe/merge helpers	2	Closed	6
52164	Base full-mode patch/bundle on GITHUB_SHA	2	Merged	6
52173	Raise error-message compliance in dispatch validation	2	Closed	13
52176	Raise error-message compliance for sandbox validation	2	Merged	8
52927	[WIP] Fix failing GitHub Actions job 'Integration: CLI...'	3	Merged	1
53274	[WIP] Fix failing GitHub Actions job Integration: CLI Compile	3	Merged	2
53394	Refactor custom job compiler into focused modules	3	Merged	6
53443	[WIP] Fix failing GitHub Actions job Integration: Workflow	3	Merged	4
53492	Add timing summaries to long-running validation workflows	3	Merged	3
54008	[WIP] Fix failing GitHub Actions job Integration: Workflow	3	Merged	1
52108	Add Agent Job Health Monitor	4	Merged	3
52122	Trim unused Matt Pocock skills from skills-reviewer workflow	4	Merged	3
52123	Add safe-output noop fallback to Daily Go Test Stubs (Aider)	4	Closed	3
52511	Document harness watchdog and retry runtime settings	4	Merged	5
52756	Update SPDD specification maintenance artifacts	4	Merged	7
52836	Fix Design Decision Gate false failures	4	Merged	8
52115	[WIP] Add AI-crawler robots.txt allow rules and llms.txt	5	Closed	0
52197	Add direct Ollama inference preflight and failure diagnostics	5	Merged	2
52540	Add regression tests preventing committed tool binaries	5	Merged	1
52566	Fix coverage.findProfile path matching	5	Merged	2
52580	Track container CVE burn-down Critical-first	5	Merged	2
52629	Retry DIFC proxy image startup	5	Closed	2
52107	Give the four log-entry structs a shared LogEntry interface	6	Merged	3
52109	Split pkg/workflow/compiler_types.go by lifecycle	6	Merged	10
52111	Tighten SPDD compliance specs and fixtures	6	Merged	15
52119	Add preview Cloud Hypervisor runtime support	6	Merged	27
52120	Consolidate JobStep/JobStepData identical structs	6	Merged	2
52129	Improve test quality in compiler_file_size_test.go	6	Merged	1

Key Findings

1. Task narrowness predicts success: Cluster 3 (single-purpose CI job fixes, avg 1.8 files) has the highest merge rate (92.9%), while Cluster 5 (broad infra/container work, avg 38.1 files) has the lowest (62.9%). Smaller, well-bounded diffs merge more reliably regardless of subject matter.
2. The gh aw/container/MCP cluster is a recurring, not one-off, weak spot: it has been the lowest-merge outlier in all three most recent daily runs (Aug 29: 66.7%, Aug 30: 65.8%, Aug 31: 62.9%), always centered on the same terms (image, container, mcp, gh-aw-firewall). This is a structural pattern, not noise.
3. Refactoring/consolidation work (Cluster 6) merges well despite heavy review (13.7 comments/PR average, the highest of any cluster) — suggesting iteration-heavy review is not itself a blocker when the underlying change is well-scoped.
4. Bulk mechanical changes merge cleanly: Cluster 1's report-formatting standardization tasks touch up to 172 files in a single PR (Migrate inline report-formatting guidance to the shared/reporting.md import #52542) yet still merge at 87.3% — template-driven changes tolerate large diffs better than exploratory ones.

Lowest-Merge Cluster Root Cause

• Cluster: 5 — Container/MCP/infra plumbing (gh aw, image, mcp)
• Merge Rate Gap: 62.9% vs 80.8% overall (gap: 17.9 points)
• Primary Blocker: Scope/complexity mismatch — the agent is frequently assigned tasks whose correct resolution lies outside this repository (in github/gh-aw-firewall) or that are opened as investigation stubs and never carried to completion
• Confidence: High — corroborated by explicit PR titles/bodies ("No gh-aw change...", "zero-diff investigation", [WIP] prefixes) and by triage-bot comments recommending defer/close
• Evidence PRs: [WIP] Update container image to resolve vulnerabilities #52630 (WIP, unresolved container vuln fix), Document upstream blocker for cli-proxy vulnerability remediation #52705 (documents upstream blocker, correctly not merged), Route Serena MCP workflows through :latest with digest pinning for security refreshes #52935 ("resolve merge conflicts" requested, stalled), Inline agentic commands router JavaScript #52986 (CHANGES_REQUESTED — inline JS diff too large for safe review), Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674 (task-preflight skill added specifically to stop zero-diff retries on this exact task type)

Manual review of 48 closed (non-merged) PRs in this cluster:

Blocker Category	Count	Share	Notes
WIP/abandoned draft (never completed)	20	41.7%	Titles prefixed [WIP]; work started but never finished across a follow-up run
Upstream-blocked scope mismatch	12	25.0%	Task requires a fix in github/gh-aw-firewall or another repo; agent correctly produces a zero-diff "no code change here" PR that gets closed rather than merged
Review friction (changes requested)	3	6.3%	Diff too large to review safely (e.g. #52986, 3,506-line inline JS bundle) or hardening concerns raised
Other/unknown	13	27.1%	Includes several PRs with clean, favorably-reviewed implementations closed for reasons outside the diff itself (likely superseded/duplicate closures) — worth a follow-up audit

Combined, 67% of this cluster's non-merges are structural (task genuinely can't be completed in this repo, or was abandoned mid-flight) rather than code-quality problems — this is a task-routing issue, not an agent-competence issue.

Recommendations

1. Route container/CVE-remediation tasks more precisely: A large share of Cluster 5 tasks are zero-diff investigations that correctly conclude "fix belongs in gh-aw-firewall." Consider pre-filtering these at issue-creation time (check which repo owns the affected image/binary) so agents don't spend a full run producing a PR that's structurally unmergeable here.
2. Add explicit completion criteria to [WIP] tasks: 20 of 48 closed PRs in the outlier cluster were abandoned drafts. A stub-detection or task-preflight check (similar to the one already added in Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674) could flag these before they consume a full agent run, or auto-continue them in a follow-up instead of leaving them to close.
3. Cap diff size for infra/container tasks before requesting review: Inline agentic commands router JavaScript #52986's 3,506-line inline-JS PR drew multiple CHANGES_REQUESTED reviews specifically citing size. Splitting large infra migrations into reviewable increments (as Cluster 1's templated changes already do successfully) would likely improve this cluster's merge rate.
4. Investigate the "Other/unknown" closures (27% of Cluster 5's non-merges): several of these PRs received clearly favorable triage/review comments yet were still closed — a follow-up audit of why well-reviewed work in this cluster gets closed anyway (duplicate/superseded, stale, or manual override) would clarify whether there's a second, distinct blocker beyond scope mismatch.

---

Generated by Prompt Clustering Analysis (Run: 33379323286). Data limitation: workflow-run turn/duration/cost metrics were unavailable for this run (no gh-aw CLI logs present in the environment), so analysis is based on PR content and interaction data only.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 147.5 AIC · ⌖ 12.3 AIC · ⊞ 15.7K · ◷

• expires on Sep 1, 2026, 2:14 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #57663.