[prompt-clustering] Prompt Clustering Analysis - 2026-08-20

[github-actions[bot]]

Summary

Analysis Period: 2026-08-03 to 2026-08-20 (most recent 1,000 copilot-authored PRs available)
Total Tasks Analyzed: 1,000
Clusters Identified: 5
Overall Merge Rate: 77.2% (772 merged / 213 closed / 15 open)

Data note: Per-run turn/duration/cost metrics from workflow logs were not used in this report. Fetching them via the logs tool only returns ~10 runs per call (~15–60s each), which would require on the order of 100 sequential calls to cover 1,000 PRs — out of budget for a daily run. This analysis relies on PR body/metadata/comments/reviews only, which was sufficient to find a clear, well-evidenced pattern (see below).

Full Analysis Report

General Insights

• Most common task type: Cluster 0, "Workflow & agentic feature config" (491 PRs, 49%) — general feature/config work on the gh-aw workflow system itself.
• Second largest: Cluster 3, "Go engineering" (388 PRs, 39%) — refactors, hardening, unit tests, formal-spec coverage in pkg/.
• Highest merge rate: Cluster 3, Go engineering, at 84.3%.
• Lowest merge rate / outlier: Cluster 4, "Stub/abandoned WIP tasks" (70 PRs, 7%) at 51.4% — 25.8 points below the 77.2% overall rate, and the only cluster meeting the outlier threshold (≥15 PRs, ≥10-point gap).
• Smallest, most homogeneous cluster: Cluster 2, "CI job-failure fixes" (16 PRs) — narrowly-scoped [WIP] Fix failing GitHub Actions job ... tasks, 81.2% merged.

Cluster Analysis

Cluster 0: Workflow & agentic feature config

• Size: 491 tasks (49.1% of total)
• Merge Rate: 75.2%
• Avg commits: 3.2 · Avg files changed: 29.7 · Avg comments: 4.3 · Avg reviews: 1.9
• Top Keywords: workflow, md, workflows, github, copilot, agent
• Characteristics: The catch-all bucket for feature/config changes to the gh-aw workflow system itself — new safe-outputs, compiler behavior, documentation for workflow authoring, MCP tool wiring.
• Example PRs: Skip GHES auto-detection during CI initialization #54112, Document compiler-generated job authentication #54100, Make failure investigator log capture fault-aware #54098, Raise max-patch-size for Deep Report repo-memory push #54029, Refresh aw.yml package dependencies during update #53974

---

Cluster 3: Go engineering (refactor/tests)

• Size: 388 tasks (38.8% of total)
• Merge Rate: 84.3% (highest)
• Avg commits: 4.1 · Avg files changed: 13.1 · Avg comments: 7.3 · Avg reviews: 3.3
• Top Keywords: pkg, test, coverage, added, error, add
• Characteristics: Focused, well-scoped Go code changes — extracting helpers, adding unit tests, hardening exec.Command usage, formal-spec coverage. Smaller diffs than Cluster 0 but more review engagement, and the best success rate of any cluster — narrow, mechanical, code-level tasks convert best.
• Example PRs: Enable external threat detection by default #54111, Add formal conformance-registry invariants for AWF config fixture index #54041, Split add_package_manifest.go into focused files #54030, Fix actionlint error: agent job referencing needs.approval_allowlist it never depends on #54028, Remove phantom permission factory documentation #54015

---

Cluster 4: Stub/abandoned WIP tasks (outlier)

• Size: 70 tasks (7.0% of total)
• Merge Rate: 51.4% (25.8 points below overall)
• Avg commits: 4.7 · Avg files changed: 47.3 · Avg comments: 9.2 · Avg reviews: 2.9
• Top Keywords: (auto-generated placeholder boilerplate — "Thanks for asking me to work on this... I will get started...")
• Characteristics: This cluster forms around PRs whose body is still the coding agent's initial auto-generated stub text rather than a filled-in description. Many originate from automated backlog-generation tools (Deep Report intelligence briefings, Avenger audits, "probe" issues, robots.txt/llms.txt requests). The averages above are skewed high by a merged subgroup that did real work but never updated its PR description; the closed subgroup (see root-cause section) never got any code at all.
• Example PRs: [WIP] Fix safe-outputs.runs-on type mismatch #53989 (closed), Improve pre-created pull request title and body #53956 (merged), pr-sous-chef: document GraphQL fetch for reviewThreads (fixes invalid gh pr view field) #53833 (merged), [WIP] Fix untrusted network data written to file in ensure-docs-slide-pdf.js #53765 (closed), [WIP] Create /llms.txt for the documentation site #53760 (closed)

---

Cluster 1: Agent-infra/meta refactoring

• Size: 35 tasks (3.5% of total)
• Merge Rate: 77.1%
• Avg commits: 2.7 · Avg files changed: 39.2 · Avg comments: 4.1 · Avg reviews: 1.7
• Top Keywords: pull request, ai, agent, target
• Characteristics: Meta-level changes to the agent/automation infrastructure itself (safeoutputs context forwarding, PR review tooling, confused-deputy protections). Roughly matches the overall merge rate.
• Example PRs: Fix safeoutputs PR trigger context forwarding #53475, Make PR Description Updater use triggering PR context for update_pull_request #53301, Extend confused deputy protection to pull_request_target #52976, Add anti-probing guidance to submit_pull_request_review tool description #51469, Warn on unsafe pull request target checkouts in non-strict mode #51103

---

Cluster 2: CI job-failure fixes

• Size: 16 tasks (1.6% of total — smallest cluster)
• Merge Rate: 81.2%
• Avg commits: 2.0 · Avg files changed: 1.6 · Avg comments: 0.8 · Avg reviews: 0.8
• Top Keywords: job, fix, actions, url, plan, started
• Characteristics: Highly templated [WIP] Fix failing GitHub Actions job <name> tasks with the smallest diffs and least discussion of any cluster — these are narrow, single-purpose CI-repair tasks that mostly get merged quickly with little review overhead.
• Example PRs: [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #54109, [WIP] Fix failing GitHub Actions job lint-go #54089, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #54088, [WIP] Fix failing GitHub Actions job Integration: Workflow Infra #54032, [WIP] Fix failing GitHub Actions job Integration: Workflow Compiler #54008

Merge Rate by Cluster

Cluster	Tasks	Merge Rate	Avg Commits	Avg Files Changed	Top Keywords
C0: Workflow/feature config	491	75.2%	3.2	29.7	workflow, md, workflows, github
C3: Go engineering	388	84.3%	4.1	13.1	pkg, test, coverage, error
C4: Stub/abandoned WIP	70	51.4%	4.7	47.3	(placeholder boilerplate)
C1: Agent-infra refactor	35	77.1%	2.7	39.2	pull request, ai, agent
C2: CI job-failure fixes	16	81.2%	2.0	1.6	job, fix, actions, url

Sample Data Table

Stratified sample (up to 5 merged + 4 non-merged per cluster, most recent first) — full dataset is 1,000 PRs; this table exists to illustrate cluster themes, not enumerate every task.

44-row sample across all 5 clusters

PR #	Title	Cluster	Outcome	Comments	Reviews
#54112	Skip GHES auto-detection during CI initialization	C0: Workflow/feature config	Merged	1	1
#54100	Document compiler-generated job authentication	C0: Workflow/feature config	Merged	1	3
#54098	Make failure investigator log capture fault-aware	C0: Workflow/feature config	Merged	0	1
#54029	Raise max-patch-size for Deep Report repo-memory push	C0: Workflow/feature config	Merged	1	1
#53974	Refresh aw.yml package dependencies during update	C0: Workflow/feature config	Merged	12	4
#54147	Extract shared gh-cli-proxy.md component to deduplicate cli-proxy	C0: Workflow/feature config	Open	2	0
#54146	Use a valid name for the activation artifact	C0: Workflow/feature config	Open	2	0
#54119	Re-resolve package manifests during workflow updates	C0: Workflow/feature config	Open	11	6
#54117	Rename activation artifact to avoid invalid-name upload failure	C0: Workflow/feature config	Open	1	0
#53475	Fix safeoutputs PR trigger context forwarding	C1: Agent-infra refactor	Merged	9	1
#53301	Make PR Description Updater use triggering PR context	C1: Agent-infra refactor	Merged	2	1
#52976	Extend confused deputy protection to pull_request_target	C1: Agent-infra refactor	Merged	11	5
#51469	Add anti-probing guidance to submit_pull_request_review tool	C1: Agent-infra refactor	Merged	5	1
#51103	Warn on unsafe pull request target checkouts in non-strict mode	C1: Agent-infra refactor	Merged	1	1
#53480	Relax approval gating for low-risk PR automations	C1: Agent-infra refactor	Closed	7	3
#51424	Verify aw-compat audit findings (#51044) are already resolved	C1: Agent-infra refactor	Closed	0	0
#50912	[WIP] Fix push_to_pull_request_branch for HEAD-only bundles	C1: Agent-infra refactor	Closed	0	0
#50469	Updating agentic workflows to new documentation standards	C1: Agent-infra refactor	Closed	0	0
#54109	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2	C2: CI job-failure fix	Merged	0	1
#54089	[WIP] Fix failing GitHub Actions job lint-go	C2: CI job-failure fix	Merged	0	1
#54088	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2	C2: CI job-failure fix	Merged	0	1
#54032	[WIP] Fix failing GitHub Actions job Integration: Workflow Infra	C2: CI job-failure fix	Merged	2	1
#54008	[WIP] Fix failing GitHub Actions job Integration: Workflow Compiler	C2: CI job-failure fix	Merged	0	1
#51148	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2	C2: CI job-failure fix	Closed	0	0
#51147	[WIP] Fix failing GitHub Actions job Integration: Workflow Infra	C2: CI job-failure fix	Closed	0	0
#50290	[WIP] Fix failing GitHub Actions job Integration: Workflow Features	C2: CI job-failure fix	Closed	0	0
#54111	Enable external threat detection by default	C3: Go engineering	Merged	3	1
#54041	Add formal conformance-registry invariants for AWF config fixture	C3: Go engineering	Merged	4	1
#54030	Split add_package_manifest.go into focused files	C3: Go engineering	Merged	11	4
#54028	Fix actionlint error: agent job referencing needs.approval_allowlist	C3: Go engineering	Merged	14	5
#54015	Remove phantom permission factory documentation	C3: Go engineering	Merged	1	1
#54148	Consolidate repository and semantic-version utilities	C3: Go engineering	Open	2	0
#54120	Support package resources with scoped ownership	C3: Go engineering	Open	10	5
#54110	Fix logs MCP tool truncating explicit date-range queries	C3: Go engineering	Open	18	12
#54097	Avoid CodeQL false positive in manifest path validation	C3: Go engineering	Open	1	0
#53956	Improve pre-created pull request title and body	C4: Stub/abandoned WIP	Merged	5	3
#53833	pr-sous-chef: document GraphQL fetch for reviewThreads	C4: Stub/abandoned WIP	Merged	1	2
#53679	Allow PR Sous Chef to approve CJS and CGO action-required runs	C4: Stub/abandoned WIP	Merged	9	6
#53576	Add pre-created pull request safe output mode	C4: Stub/abandoned WIP	Merged	27	5
#53494	Use workflow frontmatter emoji for activation status comments	C4: Stub/abandoned WIP	Merged	18	5
#53989	[WIP] Fix safe-outputs.runs-on type mismatch	C4: Stub/abandoned WIP	Closed	1	0
#53765	[WIP] Fix untrusted network data written to file in ensure-docs-slide	C4: Stub/abandoned WIP	Closed	0	0
#53760	[WIP] Create /llms.txt for the documentation site	C4: Stub/abandoned WIP	Closed	0	0
#53727	[WIP] Fix related runs link rendering issue in footer	C4: Stub/abandoned WIP	Closed	0	0

Key Findings

1. Narrow, mechanical Go tasks convert best. Cluster 3 (refactors, unit tests, hardening) has the highest merge rate (84.3%) despite being the second-largest cluster — small, well-scoped code-level asks are the safest bet for this agent.
2. CI-repair tasks are cheap and reliable. Cluster 2's median diff is tiny (1.6 files changed on average) with almost no review back-and-forth (0.8 comments/reviews average), yet merges 81.2% of the time — a good template for low-friction automation.
3. The lowest-merge cluster is really a "task never started" signature, not a topic. Cluster 4 clusters not by subject matter but by the coding agent's placeholder PR description ("Thanks for asking me to work on this...") still being present — see root-cause section below.

Lowest-Merge Cluster Root Cause

• Cluster: C4 — "Stub/abandoned WIP tasks"
• Merge Rate Gap: 51.4% vs 77.2% overall (gap: 25.8 points)
• Primary Blocker: Task abandoned before any code was written — not CI failure, not review friction, not scope disagreement.
• Confidence: High
• Evidence PRs: [WIP] Investigate sentrux '0 rules checked' tooling gap #50150, [WIP] Investigate and fix Avenger model mismatch between frontmatter and executed engine model #50317, [WIP] Update Avenger max-turns budget to match documented hard limit #50319, [WIP] Add regression coverage for model propagation in workflow #50321, [WIP] Remove redundant live CI re-check from Avenger Step 0 prompt #50322, [WIP] Surface allowed-files scope to agent prompt for restricted workflows #50351, [WIP] Document supported invocation path for ad hoc agentic-workflows evaluation #50357, [WIP] Add troubleshooting for unavailable evaluation surfaces and failure classification #50358, [WIP] Add compact multi-persona ad hoc evaluation example to workflow-creation docs #50359, [WIP] Improve discoverability by cross-linking ad hoc evaluation guidance #50360 (10 of many identical cases)

What the data shows: Of the 34 non-merged PRs in this cluster, 28 (82%) share an identical, exact fingerprint: 1 commit, 0 files changed, 0 additions, 0 deletions, 0 comments, 0 reviews. These are PRs where the agent opened the initial stub PR (title often [WIP] ..., body the generic "I will get started on it" placeholder) and then the PR was closed with zero further activity — no reviewer ever engaged, no follow-up commit was ever pushed. By contrast, the 36 merged PRs in the same cluster look nothing like this: they average 7.4 commits, 75.7 files changed, and 4.75 reviews — real work happened, the description just never got refreshed.

Many of the abandoned tasks trace back to auto-generated backlog sources — Deep Report intelligence briefings, Avenger audits, "probe" test issues, and recurring robots.txt/llms.txt requests — several of which appear more than once in the sample (e.g. "Create /llms.txt for the documentation site" recurs at #51616, #52115, #53158, #53760), suggesting duplicate or already-superseded issues get re-queued as new agent tasks.

Blocker Category	Count	Share	Notes
Abandoned pre-commit (0 files changed, 0 interaction)	28	82%	Identical 1-commit/0-diff/0-comment fingerprint; likely superseded, duplicate, or invalid before any real work began
Review friction (real code, but closed after discussion)	5	15%	e.g. #50744 (16 comments/10 reviews), #51090, #51282, #52137, #53572 — these did get reviewed and rejected/abandoned mid-flight
CI/test failure	0	0%	Not observed as a distinct blocker in this cluster
Other/unknown	1	3%	#53989 — 1 comment, closed with minimal interaction

Recommendations

1. Double down on narrow, code-level tasks. Cluster 3's pattern (small diff, focused scope, unit-test-style asks) has the best merge rate — prefer decomposing large feature work into this shape when queuing agent tasks.
2. Keep using the CI-repair template. Cluster 2's [WIP] Fix failing GitHub Actions job <name> format is cheap to run and merges reliably — a good pattern to replicate for other narrow, single-signal fixes.
3. Screen auto-generated backlog issues for staleness/duplication before assigning to the agent. A meaningful share of Cluster 4's failures are recurring or already-superseded asks (e.g. repeated llms.txt/robots.txt requests) — a dedupe/staleness check before task creation would avoid several dead-on-arrival PRs per week.
4. Add an early abandonment signal. 28 PRs sat with exactly 1 commit and 0 diff before being closed — instrumenting "PR open >N hours with 0 files changed" as a triage flag would surface these faster than waiting for a human to eventually close them, and would separate "the agent never got a real chance to work" from genuine review rejections in future analyses.

---

Generated by Prompt Clustering Analysis (Run: 32356081464)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 115 AIC · ⌖ 14.1 AIC · ⊞ 14.9K · ◷

• expires on Aug 21, 2026, 2:21 AM UTC-08:00