[prompt-clustering] Prompt Clustering Analysis - 2026-09-01

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-11 to 2026-09-01)
Total Tasks Analyzed: 1000 copilot-authored PRs
Clusters Identified: 6
Overall Merge Rate: 80.9% (809/1000)

Full Analysis Report

General Insights

Note: workflow run logs (gh-aw logs) were not available in this environment, so turn counts, duration, and token cost could not be joined to these PRs. Metrics below use comments/reviews/commits/files-changed as proxies for task complexity instead.

• Most Common Task Type: Cluster 4 — general workflow/agent/engine feature work and fixes (452 PRs, 45% of total)
• Highest Success Rate: Cluster 5 — test coverage / error-handling / rule additions (86.7%)
• Most Complex Tasks: Cluster 1 (container/image/CVE remediation) averages 67 files changed per PR — by far the largest diffs of any cluster
• Outliers: 2 clusters flagged (merge rate ≥10 points below the 80.9% baseline and ≥15 PRs): Cluster 3 (24.3%, gap 56.6 pts) and Cluster 1 (70.8%, gap 10.1 pts)

Cluster Analysis

Cluster 1: Container/dependency image pinning and CVE remediation

• Size: 65 tasks (6.5% of total)
• Merge Rate: 70.8%
• Avg files changed: 67.1 (avg commits 3.1, avg comments 4.4)
• Top Keywords: image, version, release, pin, upstream, container, pinned, scan
• Characteristics: Security/dependency housekeeping — bumping pinned container digests, remediating CVE scan findings, and documenting license exceptions. Large, sprawling diffs across pin catalogs.
• Example PRs: Track container CVE burn-down Critical-first with a consolidated parent issue #52580, Retry DIFC proxy image startup #52629, [WIP] Update container image to resolve vulnerabilities #52630, deps: bump safe patch updates for x/crypto, x/mod, and lipgloss #52669, No gh-aw change: Anthropic proxy fix requires gh-aw-firewall #52671

"Remove vulnerable gh-aw-firewall/squid:0.27.44 digest pin from shared container pin catalogs"

---

Cluster 2: GitHub Actions job-failure triage

• Size: 21 tasks (2.1% of total)
• Merge Rate: 85.7%
• Top Keywords: actions, fix, job, url, github actions, plan, started, make
• Characteristics: Reactive "fix this failing CI job" tasks, mostly titled [WIP] Fix failing GitHub Actions job .... Despite the WIP label these merge well above average.
• Example PRs: [WIP] Fix failing GitHub Actions job 'Integration: CLI Completion & Other' #52927, [WIP] Fix failing GitHub Actions job Integration: CLI Compile & Poutine #53274, [WIP] Fix failing GitHub Actions job Integration: Workflow Features #53443, [WIP] Fix failing GitHub Actions job Integration: Workflow Compiler #54008, [WIP] Fix failing GitHub Actions job Integration: Workflow Infra #54032

---

Cluster 3: Abandoned WIP placeholder PRs ⚠️ (lowest-merge outlier)

• Size: 37 tasks (3.7% of total)
• Merge Rate: 24.3% — the weakest of any cluster, 56.6 points below baseline
• Top Keywords: aw, gh aw, gh, issue, description, 08, 2026 (generic/templated language, not a coherent subject)
• Characteristics: Almost every unmerged PR here (27 of 28) has zero additions, zero deletions, zero changed files, and exactly one empty commit — an automation opened a placeholder branch/PR for a task and no code was ever pushed before it was closed. A PR Triage bot comment on [WIP] Fix safe-outputs.runs-on type mismatch #53989 confirms the pattern verbatim: "WIP placeholder PR (0 diff)... no substantive changes yet... Recommended action: defer." The few PRs in this cluster that merged ([test-parallel] Add t.Parallel() to safe Go test subtests (batch 1) #53083, Support issue field activity types in workflow schemas #53441, [test-parallel] Add t.Parallel() to safe top-level Go tests (batch) #53692, [test-parallel] test(cli): parallelize safe Go tests in pkg/cli (batch) #53721, Redirect Astro wizard docs to hosted wizard #53815, Collapse manual PR recovery instructions into a details section #55667, [test-parallel] test: add t.Parallel() to safe linter analyzer tests #55832, [test-parallel] Add t.Parallel() to safe Go test candidates (episode, fallback integration, filtering) #56216, [WIP] Add missing organization-custom-org-roles and organization-custom-repository-roles to JSON Schema #56982) all had real diffs and normal titles — it's specifically the empty-diff subset dragging the cluster down.
• Example PRs: [WIP] Remove probe5 item marked for deletion #52997, [WIP] Delete unnecessary probe4 issue #52998, [WIP] Fix issue with probe3 deletions #53003, [WIP] Fix failed jobs in Smoke DeepSeek Harness workflow #53008, [WIP] Fix MCP tool to respect strict: false frontmatter #53065

---

Cluster 4: Workflow/agent engine feature work and bug fixes

• Size: 452 tasks (45.2% of total)
• Merge Rate: 81.9%
• Top Keywords: workflow, workflows, copilot, agent, job, engine, generated, run
• Characteristics: The bulk of day-to-day gh-aw development — engine integrations, retry logic, model inventory updates, CLI diagnostics.
• Example PRs: Improve actionable CLI error messages #52175, [model-inventory] Add missing MAI-Code model entry and correct Copilot cache-read pricing #52191, Add direct Ollama inference preflight and failure diagnostics #52197, Retry cold-start Claude connection refusals as fresh runs #52198, Retry Claude connection failures as fresh runs #52199

---

Cluster 5: Test coverage, error messages, and validation rules

• Size: 338 tasks (33.8% of total)
• Merge Rate: 86.7% — highest of any cluster
• Top Keywords: added, coverage, behavior, error, test, rule, existing, tests
• Characteristics: Well-scoped, mechanical improvements (error-message compliance passes, added test coverage, validation rule tightening). Highest merge rate, consistent with narrow, low-risk diffs.
• Example PRs: Base full-mode patch/bundle on GITHUB_SHA when running from a non-default branch #52164, Raise error-message compliance in dispatch validation and CLI evaluation/logging paths #52173, Raise error-message compliance for workflow validation batch (5 files) #52174, Raise error-message compliance for sandbox and workflow run validation paths #52176, Fix error message compliance in runs_on_validation, frontmatter_extraction_metadata, reactions, call_workflow_validation, and ambient_folders #52177

---

Cluster 6: Safe-outputs feature and workflow tooling

• Size: 87 tasks (8.7% of total)
• Merge Rate: 83.9%
• Top Keywords: safe, safe output, output, pr, workflow, outputs, safe outputs, run
• Characteristics: Safe-outputs subsystem work — new output types, accounting fixes, Design Decision Gate tuning.
• Example PRs: Add safe output for approving fork pull request workflow runs #52541, Fix Go Logger safe-output accounting and failure diagnostics #52661, Update SPDD specification maintenance artifacts #52756, Fix Design Decision Gate false failures caused by the LLM invocation cap #52836, Cut AI Moderator token usage by pre-fetching capped moderation context #52891

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
1	Container/CVE pinning	65	70.8%	67.1	image, version, pin, container
2	CI job-failure triage	21	85.7%	1.9	actions, fix, job
3	Abandoned WIP placeholders	37	24.3%	1.8	aw, gh aw, issue
4	Workflow/engine features	452	81.9%	33.5	workflow, copilot, agent, engine
5	Test coverage / error messages	338	86.7%	8.0	added, coverage, test, rule
6	Safe-outputs tooling	87	83.9%	30.9	safe output, workflow

Full Data Table

Sample PRs per cluster (5 each, 30 total)

PR #	Title	Cluster	Outcome
52580	Track container CVE burn-down Critical-first with a consolidated parent issue	1	Merged
52629	Retry DIFC proxy image startup	1	Closed
52630	[WIP] Update container image to resolve vulnerabilities	1	Closed
52669	deps: bump safe patch updates for x/crypto, x/mod, and lipgloss	1	Merged
52671	No gh-aw change: Anthropic proxy fix requires gh-aw-firewall	1	Closed
52927	[WIP] Fix failing GitHub Actions job 'Integration: CLI Completion & Other'	2	Merged
53274	[WIP] Fix failing GitHub Actions job Integration: CLI Compile & Poutine	2	Merged
53443	[WIP] Fix failing GitHub Actions job Integration: Workflow Features	2	Merged
54008	[WIP] Fix failing GitHub Actions job Integration: Workflow Compiler	2	Merged
54032	[WIP] Fix failing GitHub Actions job Integration: Workflow Infra	2	Merged
52997	[WIP] Remove probe5 item marked for deletion	3	Closed
52998	[WIP] Delete unnecessary probe4 issue	3	Closed
53003	[WIP] Fix issue with probe3 deletions	3	Closed
53008	[WIP] Fix failed jobs in Smoke DeepSeek Harness workflow	3	Closed
53065	[WIP] Fix MCP tool to respect strict: false frontmatter	3	Closed
52175	Improve actionable CLI error messages	4	Merged
52191	[model-inventory] Add missing MAI-Code model entry and correct Copilot cache-read pricing	4	Merged
52197	Add direct Ollama inference preflight and failure diagnostics	4	Merged
52198	Retry cold-start Claude connection refusals as fresh runs	4	Merged
52199	Retry Claude connection failures as fresh runs	4	Merged
52164	Base full-mode patch/bundle on GITHUB_SHA when running from a non-default branch	5	Merged
52173	Raise error-message compliance in dispatch validation and CLI evaluation/logging paths	5	Merged
52174	Raise error-message compliance for workflow validation batch (5 files)	5	Merged
52176	Raise error-message compliance for sandbox and workflow run validation paths	5	Merged
52177	Fix error message compliance in runs_on_validation, frontmatter_extraction_metadata, reactions, call_workflow_validation, and ambient_folders	5	Merged
52541	Add safe output for approving fork pull request workflow runs	6	Merged
52661	Fix Go Logger safe-output accounting and failure diagnostics	6	Merged
52756	Update SPDD specification maintenance artifacts	6	Merged
52836	Fix Design Decision Gate false failures caused by the LLM invocation cap	6	Merged
52891	Cut AI Moderator token usage by pre-fetching capped moderation context	6	Merged

Full PR-to-cluster assignments for all 1000 PRs are available in the workflow's cached data (pr-cluster-assignments.json) for anyone who wants to drill in further.

Key Findings

1. Nearly half of all agent tasks (45%) are general workflow/engine feature work (Cluster 4), and it merges close to the overall average (81.9% vs 80.9%) — the bread-and-butter task type is healthy.
2. The highest-merging work is narrow and mechanical: test-coverage and error-message-compliance passes (Cluster 5, 338 PRs, 86.7%) succeed far more often than sprawling, multi-file security remediation (Cluster 1, 65 PRs, 70.8%, 67 files/PR on average).
3. A large share of "failed" tasks never actually failed — they never started. 73% of Cluster 3's non-merges (27 of 37 PRs total) are zero-diff placeholder PRs with a single empty commit, not tasks that were attempted and rejected.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 3 — Abandoned WIP placeholder PRs
• Merge Rate Gap: 24.3% vs 80.9% overall (gap: 56.6 points)
• Primary Blocker: Other/unknown — specifically, empty placeholder PRs that were opened but never received a code commit, then closed. This is not a code-review or CI outcome; it's an upstream task-dispatch/scaffolding gap.
• Confidence: High — 27 of 37 PRs (73%) in the cluster have additions=0, deletions=0, changedFiles=0, and exactly 1 commit; a PR Triage bot comment on [WIP] Fix safe-outputs.runs-on type mismatch #53989 independently confirms: "WIP placeholder PR (0 diff)... Recommended action: defer." Only 1 of the 28 unmerged PRs in this cluster had any real diff ([WIP] Fix workflow failure in daily experiment report #54245, blocked by a Copilot SDK server connectivity issue in the cloud hypervisor environment, unrelated to the placeholder pattern).
• Evidence PRs: [WIP] Remove probe5 item marked for deletion #52997, [WIP] Delete unnecessary probe4 issue #52998, [WIP] Fix issue with probe3 deletions #53003, [WIP] Fix MCP tool to respect strict: false frontmatter #53065, [WIP] Fix safe-outputs.runs-on type mismatch #53989

Blocker Category	Count	Share	Notes
Other/unknown (empty placeholder, never coded)	27	73%	additions=deletions=files=0, 1 commit, no reviews
CI/test failure	0	0%	none observed in this cluster
Review friction	0	0%	none observed — most have zero reviews
Scope/complexity mismatch (real diff, unresolved env/infra issue)	1	3%	#54245: Copilot SDK server unreachable in cloud hypervisor
Merged (successful)	9	24%	real diffs, normal review flow

For contrast, the secondary (borderline) outlier, Cluster 1 (container/CVE pinning, 70.8% merge rate, 10.1-point gap), shows a different and more conventional pattern: several unmerged PRs there also have zero diffs, but the ones with real diffs are large (up to ~6,800 lines added, e.g. #57514) multi-file dependency/pin-catalog changes — consistent with genuine scope/complexity and review friction on security-sensitive infrastructure changes, not the placeholder pattern seen in Cluster 3.

Recommendations

1. Investigate the placeholder-PR dispatch path. Whatever automation opens a branch/PR before the actual coding agent has produced a diff should either (a) not open the PR until the agent has committed real changes, or (b) auto-close/label placeholders distinctly from genuine WIP-but-attempted tasks, so they stop diluting merge-rate metrics and cluttering the PR queue.
2. Keep favoring narrowly-scoped, single-purpose tasks (error-message compliance, test coverage) — Cluster 5's 86.7% merge rate shows this pattern works well; consider decomposing large security/dependency remediation tasks (Cluster 1) into smaller, single-pin-catalog-file PRs to reduce the 67-files-per-PR average and ease review.
3. For CI-failure-triage tasks (Cluster 2), the current [WIP] Fix failing GitHub Actions job ... framing already works well (85.7% merge rate) — no change needed there; treat it as a template for other reactive-fix task types.
4. Outlier Cluster Recommendation: For Cluster 3, add a pre-flight check in the task-dispatch workflow that verifies a non-empty diff exists before marking a PR ready for review, and route the ~3% genuine-but-blocked case (like [WIP] Fix workflow failure in daily experiment report #54245's cloud-hypervisor connectivity issue) to infra owners rather than leaving it in the same closed-without-comment state as the placeholders.

---

Generated by Prompt Clustering Analysis (Run: 33493917088)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 141.4 AIC · ⌖ 11 AIC · ⊞ 15.7K · ◷

• expires on Sep 2, 2026, 2:11 AM UTC-08:00