[prompt-clustering] Prompt Clustering Analysis - 2026-08-03

[github-actions[bot]]

Summary

Analysis Period: 2026-07-15 → 2026-08-03 (last 1000 copilot-agent PRs)
Total Tasks Analyzed: 1000
Clusters Identified: 10 (best silhouette among k=4–10)
Overall Success Rate: 75.5% merged (755 merged / 239 closed / 6 open)

The merge rate has been drifting down for six weeks straight — from ~80–82% in late June/early July to 75.5% today. See the trend chart below.

Full Analysis Report

General Insights

• Most common task type: docs/spec/test-coverage work (cluster 0, 162 PRs, 16%) and "daily workflow report normalization" tasks (cluster 1, 140 PRs, 14%) dominate volume.
• Highest success rate: eslint/error-handling remediation (cluster 6, 87%) and the "sous-chef"/engine-plumbing grab-bag (cluster 3, 87%).
• Lowest success rate: container-image vulnerability/digest-pin remediation (cluster 9, 58%) — by far the weakest cluster — and daily-workflow report normalization (cluster 1, 64%).
• By PR type (from existing pr-type:* labels, backfilled from title prefix where missing): bug fixes are both the largest category (343 PRs) and the strongest performer (79%); refactors are the weakest labeled category (66%).
• By risk/priority label: success rate is roughly flat across risk tiers (76–83%) and priority tiers (74–84%) — risk/priority labels aren't strongly predictive of outcome on their own; task type (cluster) is a much sharper signal.

Methodology note

sklearn/pandas/matplotlib initially reported as available in this sandbox have no PyPy wheels here and aren't actually installed; scikit-learn/pandas could not be installed in time (no compiled wheel, would require a slow source build). This run instead uses a hand-rolled TF-IDF + spherical k-means (numpy only) over cleaned PR title+body text, and matplotlib was installed successfully for the two charts. Results are schema-compatible with prior runs cached in last-clustering-result.json / clustering/history.jsonl, so the trend line below is apples-to-apples with previous analyses.

Cluster Analysis

Cluster 0 — docs / test / spec / step / content / coverage

• Size: 162 tasks (16.2%) · Success rate: 76.5%
• Characteristics: Documentation-site compliance (llms.txt/GEO spec), executable spec coverage additions, Playwright doc tests.
• Example PRs: Add executable VAL-P1–VAL-P15 coverage for security architecture validation #46457 (closed), feat(docs): update llms.txt to GEO-compliant spec with full docs site index #49177 (closed), docs: update llms.txt with documentation index and add Playwright test #46287 (closed) — docs-site compliance tasks skew toward closed, suggesting spec-alignment PRs get re-scoped or superseded rather than merged as-is.

Cluster 1 — daily / workflows / report / issue / copilot / agent

• Size: 140 tasks (14.0%) · Success rate: 64.3% (2nd lowest)
• Characteristics: Cross-cutting "normalize report formatting/headers across N workflows" sweeps and eval-enablement batches.
• Example PRs: Normalize report header guidance across non-compliant reporting workflows #45703 (closed, header-normalization sweep), feat: enable evals in 64 agentic workflows #46774 (merged, enable evals in 64 workflows), [WIP] Normalize report formatting for non-compliant workflows #47329/[WIP] Fix normalization of report formatting for non-compliant workflows #46896 (closed, repeat WIP attempts at the same normalization task) — repeated closed WIP attempts at the same underlying task point to this being a genuinely hard category for the agent, not noise.

Cluster 4 — pkg / cli / linter / analyzer / helpers

• Size: 136 tasks (13.6%) · Success rate: 81.6%
• Characteristics: Targeted Go lint-finding remediation, package-level refactors, call-site migrations in pkg/cli.

Cluster 7 — model / copilot / claude / engine / models / awf

• Size: 100 tasks (10.0%) · Success rate: 72.0%
• Characteristics: Model routing/pinning/fallback logic (Copilot model: auto, GPT pricing catalog, default model fallback).

Cluster 2 — safe / pull request / safe outputs

• Size: 99 tasks (9.9%) · Success rate: 79.8%
• Characteristics: safe-outputs handler hardening — permission scoping, fork-backed PR support, branch-update edge cases.

Cluster 9 — image / container / version / scan / digest / pins

• Size: 93 tasks (9.3%) · Success rate: 58.1% (lowest)
• Characteristics: Reactive container-image vulnerability-scan remediation (pin/patch/digest bumps for MCP server images).
• Example PRs: Block scan remediation until patched gh-aw-mcpg image is published #49797 (closed, blocked on upstream image), Pin Serena MCP image to remediated SHA tag and propagate digest across workflow artifacts #49693 (merged, digest pin + propagation), [WIP] Fix vulnerabilities detected in container image a3d33153b6ab #47833 (closed, WIP vuln fix) — many of these depend on an upstream vendor publishing a patched image, which the agent can't control, explaining the low close rate.

Cluster 6 — error / core / cjs / require / eslint / string

• Size: 90 tasks (9.0%) · Success rate: 86.7% (2nd highest)
• Characteristics: ESLint rule authoring/tightening for error-handling conventions in setup/js.

Cluster 8 — mcp / tool / tools / context / list / gateway

• Size: 81 tasks (8.1%) · Success rate: 84.0%
• Characteristics: MCP gateway/tool-call reliability fixes (timeout handling, pagination docs).

Cluster 3 — sous-chef / aic / workflow engine plumbing

• Size: 67 tasks (6.7%) · Success rate: 86.6% (highest)
• Characteristics: Small, well-scoped engine/CI plumbing fixes (runs-on defaults, model-name normalization, sandbox flags).

Cluster 5 — job / actions / failing / integration / failure

• Size: 32 tasks (3.2%) · Success rate: 65.6%
• Characteristics: "Fix failing GitHub Actions job" auto-triggered tasks — smallest cluster, mid-low success.

Success Rate by Cluster

Cluster	Label	Tasks	Success Rate
0	docs / test / spec / coverage	162	76%
1	daily / workflows / report normalization	140	64%
4	pkg / cli / linter / analyzer	136	82%
7	model / copilot / claude / engine	100	72%
2	safe outputs / pull request	99	80%
9	image / container / scan / digest	93	58%
6	error / eslint / cjs / require	90	87%
8	mcp / tool / gateway	81	84%
3	sous-chef / engine plumbing	67	87%
5	failing Actions job triage	32	66%

Success Rate by PR Type (label-derived)

Type	Tasks	Merged	Closed	Success Rate
bug	343	270	69	79%
feature	193	144	49	75%
other	185	139	46	75%
chore	106	76	30	72%
refactor	85	56	27	66%
docs	56	44	12	79%
test	24	20	4	83%
ci	3	3	0	100%
formatting	3	2	1	67%
perf	2	1	1	50%

Success Rate by Risk / Priority Label

Risk	Tasks	Success Rate		Priority	Tasks	Success Rate
low	172	81%		medium	163	84%
medium	154	83%		low	114	74%
high	41	76%		high	88	84%
—				critical	2	100%

(Risk/priority coverage is partial — only ~37% of PRs carry these labels; treat as directional.)

Key Findings

1. Container-image vulnerability remediation (cluster 9) is the weakest task category at 58% success, well below the 75.5% average — and several closed PRs there are blocked on an upstream vendor image, not agent quality. This category may warrant excluding upstream-blocked tasks from success-rate scoring, or routing them to a "wait for upstream" state instead of closing.
2. Cross-workflow "normalize X across N workflows" sweeps (cluster 1) show repeat WIP attempts at the same task (Normalize report header guidance across non-compliant reporting workflows #45703 → [WIP] Normalize report formatting for non-compliant workflows #47329 → [WIP] Fix normalization of report formatting for non-compliant workflows #46896, all closed) — this is a task shape the agent struggles to land in one pass, likely because success requires touching many files consistently rather than a single localized fix.
3. Refactors are the weakest PR-type label (66%) despite being lower risk on paper — refactor-labeled PRs may be getting closed more often due to scope creep or conflicting with concurrent changes rather than technical difficulty.
4. Overall success rate has declined ~5 points over the trailing six weeks (82% peak on 07-01 → 75.5% today), a steady downward trend rather than a single-day blip — worth investigating whether task mix shifted toward harder categories (more cluster-9/cluster-1-style work) or whether agent/prompt quality regressed.
5. Narrowly-scoped engine/plumbing fixes (clusters 3, 6, 8) consistently outperform (84–87%), reinforcing that small, well-bounded diffs remain the agent's strongest suit.

Recommendations

1. Investigate the success-rate decline directly: pull PR review comments/close reasons for a sample of the ~240 closed PRs from the last two weeks to distinguish "agent produced a bad diff" from "task was superseded/blocked/duplicated" — the container-scan cluster suggests at least part of the drop is structural, not quality.
2. Split "normalize across N workflows" tasks (cluster 1) into smaller per-workflow or per-batch PRs rather than one sweep — the repeated-WIP pattern suggests the current task granularity is too large for reliable one-shot completion.
3. For container-scan remediation (cluster 9), gate task creation on upstream image availability where possible, or track these separately from the main success-rate metric so a single unresponsive upstream vendor doesn't drag down the aggregate.
4. Double down on the task shapes that already work well (cluster 3/6/8: small, localized engine/lint/MCP fixes) when deciding what to route to the copilot agent vs. a human, since these show the most reliable 85%+ outcomes.

---

Generated by Prompt Clustering Analysis (Run: 30807250685)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 162.5 AIC · ⌖ 36 AIC · ⊞ 6K · ◷

• expires on Aug 4, 2026, 3:25 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #50233.