[prompt-clustering] Copilot Agent Prompt Clustering — Daily Report (2026-07-25)

[github-actions[bot]]

Summary

Analysis window: last 30 days (2026-07-04 → 2026-07-25) · PRs analyzed: 1,000 copilot-authored PRs · Clusters: 10 (chosen by silhouette) · Overall merge-success rate: 76.4% (762 merged / 235 closed / 3 open)

The merge-success rate is essentially flat vs. the last two runs (76.4% → 76.3% → 76.4%) but continues a slow ~3–4 pt drift down from the 80% seen in mid-June. The story this window is not the average — it's two small, highly automated clusters that keep churning out unmergeable work, and a strong [WIP]-title signal.

Key findings

1. [WIP]-titled PRs almost never land — 22% vs 80%. 59 PRs carry a WIP marker; only 13 merged (22.0%). Non-WIP PRs merge at 79.9%. [WIP] is by far the strongest single predictor of closure in the dataset — a 3.6× gap.
2. The docs-site robots.txt / llms.txt task is stuck in a loop. Cluster txt / docs / site (n=19) merges at 15.8% (3/19). The closed PRs are near-duplicates re-attempting the same "add robots.txt / llms.txt for AI crawlers" goal ([WIP] Add robots.txt with Allow rules for AI bots #47829, docs: confirm robots.txt exists for AI bot crawling on docs site #47642, [WIP] Add robots.txt with AI bot Allow rules to docs site #47370, [WIP] Add robots.txt and llms.txt to docs site #46893, [WIP] Add robots.txt with AI bot Allow rules to docs site #46055, [WIP] Add robots.txt to docs site for AI crawler indexing #45792 ...) across many days. The agent keeps re-proposing a change that reviewers keep closing — wasted runs, not a hard task.
3. Container-scan / security-alert triage workflows underperform — 29%. Cluster issue / gh aw / workflow (n=35, 29.4%) is dominated by [WIP] container-image-scan triage and list_code_scanning_alerts bounding PRs ([WIP] Fix vulnerabilities detected in container image a3d33153b6ab #47833, [WIP] Fix container image scan findings for 0d35e8682845 #47518, [WIP] Fix container scan findings for image tag 0.27.37 #47481, [WIP] Investigate and fix container scan operational failure #47424). These automated security-triage runs open PRs that mostly don't merge.
4. Highest-yield work is small, targeted fixes. The two largest clusters — maintenance/security fixes ("sous-chef", n=170, 88%) and core Go validation/field fixes (n=165, 84%) — carry most volume and the best success. Merged PRs are only marginally bigger than closed ones (median 4 vs 3 files), so size is not the failure driver — task framing is.
5. Version/dependency bumps are large and land less often — 63%. Cluster claude / version / bump (n=55) averages ~108 changed files (lockfiles) and merges at 63%, the weakest of the high-volume clusters.

Cluster breakdown (k=10, TF-IDF + K-means, silhouette 0.065)

Labels are the top TF-IDF terms per cluster; the plain-English reading follows in the notes column.

Cluster (top terms)	Size	Dom. type	Success	Avg files	Reading
pr sous / sous chef	170	fix	88%	25.0	"sous-chef" maintenance + security fixes (e.g. #47957 git-injection fix) — best yield
string / fix / fields	165	fix	84%	21.8	Core Go fixes: validation, field/string handling, SHA integrity
job / start / coding	158	fix	76%	21.1	Linter/CI enforcement & copilot-coding-agent tasks
workshop / helpers / path	140	other	81%	21.2	Refactors & shared-helper consolidation
workflow / workflows / prompt	132	other	68%	14.4	Agentic-workflow authoring & prompt/guidance tuning — below avg
spec / docs / table	110	docs	79%	16.1	Specs, docs tables & test-quality audits
claude / version / bump	55	chore	63%	108.5	Dependency/version bumps (AWF, MCP) & security triage — large diffs
issue / gh aw / workflow	35	fix	29%	10.9	Container-scan / code-scanning-alert triage workflows — outlier
txt / docs / site	19	feat	15%	2.2	docs-site robots.txt / llms.txt (GEO) — stuck loop
actions / failing github	16	fix	75%	5.4	"Fix failing GitHub Actions job" tasks

Success rate by conventional-commit type

Type inferred from PR title prefix / verbs.

Type	Count	Success
fix	296	79%
other	223	82%
feat	218	69%
refactor	74	77%
docs	73	75%
ci	44	81%
chore	36	55%
test	27	77%
perf	9	55%

feat (new capability) lands ~10 pts below fix, and chore/perf are the weakest — consistent with "small, well-scoped fixes land; broad or open-ended asks don't."

Stuck-task sample — the robots.txt / llms.txt loop (all closed)

PR	Title
#47829	[WIP] Add robots.txt with Allow rules for AI bots
#47642	docs: confirm robots.txt exists for AI bot crawling on docs site
#47370	[WIP] Add robots.txt with AI bot Allow rules to docs site
#46893	[WIP] Add robots.txt and llms.txt to docs site
#46055	[WIP] Add robots.txt with AI bot Allow rules to docs site
#45792	[WIP] Add robots.txt to docs site for AI crawler indexing
#44599	[geo-optimizer] Strengthen docs-site LLM discovery signals (/llms.txt)

Same goal, re-proposed repeatedly, consistently closed. This is the single clearest example of an automated workflow burning runs on an unmergeable objective.

Recommendations

1. Retire or gate the docs-site robots.txt / GEO workflow. It has re-attempted the same change dozens of times at ~16% success. Either the change is already done (close the loop with a guard that detects the file exists) or it's a policy "no" — encode that decision so the agent stops re-opening it.
2. Treat [WIP] as a non-terminal state, not a PR. With 22% merge success, [WIP] PRs are mostly the agent surfacing incomplete work as a PR. Prefer draft PRs or an internal checkpoint so the queue isn't dominated by work that isn't review-ready.
3. Add a merge-gate/dedup for the container-scan triage workflow (29%). Bound how often it can open a PR for the same image digest, and require a concrete diff before opening — many of its PRs are investigation stubs.
4. Keep leaning into small, targeted fixes. The 88%/84% clusters show the agent is strongest on scoped fix/validation work. feat and large chore bumps (63–69%) warrant tighter acceptance criteria in the prompt.

Methodology & limitations

• **(redacted) 1,000 copilot-authored PRs from the last 30 days. Outcome/state taken from the fresh search summary (copilot-prs.json); PR bodies + file/commit counts enriched from cached full-PR snapshots. Note: the cached full-PR snapshots were stale for ~165 PRs (showed OPEN where the PR had since merged/closed); the authoritative search state was used for all success-rate math.
• Clustering: TF-IDF (1–2 grams, min_df=3, English + domain stopwords) → K-means; k=10 selected by silhouette over k∈[4,12] (best 0.065). Text-clustering cohesion is inherently low here — clusters are thematic groupings, not crisp partitions.
• Workflow turn/duration/cost metrics were unavailable: the only aw_info.json logs on disk are for gh-aw's own daily workflows (turns/cost null) and don't map to copilot coding-agent PRs. "Turns" is therefore proxied by commit count where noted; no per-PR turn/cost enrichment was possible this run.
• Trend chart uses 17 prior daily runs cached in clustering/history.jsonl.

References: §30154092413

Generated by 📊 Copilot Agent Prompt Clustering Analysis · age00 · 217.3 AIC · ⌖ 35.1 AIC · ⊞ 13.1K · ◷

• expires on Jul 26, 2026, 2:42 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-26T10:42:53.238Z.

Closed by Workflow