[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-15 (998 PRs, 12 clusters)

[github-actions[bot]]

Summary

Analysis Period: Last 30 days · PRs created 2026-06-25 → 2026-07-15
Total Tasks Analyzed: 998 (Copilot coding-agent PRs)
Clusters Identified: 12
Overall Success Rate (merged / decided): 78% (▼ -2.0 pts vs 2026-07-10)
Outcomes: 771 merged · 218 closed · 9 open
Clustering quality: silhouette 0.070 (typical for short, overlapping task text)

Success-rate trend (last 8 runs): ▂▂█▄▂▅▁▂ 80% → 80%

Full Analysis Report

General Insights

• Most common task type: Agent runs, model & failure triage — 143 PRs (14.3%)
• Highest success rate: ESLint rules & linters — 91% (78 PRs)
• Lowest success rate: Version bumps & CLI/Playwright — 51% (38 PRs) — closes at 49%
• 7/12 clusters beat the overall 78% success rate

Cluster Analysis

1. Agent runs, model & failure triage

• Size: 143 tasks (14.3%)
• Success Rate: 70% (100 merged / 42 closed / 1 open)
• Top terms: copilot, failure, model, agent, workflow, failures
• Representative PRs: #42869, #45599, #43555
  • e.g. Fail fast on Copilot SDK model-policy errors and repin Smoke Copilot Sub Agents #42869: “Fail fast on Copilot SDK model-policy errors and repin Smoke Copilot Sub Agents” (closed)

2. Workflow prompts & reporting

• Size: 142 tasks (14.2%)
• Success Rate: 74% (104 merged / 37 closed / 1 open)
• Top terms: workflow, workflows, prompt, report, daily, issue
• Representative PRs: #45703, #41728, #43556
  • e.g. Normalize report header guidance across non-compliant reporting workflows #45703: “Normalize report header guidance across non-compliant reporting workflows” (open)

3. Sous-chef / AIC usage tracking

• Size: 123 tasks (12.3%)
• Success Rate: 88% (108 merged / 15 closed)
• Top terms: aic, sub, sous, chef, sous chef, generated sous
• Representative PRs: #44015, #44995, #43716
  • e.g. Expose memory stores to on.steps in pre-activation for deterministic dispatch gating #44015: “Expose memory stores to on.steps in pre-activation for deterministic dispatch gating” (merged)

4. Docs, schema & engine reference

• Size: 101 tasks (10.1%)
• Success Rate: 68% (69 merged / 32 closed)
• Top terms: docs, reference, engine, schema, section, field
• Representative PRs: #45451, #42294, #45420
  • e.g. Add Antigravity to workflow schema and engine docs #45451: “Add Antigravity to workflow schema and engine docs” (closed)

5. Skills & workflow frontmatter

• Size: 97 tasks (9.7%)
• Success Rate: 82% (79 merged / 17 closed / 1 open)
• Top terms: skill, workflow, skills, frontmatter, firewall, domains
• Representative PRs: #44796, #44173, #41945
  • e.g. feat: add sandbox.agent.runtime: gvisor for gVisor container runtime support #44796: “feat: add sandbox.agent.runtime: gvisor for gVisor container runtime support” (merged)

6. Refactoring & shared helpers

• Size: 85 tasks (8.5%)
• Success Rate: 85% (71 merged / 13 closed / 1 open)
• Top terms: refactor, helpers, shared, logic, behavior, helper
• Representative PRs: #44313, #43649, #42868
  • e.g. Refactor safe-outputs config parsing into focused workflow modules #44313: “Refactor safe-outputs config parsing into focused workflow modules” (merged)

7. ESLint rules & linters

• Size: 78 tasks (7.8%)
• Success Rate: 91% (68 merged / 7 closed / 3 open)
• Top terms: rule, eslint-factory, eslint, lint, suggestion, flagged
• Representative PRs: #42057, #42223, #42042
  • e.g. eslint-factory: add no-unsafe-catch-error-property rule #42057: “eslint-factory: add no-unsafe-catch-error-property rule” (merged)

8. Compiler & integration wiring

• Size: 55 tasks (5.5%)
• Success Rate: 81% (44 merged / 10 closed / 1 open)
• Top terms: import, compile, call, integration, jobs, every
• Representative PRs: #43323, #45647, #44996
  • e.g. fix: remove unused context import breaking CI builds in parser integration tests #43323: “fix: remove unused context import breaking CI builds in parser integration tests” (merged)

9. Test coverage & specs

• Size: 48 tasks (4.8%)
• Success Rate: 85% (41 merged / 7 closed)
• Top terms: spec, coverage, missing, formal, compliance, gaps
• Representative PRs: #43594, #44358, #45106
  • e.g. SPDD 2026-07-05: outcome evaluators, policy compiler, and spec coverage for batch 20-24 #43594: “SPDD 2026-07-05: outcome evaluators, policy compiler, and spec coverage for batch 20-24” (merged)

10. Workflow prompts & reporting

• Size: 47 tasks (4.7%)
• Success Rate: 81% (38 merged / 9 closed)
• Top terms: job, github actions, actions, failing, github, workflow
• Representative PRs: #45293, #43733, #42113
  • e.g. [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #45293: “[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2” (merged)

11. Dashboard & VS Code extension

• Size: 41 tasks (4.1%)
• Success Rate: 73% (30 merged / 11 closed)
• Top terms: dashboard, extension, runs, state, usage, experiment
• Representative PRs: #42297, #42270, #42213
  • e.g. Add maintenance view to the agentic workflows dashboard #42297: “Add maintenance view to the agentic workflows dashboard” (merged)

12. Version bumps & CLI/Playwright

• Size: 38 tasks (3.8%)
• Success Rate: 51% (19 merged / 18 closed / 1 open)
• Top terms: bump, version, playwright, cli, golden, release
• Representative PRs: #45215, #42569, #43196
  • e.g. Bump pinned agentic CLI/MCP tool versions for 2026-07-13 #45215: “Bump pinned agentic CLI/MCP tool versions for 2026-07-13” (merged)

Success Rate by Cluster

Cluster	Tasks	%Total	Merged	Closed	Success
Agent runs, model & failure triage	143	14.3%	100	42	70%
Workflow prompts & reporting	142	14.2%	104	37	74%
Sous-chef / AIC usage tracking	123	12.3%	108	15	88%
Docs, schema & engine reference	101	10.1%	69	32	68%
Skills & workflow frontmatter	97	9.7%	79	17	82%
Refactoring & shared helpers	85	8.5%	71	13	85%
ESLint rules & linters	78	7.8%	68	7	91%
Compiler & integration wiring	55	5.5%	44	10	81%
Test coverage & specs	48	4.8%	41	7	85%
Workflow prompts & reporting	47	4.7%	38	9	81%
Dashboard & VS Code extension	41	4.1%	30	11	73%
Version bumps & CLI/Playwright	38	3.8%	19	18	51%

Representative PRs by Cluster

PR #	Cluster	Outcome	Title
#42869	Agent runs, model & failure triage	closed	Fail fast on Copilot SDK model-policy errors and repin Smoke Copilot S
#45599	Agent runs, model & failure triage	merged	Suppress spurious report_incomplete failures after successful Copilo
#45703	Workflow prompts & reporting	open	Normalize report header guidance across non-compliant reporting workfl
#41728	Workflow prompts & reporting	merged	Normalize report-format guidance across non-compliant agentic workflow
#44015	Sous-chef / AIC usage tracking	merged	Expose memory stores to on.steps in pre-activation for deterministic
#44995	Sous-chef / AIC usage tracking	merged	perf: fix CompileMemoryUsage regression by optimizing normalizeBlankLi
#45451	Docs, schema & engine reference	closed	Add Antigravity to workflow schema and engine docs
#42294	Docs, schema & engine reference	closed	Document Antigravity engine support in reference docs and schema guida
#44796	Skills & workflow frontmatter	merged	feat: add sandbox.agent.runtime: gvisor for gVisor container runtime s
#44173	Skills & workflow frontmatter	merged	Bump AWF to v0.27.27 and MCPG to v0.4.0
#44313	Refactoring & shared helpers	merged	Refactor safe-outputs config parsing into focused workflow modules
#43649	Refactoring & shared helpers	merged	refactor(linters): consolidate duplicated AST/context helpers into int
#42057	ESLint rules & linters	merged	eslint-factory: add no-unsafe-catch-error-property rule
#42223	ESLint rules & linters	merged	eslint-factory: suggestions for .stack/.code and flag computed string-
#43323	Compiler & integration wiring	merged	fix: remove unused context import breaking CI builds in parser integ
#45647	Compiler & integration wiring	open	fix: reject submit_pull_request_review max > 1 at compile time
#43594	Test coverage & specs	merged	SPDD 2026-07-05: outcome evaluators, policy compiler, and spec coverag
#44358	Test coverage & specs	merged	SPDD 2026-07-08: close spec sync gaps and enforce add-flow rollback se
#45293	Workflow prompts & reporting	merged	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2
#43733	Workflow prompts & reporting	merged	[WIP] Fix failing GitHub Actions job: Integration Workflow Misc Part 2
#42297	Dashboard & VS Code extension	merged	Add maintenance view to the agentic workflows dashboard
#42270	Dashboard & VS Code extension	merged	Detect missing gh aw in dashboard and surface install guidance
#45215	Version bumps & CLI/Playwright	merged	Bump pinned agentic CLI/MCP tool versions for 2026-07-13
#42569	Version bumps & CLI/Playwright	closed	Bump default Claude Code and Playwright MCP versions; regenerate lockf

Key Findings

1. Work is concentrated in workflow/agent plumbing. The two largest clusters — Agent runs, model & failure triage (143) and Workflow prompts & reporting (142) — together account for 285 PRs (28% of all tasks), and both sit at or below the overall success rate.
2. Narrow, well-specified tasks convert best. ESLint rules & linters leads at 91%; mechanical, single-purpose work (lint rules, coverage, usage tracking) merges more reliably than open-ended triage.
3. Version bumps & CLI/Playwright is the weak spot at 51% — nearly 49% of these PRs are closed unmerged, suggesting churn or superseded attempts in that area.
4. Success rate is stable around 78%, consistent with the 8-run history (range 79%–82%).

Recommendations

1. Tighten prompts for “Version bumps & CLI/Playwright”. Its 49% close rate signals ambiguous or overlapping tasks — add explicit acceptance criteria and dedupe against in-flight PRs before dispatch.
2. Replicate the “ESLint rules & linters” pattern. High-conversion clusters share tight scope and clear validation steps; template that structure for the larger triage clusters.
3. Watch the two mega-clusters (Agent runs, model & failure triage, Workflow prompts & reporting). At 28% of volume, small success-rate gains here dominate overall throughput — consider splitting them into finer sub-tasks with per-task success gates.
4. Keep enriching with turn/duration metrics. Workflow-log matching (turns, cost) was unavailable this run; wiring aw_info.json turn counts to PR numbers would let us correlate iteration count with cluster success.

Methodology & limitations

• Corpus: 998 bot-authored PRs (990 copilot-swe-agent, plus a few github-actions) created in the last 30 days.
• Text: PR title (weighted ×2) + body, with gh-aw HTML trackers, code blocks, URLs and tables stripped.
• Vectorization: hand-rolled TF-IDF (unigrams+bigrams, min_df=6, top 450 features, sublinear TF, L2-normalized) — no numpy/sklearn available in this sandbox.
• Clustering: spherical (cosine) K-means, k selected from {8..12} by approximate silhouette → k=12.
• Success rate = merged / (merged + closed); open PRs excluded from the denominator.
• Limitations: PR summary bodies (agent's self-description) are used as prompt proxies; the full-PR cache was stale (different PR range) so comments/reviews/file-churn and workflow turn counts are not incorporated this run.

References: §29407669494

Generated by Prompt Clustering Analysis (Run: 29407669494)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 143.8 AIC · ⌖ 42.9 AIC · ⊞ 13.2K · ◷

• expires on Jul 16, 2026, 2:46 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #45974.