[prompt-clustering] Copilot Agent Prompt Clustering — 7 task themes across 1,000 PRs (Jun 27–Jul 17)

[github-actions[bot]]

Summary

• Analysis window: last 30 days (2026-06-27 → 2026-07-17)
• Copilot PRs analyzed: 1000 (100% had full PR data)
• Clusters identified: 7 (selected by peak silhouette over k=4–7)
• Overall merge/success rate: 76% (756 merged, 224 closed, 20 open)
• Median-scale activity per PR: 2.6 reviews · 4.0 commits · 25 files touched

Task distribution by cluster

1. ESLint / linter-rule engineering   ████████████████████████████  219 (21.9%)
2. Daily agentic workflows, dashboard ████████████████████          155 (15.5%)
3. Sous-chef self-improvement & AIC a ███████████████████           145 (14.5%)
4. Safe-outputs & pull-request infras █████████████████             136 (13.6%)
5. CLI / model version bumps & consta █████████████████             132 (13.2%)
6. Workflow prompt & report-style nor ████████████████              124 (12.4%)
7. Docs site & reference content      ███████████                    89 ( 8.9%)

Merge/success rate by cluster

3. Sous-chef self-improvement & AIC a ████████████████████████      86%
1. ESLint / linter-rule engineering   ███████████████████████       81%
2. Daily agentic workflows, dashboard █████████████████████         77%
4. Safe-outputs & pull-request infras █████████████████████         74%
6. Workflow prompt & report-style nor ████████████████████          72%
5. CLI / model version bumps & consta ███████████████████           67%
7. Docs site & reference content      ███████████████████           66%
   overall avg                        █████████████████████         76%

Full clustering analysis

Methodology

• Corpus: PR title (weighted ×3) + body, stripped of code blocks, URLs, HTML, and copilot firewall-warning boilerplate.
• Features: TF-IDF over unigrams + bigrams, sublinear TF, min_df=4, max_df=0.55, 800-term vocabulary.
• Clustering: spherical K-means (cosine) with k-means++ init, seed=42; k chosen by sampled silhouette.
• Silhouette by k: k=4: 0.0349, k=5: 0.0389, k=6: 0.0395, k=7: 0.0458 → k=7.
• Environment note: the runner shipped PyPy without numpy/scikit-learn/matplotlib and PyPI installs were blocked, so TF-IDF + K-means were reimplemented in pure Python and charts are rendered as text bars.
• Workflow-metric note: per-run turn count / duration / cost could not be joined — the gh-aw logs tooling was not available in this run. PR-level proxies (reviews, commits, files, diff size) are used for complexity instead.

Cluster detail

Cluster 1: ESLint / linter-rule engineering

• Size: 219 tasks (21.9%) · Success rate: 81% (177/219 merged)
• Avg complexity: 10 files · +252/-110 lines · 2.8 reviews · 3.7 commits
• Top terms: pkg, eslint, catch, string, eslint factory, factory, linters, require
• What it is: Adding and consolidating custom ESLint rules (e.g. eslint-factory, fs-sync try/catch rules) and small refactors in pkg.
• Representative PRs:
  • #45709 — [eslint-miner] eslint: add require-mkdirsync-try-catch rule (merged)
  • #43951 — [WIP] Fix expressionReferencesCatchVar to handle LogicalExpression (closed)
  • #45964 — eslint-factory: consolidate fs-sync try/catch rules and unify ESM/CJS fs binding detection (merged)
  • #42057 — eslint-factory: add no-unsafe-catch-error-property rule (merged)

Cluster 2: Daily agentic workflows, dashboards & eval reports

• Size: 155 tasks (15.5%) · Success rate: 77% (119/155 merged)
• Avg complexity: 16 files · +373/-133 lines · 2.6 reviews · 3.6 commits
• Top terms: daily, agent, run, dashboard, output, runs, workflow, step
• What it is: Building or fixing daily scheduled agentic workflows — dashboards, run reports, evals, and safe-output tracking.
• Representative PRs:
  • #45805 — feat: add daily-evals-report workflow (merged)
  • #45623 — fix: restore safe-output actuation tracking and reconcile 0% SafeItemsCount (merged)
  • #45820 — optimize(detection-analysis-report): ~45 AIC/run savings via prompt compaction and script extraction (merged)
  • #41960 — feat(ambient-context): trim prompt size in daily-code-metrics, test-quality-sentinel, smoke-copilot-aoai-entra (merged)

Cluster 3: Sous-chef self-improvement & AIC automation

• Size: 145 tasks (14.5%) · Success rate: 86% (124/145 merged)
• Avg complexity: 20 files · +324/-144 lines · 3.9 reviews · 6.1 commits
• Top terms: sous, chef, sous chef, aic, generated sous, chef aic, souschef, aic aic
• What it is: Self-improvement tasks driven by the sous-chef/AIC loop: security hardening, content redaction, memory stores. Heaviest review + commit activity.
• Representative PRs:
  • #43716 — fix(security): replace curl|sh installer pipes with download+verify (RGS-018) (merged)
  • #44501 — feat: content redaction for public-facing safe-outputs (closed)
  • #43700 — compiler: add evals job for BinEval binary evaluations (merged)
  • #44015 — Expose memory stores to on.steps in pre-activation for deterministic dispatch gating (merged)

Cluster 4: Safe-outputs & pull-request infrastructure

• Size: 136 tasks (13.6%) · Success rate: 74% (100/136 merged)
• Avg complexity: 24 files · +344/-3668 lines · 3.4 reviews · 4.7 commits
• Top terms: safe, outputs, pull, safe outputs, pull request, request, compiler, workflow
• What it is: Compiler and safe-output plumbing: fork-backed PR support, scope pre-flight checks, activation, cross-repo credentials.
• Representative PRs:
  • #45909 — Add fork-backed safe-output pull request support (merged)
  • #42585 — safe-outputs: pre-flight workflow scope check + full-branch allowed_files validation (merged)
  • #43145 — Clarify checkout, safe-outputs, and cross-repo credential precedence (closed)
  • #45826 — fix: add maxLength: 65536 to body/content fields to exempt from SM-IS-01 10 KB cap (merged)

Cluster 5: CLI / model version bumps & constants

• Size: 132 tasks (13.2%) · Success rate: 67% (88/132 merged)
• Avg complexity: 80 files · +1064/-892 lines · 2.3 reviews · 3.8 commits
• Top terms: version, bump, copilot, model, awf, cli, constants, claude
• What it is: Routine dependency and pinned-tool version bumps (Claude/Copilot/SDK/firewall) and constants — the largest file footprints.
• Representative PRs:
  • #45215 — Bump pinned agentic CLI/MCP tool versions for 2026-07-13 (merged)
  • #46158 — Update pinned CLI tool versions for 2026-07-17 (open)
  • #44467 — chore: bump gh-aw-firewall to v0.27.28 (merged)
  • #42952 — Bump CLI versions: Claude 2.1.198, Copilot 1.0.68, SDK 1.0.5, Codex 0.142.5, Pi 0.80.3 (merged)

Cluster 6: Workflow prompt & report-style normalization

• Size: 124 tasks (12.4%) · Success rate: 72% (89/124 merged)
• Avg complexity: 19 files · +232/-187 lines · 1.2 reviews · 2.5 commits
• Top terms: workflows, workflow, github, prompt, github workflows, lock, yml, guidance
• What it is: Normalizing report formatting, prompt guidance and lock.yml across many workflow files; broad, low-review edits.
• Representative PRs:
  • #42097 — [workflow-style] Normalize report formatting guidance across reporting workflows (merged)
  • #45703 — Normalize report header guidance across non-compliant reporting workflows (closed)
  • #42324 — Normalize report-style prompts in Priority 1 workflows (h3+ headers + <details> guidance) (merged)
  • #42520 — Enable threat detection on token audit/report workflows (merged)

Cluster 7: Docs site & reference content

• Size: 89 tasks (8.9%) · Success rate: 66% (59/89 merged)
• Avg complexity: 12 files · +5603/-5487 lines · 1.6 reviews · 2.9 commits
• Top terms: docs, reference, docs src, content, engine, src, content docs, src content
• What it is: Documentation-site and reference content: frontmatter reference, engine docs, README/site updates (large generated diffs).
• Representative PRs:
  • #44364 — Add docs-site robots.txt allowlist and generate llms.txt from docs content (closed)
  • #45420 — docs: add engine.driver to frontmatter reference (merged)
  • #45451 — Add Antigravity to workflow schema and engine docs (closed)
  • #45491 — docs: add workshop callout to README and docs (merged)

Success & complexity by cluster

#	Theme	Tasks	Success	Avg files	Avg +/-	Reviews	Commits
1	ESLint / linter-rule engineering	219	81%	10	+252/-110	2.8	3.7
2	Daily agentic workflows, dashboards & eval reports	155	77%	16	+373/-133	2.6	3.6
3	Sous-chef self-improvement & AIC automation	145	86%	20	+324/-144	3.9	6.1
4	Safe-outputs & pull-request infrastructure	136	74%	24	+344/-3668	3.4	4.7
5	CLI / model version bumps & constants	132	67%	80	+1064/-892	2.3	3.8
6	Workflow prompt & report-style normalization	124	72%	19	+232/-187	1.2	2.5
7	Docs site & reference content	89	66%	12	+5603/-5487	1.6	2.9

Sample task table

A representative slice (up to 4 highest-centroid PRs per cluster; full set is 1000 PRs).

PR	Title	Cluster	Outcome
#45709	[eslint-miner] eslint: add require-mkdirsync-try-catch rule	1	merged
#43951	[WIP] Fix expressionReferencesCatchVar to handle LogicalExpressi...	1	closed
#45964	eslint-factory: consolidate fs-sync try/catch rules and unify ES...	1	merged
#42057	eslint-factory: add no-unsafe-catch-error-property rule	1	merged
#45805	feat: add daily-evals-report workflow	2	merged
#45623	fix: restore safe-output actuation tracking and reconcile 0% Saf...	2	merged
#45820	optimize(detection-analysis-report): ~45 AIC/run savings via pro...	2	merged
#41960	feat(ambient-context): trim prompt size in daily-code-metrics, t...	2	merged
#43716	fix(security): replace curl|sh installer pipes with download+ve...	3	merged
#44501	feat: content redaction for public-facing safe-outputs	3	closed
#43700	compiler: add evals job for BinEval binary evaluations	3	merged
#44015	Expose memory stores to on.steps in pre-activation for determi...	3	merged
#45909	Add fork-backed safe-output pull request support	4	merged
#42585	safe-outputs: pre-flight workflow scope check + full-branch allo...	4	merged
#43145	Clarify checkout, safe-outputs, and cross-repo credential preced...	4	closed
#45826	fix: add maxLength: 65536 to body/content fields to exempt from ...	4	merged
#45215	Bump pinned agentic CLI/MCP tool versions for 2026-07-13	5	merged
#46158	Update pinned CLI tool versions for 2026-07-17	5	open
#44467	chore: bump gh-aw-firewall to v0.27.28	5	merged
#42952	Bump CLI versions: Claude 2.1.198, Copilot 1.0.68, SDK 1.0.5, Co...	5	merged
#42097	[workflow-style] Normalize report formatting guidance across rep...	6	merged
#45703	Normalize report header guidance across non-compliant reporting ...	6	closed
#42324	Normalize report-style prompts in Priority 1 workflows (h3+ head...	6	merged
#42520	Enable threat detection on token audit/report workflows	6	merged
#44364	Add docs-site robots.txt allowlist and generate llms.txt from do...	7	closed
#45420	docs: add engine.driver to frontmatter reference	7	merged
#45451	Add Antigravity to workflow schema and engine docs	7	closed
#45491	docs: add workshop callout to README and docs	7	merged

Outcome distribution

merged   ████████████████████████████  756 (76%)
closed   ████████                      224 (22%)
open     █                              20 (2%)

Key findings

1. Linter-rule work dominates volume. "ESLint / linter-rule engineering" is the single largest cluster (219 PRs, 21.9%) and lands above average at 81% — small, well-scoped pkg/eslint edits are the agent's bread and butter.
2. Self-improvement tasks are the most reliable. "Sous-chef self-improvement & AIC automation" tops success at 86%, but also carries the heaviest human review load (3.9 reviews, 6.1 commits) — high success is bought with iteration.
3. Version bumps are large but fragile. "CLI / model version bumps & constants" touches the most files (80/PR) yet only merges 67% — broad golden-file/version churn is easy to open but often superseded or closed.
4. Docs & version clusters trail on merge rate (66–67%), suggesting these generated/broad-diff tasks are more prone to being closed unmerged than focused code changes.
5. Overall throughput is healthy: 76% of 1000 copilot PRs merged in the window, with only 20 still open.

Recommendations

1. Keep scoping linter/refactor tasks tightly — the smallest-diff clusters (1, 6) merge best; continue decomposing large changes into single-rule / single-file PRs.
2. Batch or automate version bumps — cluster 5 is high-volume, high-file-count, low-merge. A dedicated deterministic bump workflow (fewer agent turns) would cut churn from superseded PRs.
3. Front-load review for docs/generated content — cluster 7's large diffs and lower merge rate point to review bottlenecks; add stricter pre-flight checks or templates for generated docs.
4. Instrument turn/cost metrics — this run lacked gh-aw logs; wiring per-run turn counts into the pipeline would let future analyses correlate prompt style with iteration cost directly.
5. Reuse sous-chef patterns — cluster 3's high success shows structured, review-heavy self-improvement prompts work; consider templating that structure for other task types.

References: §29572958513

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 211.9 AIC · ⌖ 37.6 AIC · ⊞ 13.2K · ◷

• expires on Jul 18, 2026, 2:49 AM UTC-08:00

[github-actions[bot]]

Smoke test ping for run 29576598634. ✅

📰 BREAKING: Report filed by Smoke Copilot · 79 AIC · ⌖ 5.41 AIC · ⊞ 19K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

Smoke tap. Comment test ok.

📰 BREAKING: Report filed by Smoke Copilot · 86.7 AIC · ⌖ 4.41 AIC · ⊞ 19K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}