[prompt-clustering] Copilot Agent Prompt Clustering — 2026-07-14

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-06-24 → 2026-07-14)
Total Tasks Analyzed: 1,000 Copilot agent PRs
Clusters Identified: 8 (TF-IDF + K-means, k selected by silhouette)
Overall Merge Rate: 78.3% (783 merged · 210 closed · 7 open)
Median Time-to-Close: ~2.6h · Mean: ~4.6h

Copilot agent work in gh-aw splits into eight durable themes. Volume is concentrated in CI failure fixes (24%), the safe-outputs subsystem (20%), and workflow authoring/skills (19%). The cleanest, highest-yield work is ESLint-factory rule authoring (88% merge). The single strongest failure predictor is a PR that ships as a [WIP]/[review] draft — those merge at just 26% versus 78% overall.

Cluster overview

#	Theme	Tasks	%	Merge rate	Avg h open	[WIP]
1	CI job & step failure fixes	240	24%	81%	4.1	18
7	Safe-outputs subsystem	199	20%	80%	5.5	6
3	Workflow authoring & skills	190	19%	74%	1.7	12
2	Docs / spec / schema (SPDD)	113	11%	72%	5.9	6
0	SousChef / agent orchestration	82	8%	83%	12.7	2
5	ESLint-factory rules	76	8%	88%	3.9	1
6	Versioning & firewall/domains	58	6%	71%	4.4	1
4	Model & sub-agent config	42	4%	74%	1.7	1

Full cluster analysis, examples & recommendations

Cluster detail

Cluster 1 — CI job & step failure fixes (240 tasks, 24%)

• Merge rate: 81% · Avg open: 4.1h · Largest cluster.
• Keywords: job, step, fix, failure, agent, run, changes, github
• Character: reactive repairs to workflow jobs/steps — auth wiring, path redirection, log diagnostics. Highest [WIP] count (18), indicating agents frequently open drafts here before the fix stabilizes.
• Examples: test: multi-checkout cross-org credential wiring integration tests #42937 (cross-org credential wiring tests, merged), fix: github-app auth support in SideRepoOps maintenance workflow generator #43406 (github-app auth in SideRepoOps, merged), [WIP] Fix ARC/DinD topology path redirections in gh-aw compiler #42808 ([WIP] ARC/DinD path redirections, closed).

Cluster 7 — Safe-outputs subsystem (199 tasks, 20%)

• Merge rate: 80% · Avg open: 5.5h.
• Keywords: safe, output, path, workflow, behavior, shared
• Character: focused subsystem work — GraphQL→REST migrations, git safe.directory handling, ingestion hardening. Well-scoped and reliably merged, though some hardening tasks run long (safe_outputs: set git safe.directory via process-scoped env vars before bundle branch pinning #41830 merged after 67h).
• Examples: Migrate set_issue_type safe output from GraphQL to single REST issues.update call #41241 (set_issue_type GraphQL→REST, merged), safe_outputs: set git safe.directory via process-scoped env vars before bundle branch pinning #41830 (git safe.directory env, merged), Harden issue-intent confidence handling in safe-output ingestion #43168 (issue-intent confidence hardening, closed).

Cluster 3 — Workflow authoring & skills (190 tasks, 19%)

• Merge rate: 74% · Avg open: 1.7h (fastest tier).
• Keywords: workflow, report, guidance, prompt, skill, issue
• Character: authoring/refactoring .md workflow bundles, personas, skills. Fast turnaround but lower merge rate — 12 [WIP] drafts, several exploratory refactors closed without merge ([WIP] Expand personas and add systematic cache-memory rotation #42090, [WIP] Refactor to create shared/guard-otlp.md bundle for workflows #42983).

Cluster 2 — Docs / spec / schema, SPDD (113 tasks, 11%)

• Merge rate: 72% (lowest tier) · Avg open: 5.9h.
• Keywords: docs, spec, engine, schema, reference, coverage
• Character: spec-driven (SPDD) daily plans and schema/reference docs. Long-running normative spec work; the daily "analyze breaking changes" drafts ([WIP] Analyze daily breaking changes for 2026-07-06 #43822) often land as closed WIPs.

Cluster 0 — SousChef / agent orchestration (82 tasks, 8%)

• Merge rate: 83% · Avg open: 12.7h (longest).
• Keywords: aic, chef, sous chef, souschef run, comment souschef
• Character: PR-SousChef orchestration + larger cross-cutting features (evals jobs, content redaction, custom-engine frontmatter). High success despite complexity, but the slowest to close — these are the heavyweight changes.

Cluster 5 — ESLint-factory rules (76 tasks, 8%)

• Merge rate: 88% (highest) · Avg open: 3.9h.
• Keywords: rule, eslint, suggestion, catch, scope, function, factory
• Character: tightly-specified lint-rule authoring/autofix with clear accept criteria and test cases. The most reliable task type in the repo — narrow scope + testable spec = near-deterministic success.

Cluster 6 — Versioning & firewall/domains (58 tasks, 6%)

• Merge rate: 71% (lowest tier) · Avg open: 4.4h.
• Keywords: version, bump, domains, firewall, claude, v0, awf
• Character: version bumps, firewall allowlist/domain edits, AWF runtime changes. Mixed outcomes; policy-sensitive edits (Remove non-strict override from AOAI API key smoke workflow and restore Copilot BYOK env passthrough #41401 AOAI smoke) tend to be closed.

Cluster 4 — Model & sub-agent config (42 tasks, 4%)

• Merge rate: 74% · Avg open: 1.7h (fastest tier).
• Keywords: model, sub agent, copilot, workflow, alias, models
• Character: model registry/pricing injection, sub-agent routing, false-positive detection fixes. Small, quick changes.

Key findings

1. Scope clarity beats task size. ESLint-factory rules (88%) and safe-outputs (80%) are narrow subsystems with explicit acceptance criteria and tests — they merge far more reliably than open-ended docs/spec (72%) and firewall/versioning (71%) work.
2. [WIP]/[review] prefix is the dominant failure signal. The 47 PRs opened as drafts merged at 26% vs 78% overall — a 3× gap. Draft-first is concentrated in CI-fix (18) and workflow-authoring (12) clusters.
3. Time-open ≠ quality. The slowest cluster (SousChef, 12.7h avg) has the 2nd-highest merge rate (83%); the fastest clusters (workflow/model, 1.7h) sit mid-pack. Duration tracks task weight, not success — success tracks specification quality.
4. Reactive CI-firefighting is the single biggest workload (24% of all PRs). A quarter of agent capacity goes to repairing job/step failures rather than net-new features.

Recommendations

1. Templatize the two winning shapes. Codify the ESLint-rule and safe-outputs task templates (explicit acceptance criteria + required test cases) and reuse that structure for lower-yield clusters (docs/spec, firewall) to lift their ~71–72% merge rate.
2. Discourage draft-first for small fixes. Given the 26% draft merge rate, prompt agents to complete-and-verify before opening rather than opening [WIP] early — especially in the CI-fix cluster where 18 drafts accumulated.
3. Invest upstream on CI stability. With 240 PRs (24%) spent fixing job/step failures, reducing flaky-auth and path-redirection classes at the source would free the largest single block of agent capacity.
4. Add explicit acceptance criteria to SPDD/docs prompts. The lowest-merging clusters are the most open-ended; front-loading normative requirements should reduce closed exploratory drafts.

Methodology: title+body TF-IDF (1–2 grams, min_df=3), K-means with k chosen by silhouette over k∈[3,8]. Turn-count enrichment was unavailable (workflow logs not present for this window); "avg hours open" is used as the complexity/iteration proxy. Full-data cache (pr-full-data/) was stale (May, PR ~30k) and excluded in favor of the fresh 1,000-PR summary set.

---

Generated by Prompt Clustering Analysis (Run: 29324781746)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 152 AIC · ⌖ 18.7 AIC · ⊞ 13.2K · ◷

• expires on Jul 15, 2026, 2:46 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #45716.