[prompt-clustering] Prompt Clustering Analysis - 2026-08-04

[github-actions[bot]]

Summary

Analysis Period: 2026-07-16 → 2026-08-04 (~19 days, most recent 1,000 copilot-authored PRs)
Total Tasks Analyzed: 1,000
Clusters Identified: 9
Overall Merge Rate: 75.7% (757 merged, 237 closed unmerged, 6 still open)

Task prompts were reconstructed from each PR's title + body (the agent's own running task description, truncated before the START COPILOT CODING AGENT marker), cleaned of code fences/URLs/issue refs, then clustered with TF‐IDF (unigrams+bigrams, 300 features) + K‐means (k selected via elbow method, capped 5–9). No aw_info.json / turn-count data was available in this run (the gh-aw CLI is not installed in this sandbox), so "iteration effort" below is approximated with comment/review/commit counts rather than agent turn counts — see Limitations.

Full Analysis Report

General Insights

• Most common task type: General bug fixes & test coverage (311 PRs, 31%) — a broad catch-all of linter-rule work (eslint-factory), one-off bug fixes, and new test coverage. Its size and generic top terms (test, fix, add, error) reflect that this repo's agent workload is dominated by small, mechanical fix/test tasks rather than any single coherent feature.
• Highest merge rate: Workshop UI/UX (95.8%) and Feature/refactor grab-bag (92.8%) — both are narrowly-scoped, self-contained frontend/CLI tweaks with few review rounds.
• Lowest merge rate: Dependency & container pinning (51.4%) and Large-function refactors & docs (56.3%) — both involve cross-cutting changes (version bumps, container digests, multi-file splits) that are easy to get subtly wrong or that maintainers frequently reject/redirect.
• Most review-heavy: Feature/refactor grab-bag (avg 12.6 comments, 5.0 reviews) and Engine/model/auth bug fixes (avg 9.6 comments) — these touch shared engine/runtime code and PR-Sous-Chef tooling, drawing more scrutiny.
• Largest diffs: Large-function refactors & docs (avg ~1,281 additions / ~9,017 deletions) — dominated by big file splits and generated-doc regeneration.

Cluster Analysis

Cluster 1: General bug fixes & test coverage

• Size: 311 tasks (31.1% of total)
• Merge rate: 80.7%
• Avg comments / reviews / commits: 6.4 / 3.4 / 4.0
• Avg files changed: 14.7 (avg +248 / −3,494 lines — many touch generated/lock files)
• Top keywords: test, fix, add, error, tests, added, new, spec
• Characteristics: The default bucket for eslint-factory linter-rule fixes, one-off bug patches, and added test coverage. High volume, above-average success — these are well-scoped, mechanical tasks the agent handles reliably.
• Example PRs: fix(setup-js): wrap all sync fs calls in try/catch (eslint-monster remediation) #46102 (wrap sync fs calls in try/catch), feat: add --rootless flag to install_copilot_cli.sh for ARC/DinD runners #46047 (--rootless flag for install script), Use native go-gh REST clients for release and ref lookups #47547 (native go-gh REST clients)

---

Cluster 2: Workflow & evals maintenance

• Size: 213 tasks (21.3%)
• Merge rate: 75.1%
• Avg comments / reviews / commits: 3.8 / 2.3 / 3.1
• Avg files changed: 18.7
• Top keywords: workflow, workflows, agent, prompt, daily, report, guidance, copilot
• Characteristics: Maintenance of the agentic-workflow definitions themselves — shared bundles, evals, OTLP/telemetry, docs for ARC/DinD. Lower comment volume suggests these are relatively routine to review.
• Example PRs: OTLP conclusion: mark non-zero Copilot CLI exits as run failures when conclusion/error signals are absent #47268 (OTLP conclusion for non-zero exits), [evals] Enforce binary YES/NO outputs in evals parser #47245 (enforce binary YES/NO in evals parser), docs: improve ARC/DinD guide with expanded troubleshooting, log paths, and tool cache guidance #46045 (ARC/DinD guide)

---

Cluster 3: Engine/model/auth bug fixes

• Size: 114 tasks (11.4%)
• Merge rate: 71.9% (below average)
• Avg comments / reviews / commits: 9.6 / 3.5 / 4.9
• Avg files changed: 49.0 (largest footprint after refactors)
• Top keywords: model, engine, copilot, pricing, alias, fix, claude, ai
• Characteristics: Bug fixes in engine routing, model pricing/aliasing, and auth-error detection across Claude/Copilot/Codex harnesses. Highest comment count among bug-fix clusters — these touch shared, high-blast-radius code and draw more back-and-forth.
• Example PRs: Fix Claude engine routing for model-provider: github #46013 (Claude engine routing for model-provider:github), feat(copilot): support model: auto/model: none to bypass model pinning on automatic-selection plans #46556 (support model: auto/none), fix: escape MCP env secrets with \${VAR} for non-Copilot engines to prevent heredoc JSON corruption #49450 (escape MCP env secrets)

---

Cluster 4: Dependency & container pinning

• Size: 74 tasks (7.4%)
• Merge rate: 51.4% (lowest of all clusters)
• Avg comments / reviews / commits: 5.5 / 1.9 / 3.0
• Avg files changed: 21.6
• Top keywords: image, container, digest, pin, scan, container image, vulnerabilities, high
• Characteristics: Version/digest bumps and container vulnerability-scan fixes. Low review count but the lowest merge rate — suggests many of these are opened, then superseded/closed rather than iterated on (e.g. duplicate/stale version-bump PRs).
• Example PRs: fix(multi-device-docs-tester): Chrome SUID sandbox and server unreachable from agent container #46157 (Chrome SUID sandbox fix), chore: bump CLI versions and pin Docker image digests (2026-07-29) #48794 (bump CLI versions + pin digests), chore: upgrade gh-aw-firewall v0.27.37 → v0.27.38 to clear critical CVE gate #47469 (upgrade gh-aw-firewall)

---

Cluster 5: Feature/refactor grab-bag

• Size: 97 tasks (9.7%)
• Merge rate: 92.8% (2nd highest)
• Avg comments / reviews / commits: 12.6 / 5.0 / 5.8 (most reviewed cluster)
• Avg files changed: 43.9
• Top keywords: (mixed — PR-Sous-Chef, eslint-factory aliasing, workshop cards, CLI version bumps)
• Characteristics: A heterogeneous mix TF-IDF grouped by shared secondary vocabulary rather than one theme; dominated by PR-Sous-Chef nudge-comment features. High merge rate despite heavy review suggests these get resolved through iteration rather than abandonment.
• Example PRs: Bump default AWF and MCP Gateway versions to v0.27.42 / v0.4.6 and refresh pinned artifacts #48236 (bump AWF/MCP Gateway versions), fix: guard conclude_threat_detection.sh so threat-detect never runs when RUN_DETECTION=false #47019 (guard threat-detect conclude script), Preserve HTTPS angle-bracket autolinks in safe-output sanitization #48169 (preserve autolinks in sanitizer)

---

Cluster 6: Large-function refactors & docs

• Size: 71 tasks (7.1%)
• Merge rate: 56.3% (2nd lowest)
• Avg comments / reviews / commits: 5.6 / 3.0 / 4.3
• Avg files changed: 19.0, but by far the largest diffs (avg +1,281 / −9,017 lines)
• Top keywords: docs, helpers, refactor, txt, lines, site, cli, functions
• Characteristics: Splitting oversized functions/files to satisfy largefunc/funcparams linters, plus docs-site restructuring. Large mechanical diffs plus below-average merge rate hint that these are more likely to be redirected, split, or superseded by a cleaner follow-up PR.
• Example PRs: fix: address 14 CLI consistency issues (branding, descriptions, docs) #47592 (CLI consistency fixes), Refactor schedule heatmap styling to use centralized theme styles with NO_COLOR-aware degradation #48223 (refactor schedule heatmap styling), Refactor token usage CLI into focused modules #47088 (refactor token-usage CLI, closed)

---

Cluster 7: Safe Outputs & sandbox hardening

• Size: 80 tasks (8.0%)
• Merge rate: 76.2% (~average)
• Avg comments / reviews / commits: 7.8 / 3.3 / 4.8
• Avg files changed: 34.8
• Top keywords: safe, safe output, output, safe outputs, outputs, fix, retry, tools
• Characteristics: Security/robustness work on the safe-outputs bridge and sandbox/credential isolation — a security-sensitive area with above-average review depth.
• Example PRs: fix(safe-outputs): bridge git safe.directory trust via GIT_CONFIG_* env vars #46057 (bridge git safe.directory trust), fix: move sink-visibility to env var to prevent zizmor template-injection #48589 (move sink-visibility to env var), fix: forward GH_AW_INPUT_* to MCP container env for dynamic safe-outputs config #48099 (forward GH_AW_INPUT_* to MCP container)

---

Cluster 8: Workshop UI/UX

• Size: 24 tasks (2.4%, smallest coherent cluster)
• Merge rate: 95.8% (highest of all clusters)
• Avg comments / reviews / commits: 4.2 / 2.3 / 3.4
• Avg files changed: 3.4 (smallest footprint)
• Top keywords: workshop, content, step, entry, progress, path, docs, remove
• Characteristics: Small, narrowly-scoped frontend fixes for the interactive docs "workshop" experience. Small diffs + high merge rate = the agent's best-performing task category.
• Example PRs: Render workshop content through Astro-generated markdown sources #46406 (render workshop content via Astro), feat(docs): encode workshop state in URL hash for back/forward navigation #46405 (encode workshop state in URL hash), Route workshop landing to org-scoped path (/workshop/hackathon-blue-bat-18/) #46579 (route workshop to org-scoped path)

---

Cluster 9: CI job auto-fix (WIP triage)

• Size: 16 tasks (1.6%, smallest cluster)
• Merge rate: 75.0%
• Avg comments / reviews / commits: 1.6 / 1.3 / 2.6 (least discussion)
• Avg files changed: 1.9 (smallest, most surgical diffs)
• Top keywords: actions, github actions, job, failing, github, wip fix, fix, wip
• Characteristics: Auto-generated "[WIP] Fix failing GitHub Actions job X" tasks from a CI-failure-triage workflow. Very low review overhead — small, targeted, mostly rubber-stamped.
• Example PRs: [WIP] Fix failing GitHub Actions job agent #45935 (fix failing job "agent"), [WIP] Fix failing GitHub Actions job Integration: CLI Add & List Commands #46160 (fix CLI Add & List integration), [WIP] Fix failing GitHub Actions job for CLI completion #46859 (fix CLI completion job)

Success Rate by Cluster

Cluster	Tasks	% of total	Merge Rate	Avg Comments	Avg Files Changed	Top Keywords
Workshop UI/UX	24	2.4%	95.8%	4.2	3.4	workshop, content, step
Feature/refactor grab-bag	97	9.7%	92.8%	12.6	43.9	sous chef, eslint-factory
General bug fixes & test coverage	311	31.1%	80.7%	6.4	14.7	test, fix, add, error
Safe Outputs & sandbox hardening	80	8.0%	76.2%	7.8	34.8	safe outputs, retry, tools
Workflow & evals maintenance	213	21.3%	75.1%	3.8	18.7	workflow, agent, prompt
CI job auto-fix (WIP triage)	16	1.6%	75.0%	1.6	1.9	job, failing, wip
Engine/model/auth bug fixes	114	11.4%	71.9%	9.6	49.0	model, engine, pricing
Large-function refactors & docs	71	7.1%	56.3%	5.6	19.0	refactor, functions, docs
Dependency & container pinning	74	7.4%	51.4%	5.5	21.6	image, digest, pin, scan

Sample Data Table

3 highest-discussion PRs per cluster (27 of 1,000 shown)

PR #	Title	Cluster	Outcome	Comments	Files
#46057	fix(safe-outputs): bridge git safe.directory trust via GIT_CONFIG_*	Safe Outputs & sandbox hardening	Merged	55	10
#48589	fix: move sink-visibility to env var to prevent zizmor template errors	Safe Outputs & sandbox hardening	Merged	41	278
#48099	fix: forward GH_AW_INPUT_* to MCP container env for dynamic inputs	Safe Outputs & sandbox hardening	Merged	40	9
#46859	[WIP] Fix failing GitHub Actions job for CLI completion	CI job auto-fix	Merged	9	6
#45935	[WIP] Fix failing GitHub Actions job agent	CI job auto-fix	Merged	3	2
#46160	[WIP] Fix failing GitHub Actions job Integration: CLI Add & List	CI job auto-fix	Merged	3	9
#46013	Fix Claude engine routing for model-provider: github	Engine/model/auth bug fixes	Closed	104	75
#46556	feat(copilot): support model: auto/none to bypass	Engine/model/auth bug fixes	Closed	68	294
#49450	fix: escape MCP env secrets with \${VAR} for non-Copilot engines	Engine/model/auth bug fixes	Merged	51	12
#46157	fix(multi-device-docs-tester): Chrome SUID sandbox and server unreachable	Dependency & container pinning	Merged	81	3
#48794	chore: bump CLI versions and pin Docker image digests	Dependency & container pinning	Merged	32	43
#47469	chore: upgrade gh-aw-firewall v0.27.37 → v0.27.38	Dependency & container pinning	Merged	31	279
#48236	Bump default AWF and MCP Gateway versions to v0.27.42 / v0.4.x	Feature/refactor grab-bag	Merged	55	278
#47019	fix: guard conclude_threat_detection.sh so threat-detect never hangs	Feature/refactor grab-bag	Merged	49	3
#48169	Preserve HTTPS angle-bracket autolinks in safe-output sanitizer	Feature/refactor grab-bag	Merged	38	4
#47592	fix: address 14 CLI consistency issues (branding, descriptions)	Large-function refactors & docs	Merged	22	9
#48223	Refactor schedule heatmap styling to use centralized theme scales	Large-function refactors & docs	Merged	22	2
#47088	Refactor token usage CLI into focused modules	Large-function refactors & docs	Closed	20	7
#46102	fix(setup-js): wrap all sync fs calls in try/catch (eslint-monster)	General bug fixes & test coverage	Merged	36	59
#46047	feat: add --rootless flag to install_copilot_cli.sh for ARC/DinD	General bug fixes & test coverage	Merged	35	16
#47547	Use native go-gh REST clients for release and ref lookups	General bug fixes & test coverage	Merged	34	282
#46406	Render workshop content through Astro-generated markdown sources	Workshop UI/UX	Merged	15	11
#46405	feat(docs): encode workshop state in URL hash for back/forward nav	Workshop UI/UX	Merged	10	3
#46579	Route workshop landing to org-scoped path (/workshop/hackathon)	Workshop UI/UX	Merged	10	7
#47268	OTLP conclusion: mark non-zero Copilot CLI exits as run failures	Workflow & evals maintenance	Merged	36	175
#47245	[evals] Enforce binary YES/NO outputs in evals parser	Workflow & evals maintenance	Merged	30	128
#46045	docs: improve ARC/DinD guide with expanded troubleshooting	Workflow & evals maintenance	Merged	23	6

Key Findings

1. Task mix is dominated by mechanical maintenance, not net-new features. Two-thirds of all 1,000 PRs fall into three clusters — general bug/test fixes (31%), workflow/evals maintenance (21%), and engine/model bug fixes (11%) — reflecting a repo whose agent workload leans heavily on linter enforcement, CI stability, and internal tooling upkeep rather than user-facing feature work.
2. Merge rate correlates inversely with diff size and cross-cutting scope, not with topic difficulty per se. The two lowest-performing clusters (dependency/container pinning at 51%, large-function refactors at 56%) both involve either duplicate/superseded version bumps or sprawling mechanical splits — patterns that are more likely to be abandoned/redirected than genuinely "hard" bugs.
3. Narrow, single-purpose tasks perform best. Workshop UI/UX (96%) and CI-job auto-fix (75%, but with minimal review overhead) both have very small average diffs (3.4 and 1.9 files) — the agent's clearest wins are tightly-scoped, well-bounded tasks.
4. High review volume doesn't necessarily predict failure. The Feature/refactor grab-bag cluster has the most comments/reviews (12.6/5.0) and a 93% merge rate — suggesting the agent iterates well when given feedback, at least for PR-Sous-Chef-adjacent work.

Recommendations

1. Investigate the Dependency & container pinning cluster's low merge rate (51%). Check whether closed PRs here are duplicates racing against a batching/scheduling process (e.g. multiple concurrent version-bump PRs) rather than genuine agent failures — if so, consider debouncing or consolidating these triggers before assigning them to the agent.
2. Give large-function-refactor tasks tighter scope guidance. These produce the largest diffs (avg +1,281/−9,017 lines) and a below-average 56% merge rate; splitting them into smaller, single-file refactor tasks (as already done successfully in some eslint-factory bug-fix tasks) may raise the merge rate.
3. Re-run this analysis with gh-aw logs turn/cost data included. This run had no workflow-run correlation (see Limitations) — turn counts and cost per cluster would sharpen the "iteration effort" story beyond the comment/review proxy used here.
4. Consider the CI job auto-fix and Workshop UI/UX clusters as reference cases for prompt design — both show that narrow, single-symptom prompts with a clear success criterion (a specific failing check, a specific UI regression) yield the highest merge rates with the least review friction.

Limitations

• No workflow-run/turn-count data: the gh-aw CLI was not available in this sandbox, so Phase 2 (turns, duration, cost per PR) could not be completed. All "effort" metrics above are PR-level proxies (comments, reviews, commits, files changed), not agent iteration counts.
• Cluster 5 ("Feature/refactor grab-bag") is a TF-IDF artifact — it grouped PRs sharing secondary vocabulary (tokenized fragments of "PR Sous Chef", version strings) rather than one clean theme. Treat its keyword list as noisy; the example PRs are a better guide to its actual content.
• Task prompts were derived from PR bodies (the agent's own generated running description), not the original issue/instruction text verbatim — for the ~33% of PRs without a START COPILOT CODING AGENT marker, the full body (an agent-authored changelog) was used as a proxy.

---

Generated by Prompt Clustering Analysis (Run: 30901094702)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 120.3 AIC · ⌖ 41.6 AIC · ⊞ 13.7K · ◷

• expires on Aug 5, 2026, 2:59 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #50534.