[prompt-clustering] Copilot Agent Prompt Clustering — Daily Analysis (2026-05-19)

[github-actions[bot]]

Summary

• Analysis window: last 30 days (cut-off 2026-04-19 → 2026-05-19)
• PRs analyzed: 1165 copilot-agent PRs in github/gh-aw
• Clusters identified: 7 (k chosen via silhouette sweep, score 0.035)
• Overall merge rate: 79.7% (928 merged · 214 closed-unmerged · 23 open)
• Largest theme: Compiler, schema & docs refactors (484 PRs, 81% merged)
• Highest merge rate: Daily/experiment workflow features at 85% (110 PRs)
• Lowest merge rate: WIP / closed-before-implementation at 55% (33 PRs)

Headline Findings

1. The agent is most successful on small, well-scoped changes. Every cluster except WIP/closed-before-implementation and Dependency & CLI version bumps lands above 80% merge rate. The WIP cluster (33 PRs) sits at 55% — these are PRs where the agent never wrote a real description and was abandoned early. Treat these as failed task assignments, not failed code.
2. Dependency bumps under-merge (61%). 130 PRs touched CLI versions, gateway images, and firewall configs. They average 4.4 comments — almost 2× the typical PR — suggesting the bumps frequently fail smoke tests or surface a follow-up incompatibility. Consider gating these on automated CLI version smoke tests before the agent opens the PR.
3. Daily/experiment scaffolding work has the heaviest human feedback loop — 2.3 human comments per PR vs ~1.3 for dependency bumps. This is iterative prompt-design work, not bugfixing. The high merge rate (85%) confirms the human-in-the-loop is productive here, not corrective.
4. Safe-outputs work is the largest-change cluster. Avg additions 334, avg files changed 20.1, and the highest commits/PR (5.0). Despite that, it merges at 85% — the agent handles wider blast radius well when the surface is well-typed (safe-outputs has a strict schema).

Full cluster analysis

Cluster Details

Cluster 5 — Compiler, schema & docs refactors

• Size: 484 PRs (41.5% of total)
• Merge rate: 81.4% (394 merged, 77 closed unmerged, 13 open)
• Engagement: avg 2.1 comments (1.4 human), 3.9 commits, 24.5 files changed
• Diff size: +368 / -216 lines on average
• Top terms: enhancement, shared, feature, docs, refactor, test, otlp, ci, package, path
• Characterization: The catch-all bucket: compiler validators, YAML/schema work, refactors, docs edits, OTLP shared workflows, and small feat:/fix: items.
• Examples: #29008 ✓ · #31234 ✓ · #31363 ✓

Cluster 3 — Engine, model & cache plumbing

• Size: 183 PRs (15.7% of total)
• Merge rate: 84.2% (154 merged, 27 closed unmerged, 2 open)
• Engagement: avg 2.1 comments (1.4 human), 3.6 commits, 25.6 files changed
• Diff size: +265 / -82 lines on average
• Top terms: agent, run, model, daily, cache, sub, inline, inventory, token, enhancement
• Characterization: Fixes to detection, cache-memory, inline agents, model inventory, threat-detection, and per-engine config.
• Examples: #29031 ✓ · #31266 ✓ · #31193 ✓

Cluster 0 — Dependency & CLI version bumps

• Size: 130 PRs (11.2% of total)
• Merge rate: 60.8% (79 merged, 47 closed unmerged, 4 open)
• Engagement: avg 4.4 comments (1.3 human), 3.4 commits, 65.5 files changed
• Diff size: +687 / -574 lines on average
• Top terms: cli, awf, mcp, copilot, version, bump, gateway, firewall, default, playwright
• Characterization: Bumps to CLI tools (Copilot CLI, Claude Code, Gemini, Playwright), gateway/firewall image versions, and related compatibility fixes.
• Examples: #30126 ✓ · #30040 ✓ · #32351 ✓

Cluster 4 — Failure recovery & lint hygiene

• Size: 129 PRs (11.1% of total)
• Merge rate: 82.9% (107 merged, 22 closed unmerged, 0 open)
• Engagement: avg 2.2 comments (1.6 human), 4.1 commits, 33.7 files changed
• Diff size: +348 / -369 lines on average
• Top terms: bug, failures, failure, lint, bash, slash, integration, release, bundle, cache
• Characterization: Workflow failures, bug fixes, dependency updates, lint/fuzz stabilization, and miscellaneous "keep CI green" work.
• Examples: #30222 ✓ · #31829 ✓ · #31912 ✓

Cluster 2 — Daily/experiment workflow features

• Size: 110 PRs (9.4% of total)
• Merge rate: 85.5% (94 merged, 16 closed unmerged, 0 open)
• Engagement: avg 4.0 comments (2.3 human), 3.7 commits, 21.4 files changed
• Diff size: +648 / -137 lines on average
• Top terms: summary, experiment, feat, daily, experiments, prompt, variant, feat daily, schema, docs
• Characterization: Net-new feature work, daily-workflow additions, prompt experiments, and feat: PRs against the workflow runtime.
• Examples: #29037 ✓ · #30016 ✓ · #30046 ✓

Cluster 6 — Safe-outputs & PR/issue automation

• Size: 96 PRs (8.2% of total)
• Merge rate: 85.4% (82 merged, 11 closed unmerged, 3 open)
• Engagement: avg 4.1 comments (2.4 human), 5.0 commits, 20.1 files changed
• Diff size: +334 / -79 lines on average
• Top terms: safe, safe output, output, safe outputs, outputs, branch, bundle, job, failure, causing
• Characterization: Changes to the safe_outputs surface (create_issue, add_comment, create_pull_request, labels) and downstream PR/issue side-effects.
• Examples: #29054 ✓ · #32027 ✓ · #32246 ✓

Cluster 1 — WIP / closed-before-implementation

• Size: 33 PRs (2.8% of total)
• Merge rate: 54.5% (18 merged, 14 closed unmerged, 1 open)
• Engagement: avg 0.3 comments (0.2 human), 2.4 commits, 24.2 files changed
• Diff size: +182 / -22 lines on average
• Top terms: asking, asking work, plan progress, thanks asking, form plan, date form, started description, work started, thanks, description date
• Characterization: PRs that never got past the agent's placeholder body ("Thanks for asking me to work on this..."). Almost all were closed early.
• Examples: #32004 ✓ · #32036 ✓ · #32831 ✓

Per-Cluster Summary Table

Cluster	Theme	PRs	Merge rate	Avg comments	Avg commits	Avg +/− lines
C5	Compiler, schema & docs refactors	484	81%	2.1	3.9	+368 / −216
C3	Engine, model & cache plumbing	183	84%	2.1	3.6	+265 / −82
C0	Dependency & CLI version bumps	130	61%	4.4	3.4	+687 / −574
C4	Failure recovery & lint hygiene	129	83%	2.2	4.1	+348 / −369
C2	Daily/experiment workflow features	110	85%	4.0	3.7	+648 / −137
C6	Safe-outputs & PR/issue automation	96	85%	4.1	5.0	+334 / −79
C1	WIP / closed-before-implementation	33	55%	0.3	2.4	+182 / −22

Sampled PR table (top 40 most-recent merged + 20 most-recent unmerged)

PR	Cluster	State	Title
#33276	C0	open	Adopt Lipgloss tree rendering for MCP hierarchy and status dependencies
#33275	C1	open	[WIP] Improve OTel instrumentation to emit gen_ai.response.model
#33273	C5	open	Add on.pull_request_reviewer: slash_command synthetic trigger and reviewer lifecycle rou
#33272	C5	open	Replace raw log.Print* usage in library packages with package-scoped logger instances
#33271	C3	open	[WIP] Extract shared/github-proxy-default.md to DRY 63 workflows
#33250	C5	✓ merged	Accept run_id alias for audit MCP tool parameters
#33249	C3	✓ merged	Self-heal cache-memory git setup when restored metadata is corrupt
#33248	C5	✓ merged	Tighten golden-test coverage: remove dead console fixtures and goldenize wasm all-engines
#33247	C3	✓ merged	Reduce Step Name Alignment agent turns via deterministic pre-agent manifest
#33246	C0	open	Bump default CLI/MCP toolchain versions and regenerate workflow locks
#33245	C5	✓ merged	OTel: propagate GenAI semantic attrs to gh-aw.agent.conclusion spans
#33242	C5	✓ merged	Refactor create-* config parsing to shared helper and unify string→array coercion
#33241	C5	✓ merged	Refactor parser engine-include tests to table-driven testify assertions
#33240	C5	open	Make setup-gh-aw install idempotent when gh-aw is already available
#33231	C6	✓ merged	refactor: extract resolveSafeOutputIssueTarget to eliminate safe-output target resolution
#33221	C3	✓ merged	Multi-Device Docs Tester: move Astro server startup to pre-agent steps to unblock backgrou
#33220	C5	✓ merged	Normalize report-style guidance across non-compliant issue/report workflows
#33219	C0	open	Bind Node toolcache into AWF chroot for Copilot-engine workflow startup reliability
#33218	C3	✓ merged	[aw] Prevent Step Name Alignment from using invalid gh search issues --state all
#33217	C5	✓ merged	Add panic recovery to mcp inspect background goroutines
#33216	C5	✓ merged	Enforce prod errstringmatch findings and centralize 403/410 error classification
#33205	C1	✓ merged	[WIP] Fix failing GitHub Actions job 'test'
#33204	C1	✓ merged	[WIP] Fix failing GitHub Actions job lint-go
#33203	C1	✓ merged	[WIP] Fix failing GitHub Actions job build-wasm
#33202	C5	✓ merged	Split shared OTLP backend import into Sentry, Grafana, and Datadog components
#33201	C5	closed	Replace ctx-local context.Background() fallbacks in action resolution and Docker image h
#33200	C5	✓ merged	Expose reusable workflow network allowlist as an opt-in workflow_call input
#33193	C1	✓ merged	[WIP] Fix failing GitHub Actions job Lint Gate
#33192	C6	closed	Add orthogonal parameter aliases to safe output tool schemas
#33191	C5	✓ merged	Validate required workflows on workflow_run during compile
#33182	C3	✓ merged	Add per-engine default model aliases with Sonnet-sized agent defaults
#33181	C5	✓ merged	Resolve #aw_* temporary IDs during bundle-based signed commit replay
#33180	C5	✓ merged	Refactor current-repository lookup caches to shared syncutil.OnceLoader
#33179	C3	✓ merged	Optimize CLI Consistency Checker via inline small-model sub-agents
#33178	C5	✓ merged	Restore authenticated ls-remote in signed commit push path
#33177	C3	✓ merged	Model inventory 2026-05-19: add raptor-mini alias coverage and missing GPT-5 search mult
#33171	C6	✓ merged	Handle resolve_pull_request_review_thread integration-token 403 as actionable soft-skip
#33170	C6	✓ merged	fix(safe-outputs): warn agents about field-name divergence in add_labels, remove_labels, a
#33168	C5	✓ merged	Fix gh-aw install guard to handle github/gh-aw owner-prefix in gh extension list outpu
#33165	C1	✓ merged	[WIP] Fix failing GitHub Actions job JS Tests (shard 4/4)
#33164	C5	✓ merged	feat: import workflow or JSON workflow from arbitrary HTTP(S) URL
#33163	C5	✓ merged	fix: rename branch when recreate-ref deleteRef is blocked by branch protection rules
#33152	C2	✓ merged	Wrap experiment assignment summary in collapsible details block
#33144	C1	✓ merged	[WIP] Fix failing GitHub Actions job 'Lint Gate'
#33134	C6	✓ merged	Analyze: update_pull_request with empty args silently drops safe outputs
#33130	C0	✓ merged	Align CLI help/docs consistency for outcomes, project setup, URLs, and wording
#33129	C5	✓ merged	Fix compound || expressions in prompt markdown never substituting at runtime
#33128	C5	closed	Drop shadow map[string]any fields: remove WorkflowData.Tools and ToolsConfig.raw
#33127	C0	closed	fix: add scope-guard, acceptance criteria, and version-bump constraints to prevent PR scop
#33125	C5	✓ merged	feat(pr-sous-chef): run formatters and push to branch
#33124	C0	✓ merged	Bump default AWF firewall to v0.25.49 and refresh compiled lockfiles
#33116	C0	✓ merged	fix: use COPILOT_DUMMY_BYOK indirection to suppress secret-scanner false positives on lock
#33088	C0	closed	Add AWF installer fallback when pinned firewall release is missing
#32938	C3	closed	fix(daily-model-inventory): remove runner-host /reflect pre-step and query reflect in-agen
#32937	C0	closed	Pin Daily Compiler Threat Spec Optimizer to AWF v0.25.46
#32911	C5	open	Add SSL Skill Normalizer and convert reporting, error-messages, jqschema to SSL JSON
#32906	C5	open	Harden privileged checkout path in q.lock.yml for comment-triggered runs
#32905	C5	closed	Add BMAD-guided PR reviewer workflow and slash-command routing
#32863	C5	closed	GEO: Fix broken links and reduce keyword stuffing in README
#32862	C5	closed	Add /sitemap.xml alias and strengthen README GEO/brand signals

Recommendations

1. Triage the WIP/abandoned cluster. 33 PRs were closed with only the agent's placeholder body. Either these tasks were duplicates the human cancelled, or the agent failed silently before writing anything. Worth tagging these in a follow-up to see which fraction is cancellation vs crash.
2. Pre-flight dependency bumps with a smoke run. The Dependency & CLI cluster has the most comments and the lowest merge rate among real work. Have the agent run the relevant smoke test locally before opening the PR, and only file the PR if smoke passes.
3. Don't over-cluster. Silhouette scores were low (best 0.035 at k=8); the prompts are stylistically very uniform ("feat: ... " / "fix: ..."). The clusters are still useful as broad themes, but treat the boundaries as fuzzy — a single PR could plausibly belong to two clusters.
4. Track success rate per cluster over time. Today's snapshot doesn't show drift. Persist cluster-info.json to cache-memory so the next run can show week-over-week movement (e.g., "Safe-outputs merge rate fell from 85% → 72%").

Methodology

• Data source: /tmp/gh-aw/prompt-cache/pr-full-data/pr-*.json (1,174 files, 1,170 valid records, 1,165 after length filter)
• Prompt extraction: PR title + first paragraph of body; stripped of code blocks, URLs, markdown formatting; min 40 chars
• Vectorization: TF-IDF, unigrams+bigrams, max 800 features, min_df=4, max_df=0.6, with domain stopwords (gh, aw, workflow, add, update, ...)
• Clustering: K-means, k swept 3..8; silhouette best at k=8 (0.035), capped at k=7 for interpretability
• Caveat: silhouette scores are uniformly low, indicating substantial cluster overlap — interpret cluster boundaries as fuzzy themes, not partitions

References: §26092309724

Generated by 📊 Copilot Agent Prompt Clustering Analysis · ● 12.6M · ◷

• expires on May 20, 2026, 11:14 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #33516.