[prompt-clustering] Prompt Clustering Analysis - 2026-08-22

[github-actions[bot]]

Summary

Analysis Period: Last ~17 days of activity (2026-08-05 → 2026-08-22), capped at 1,000 most recent copilot-authored PRs
Total PRs Pulled: 1,000 (643 organic coding-agent task PRs, 357 automation/meta-workflow PRs — see note below)
Clusters Identified: 4
Overall Merge Rate (task PRs): 77.4%

Data-quality note: 357 of the 1,000 PRs (36%) were not organic "give the agent an issue" tasks — they were generated by other gh-aw automation workflows running under the same bot account (dominated by PR Sous Chef, 320 PRs, 88% merged, which posts automated review/fix suggestions on other PRs). These were excluded from clustering so the prompt themes below reflect real task prompts. They're broken out separately in the table below since they're still relevant to overall bot activity.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 1 — general bug fixes / test coverage / feature additions across the agentic-workflow engine (273 tasks, 42% of the corpus)
• Highest Success Rate: Cluster 0 — CI/Actions job-failure triage and fixes (88.2%)
• Most Complex Tasks: Cluster 2 — CLI/MCP/container-image infrastructure work, averaging 42 files changed per PR, more than double Cluster 0's 7.4
• Outlier: Cluster 2 is the only cluster whose merge rate falls ≥10 points below the overall rate at a size ≥15 PRs (67.4% vs 77.4%, a 10.3-point gap, n=132)

Cluster Analysis

Cluster 0: CI/Actions failure triage and fixes

• Size: 34 tasks (5% of total)
• Success Rate: 88.2%
• Avg files changed: 7.4 · Avg comments: 2.1 · Avg reviews: 1.4
• Top Keywords: job, actions, fix, failure, check, root cause, logs, cause
• Characteristics: Narrowly scoped "a CI job is red, find the root cause and fix it" tasks. Small diffs, fast turnaround, highest success rate of any cluster.
• Example PRs: [WIP] Fix failing GitHub Actions job activation #50468 ([WIP] Fix failing GitHub Actions job activation), Fix docs: custom safe-output jobs run once per run, not once per tool call #50635 (safe-output jobs run-once doc fix), cjs.yml: add checkout-cache job to deduplicate checkouts across parallel jobs #50706 (checkout-cache dedup job), [WIP] Fix failing GitHub Actions job Integration: Workflow Features #50788 ([WIP] Fix failing Integration workflow job)

---

Cluster 1: General bug fixes, test coverage, feature additions

• Size: 273 tasks (42% of total)
• Success Rate: 77.3%
• Avg files changed: 17.7 · Avg comments: 3.8 · Avg reviews: 2.2
• Top Keywords: coverage, add, test, fixes, agent, added, error, existing
• Characteristics: The bulk of day-to-day agent work — incremental fixes and additions across the engine/CLI/tests. Success rate tracks the overall baseline closely.
• Example PRs: Creating sample smoke-langchain agentic workflow in Python #50448 (smoke-langchain sample workflow), Use safe-outputs token during agent output ingest #50449 (safe-outputs token during agent output ingest), Creating shared agentic workflow for cursor engine #50451 (shared workflow for cursor engine), Fixing crush configuration error #50452 (crush config fix)

---

Cluster 2: CLI / MCP / container-image infrastructure ⚠️ outlier

• Size: 132 tasks (21% of total)
• Success Rate: 67.4% (10.3 points below overall)
• Avg files changed: 42.0 · Avg comments: 3.0 · Avg reviews: 1.7
• Top Keywords: pkg, aw, gh, mcp, image, gh aw, github, cli
• Characteristics: Larger, more sprawling changes touching compiled/generated workflow artifacts, MCP gateway/container image pins, and CLI internals. Nearly 6x the file-change footprint of Cluster 0. This is the lowest-merge cluster — see root-cause analysis below.
• Example PRs: Add open-ontologies shared MCP server, apply to glossary-maintainer #50450 (shared MCP server for glossary-maintainer), Harden CLI external-tool execution paths against command injection #50510 (harden CLI exec paths against command injection), Clarify ad hoc scenario evaluation invocation surface for agentic-workflows agent #50606 (clarify ad hoc scenario invocation), compile: stop adding ineffective merge=ours to lock.yml .gitattributes entry #50639 (compile: fix .gitattributes merge strategy)

---

Cluster 3: Workflow/engine authoring and docs

• Size: 204 tasks (32% of total)
• Success Rate: 82.4%
• Avg files changed: 23.8 · Avg comments: 2.4 · Avg reviews: 1.4
• Top Keywords: workflow, copilot, workflows, md, daily, pr, engine, agent
• Characteristics: Authoring/fixing .md workflow definitions, daily workflows, and multi-engine support (Cursor, Kiro, etc.). Second-highest success rate.
• Example PRs: Remove stale token telemetry tracking link #50458 (remove stale token telemetry link), [WIP] Fix smoke cursor workflow failure due to authentication issue #50472 ([WIP] fix smoke cursor workflow auth failure), [WIP] Fix kiro config issues #50480 ([WIP] fix kiro config issues), docs: add Cursor and Kiro to agentic engine documentation #50487 (docs: add Cursor/Kiro to engine docs)

Success Rate by Cluster

Cluster	Tasks	Merged	Success Rate	Avg Files	Avg Comments	Top Keywords
0 — CI/Actions triage	34	30	88.2%	7.4	2.1	job, actions, fix, failure, root cause
1 — General fixes/tests	273	211	77.3%	17.7	3.8	coverage, add, test, fixes, agent
2 — CLI/MCP/container infra ⚠️	132	89	67.4%	42.0	3.0	pkg, aw, gh, mcp, image, cli
3 — Workflow/engine authoring	204	168	82.4%	23.8	2.4	workflow, copilot, workflows, md, daily

Meta-Workflow Segment (excluded from clustering)

Workflow	PRs	Merged	Merge Rate
pr-sous-chef	320	281	87.8%
daily-geo-optimizer	6	0	0%
daily-squid-image-scan	6	0	0%
deep-report	6	0	0%
daily-go-test-parallelizer	3	3	100%
sergo	3	0	0%
aw-failure-investigator	2	0	0%
workflow-normalizer	2	0	0%
smoke-claude	2	2	100%
daily-spdd-spec-planner	2	0	0%
others (single PR each)	5	~1	—

The daily-* report-style workflows show 0% merge rates, but each has only 1-6 PRs — read as "these are typically informational/report PRs not intended to merge," not as a quality signal.

Key Findings

1. Task complexity correlates with merge failure. Across all four clusters, avg files-changed tracks inversely with success rate: Cluster 0 (7.4 files, 88% merged) vs Cluster 2 (42.0 files, 67% merged). Within Cluster 2 itself, closed/abandoned PRs averaged 62 files changed vs 32 for merged ones — nearly double.
2. PR Sous Chef dominates automated review activity. 320 of the 1,000 pulled PRs (32%) are from a single meta-workflow with an 88% merge rate, separate from genuine coding-agent tasks — worth knowing when interpreting "copilot PR" volume metrics elsewhere.
3. CI/Actions-failure-triage tasks are the safest bet. Narrowly-scoped "find root cause, fix the failing job" prompts (Cluster 0) succeed at the highest rate (88%) with the smallest diffs — a good template for other task types.
4. Workflow-authoring tasks (Cluster 3) trail closely behind at 82%, suggesting .md workflow/engine config changes are similarly well-scoped for the agent even though they touch more files than Cluster 0.

Lowest-Merge Cluster Root Cause

• Cluster: 2 — CLI / MCP / container-image infrastructure
• Merge Rate Gap: 67.4% vs 77.4% overall (gap: 10.3 points), n=132 (43 non-merged)
• Primary Blocker: Scope/complexity mismatch, compounded by upstream-dependency blocks
• Confidence: Medium-high
• Evidence PRs: Document blocker for gh-aw-firewall/agent scan remediation #51981 (blocked on upstream gh-aw-firewall CVE fix), No gh-aw change: Anthropic proxy fix requires gh-aw-firewall #52671 (blocked on upstream Anthropic-proxy release), Retain firewall and MCP observability artifacts #51153 (291-file observability PR, CHANGES_REQUESTED across 5 automated reviewers), Bump gh-aw-mcpg to v0.4.9 #51796/Bump gh-aw-mcpg image to v0.4.9 #51770 (289-file MCP gateway digest bumps, closed as duplicates), Add regression guard for vulnerable gh-aw-firewall squid 0.27.44 container pin #52939/Suppress RGS-016 false positive in agentic_commands.yml generator #53609 (draft PRs with 0-diff, "revisit next run")

Classifying the 43 non-merged Cluster-2 PRs by the review/comment evidence in their full PR data:

Blocker Category	Count	Share	Notes
No engagement / abandoned (0 comments, 0 reviews)	15	35%	Closed with no human or bot interaction recorded — consistent with stale or superseded container/version-bump PRs
Other/unclassified	8	19%	Mix of routine security-patch and digest-refresh PRs closed without a clear terminal signal
Draft/WIP/empty diff	6	14%	PR opened but branch never populated (0 files changed), later closed as stale
Upstream-blocked dependency	6	14%	Fix requires a release from an external repo (e.g. github/gh-aw-firewall) before this repo's PR can proceed
CI/automation-check failure	3	7%	"Threat Detection Engine Failure" / automated code-quality or design-gate check failures
Duplicate	3	7%	Explicitly marked duplicate of another PR (common for MCP gateway digest bumps landing twice)
Merge conflict	2	5%	Stale branch, conflicts never resolved

The pattern: this cluster's tasks (compiled workflow artifacts, MCP gateway pins, container CVE remediation, CLI internals) tend to (a) touch far more files per PR than other clusters, (b) frequently depend on an external repo's release before they can be completed, and (c) recur rapidly as new digest/version-bump PRs supersede older ones — producing a steady stream of abandoned or duplicate PRs alongside merged ones.

Recommendations

1. Scope container/MCP/CLI infrastructure tasks more narrowly. Where possible, split "bump this digest + regenerate all compiled workflows" tasks from broader CLI refactors — the 289-file mega-PRs are the ones getting closed as duplicates or abandoned.
2. Surface upstream-dependency blocks earlier. For CVE/container fixes that require an external repo release first (e.g. gh-aw-firewall), consider having the agent open a tracking issue instead of a PR until the upstream fix lands, to avoid the PR sitting open/stale.
3. Replicate the Cluster-0 pattern for other task types. "Root cause + fix" style prompts with a clear, bounded failure signal (a red CI job) consistently outperform broader prompts — worth using as a template when writing new task prompts for infra work.
4. Cluster 2 specifically: add an automated staleness check that closes or re-triages container/digest-bump PRs after N days of no engagement, since 35% of this cluster's non-merges had zero comments or reviews recorded.

---

Generated by Prompt Clustering Analysis (Run: 32565926376)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 132.4 AIC · ⌖ 12.2 AIC · ⊞ 15.5K · ◷

• expires on Aug 23, 2026, 2:18 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #55056.