[prompt-clustering] Copilot Agent Prompt Clustering Analysis - 2025-12-10

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts using K-means clustering and TF-IDF vectorization.

Summary

Analysis Period: Historical data (1,811 PRs)
Total Tasks Analyzed: 1789
Clusters Identified: 6
Overall Success Rate: 75.0%
Merged PRs: 1341
Closed (not merged): 372
Open PRs: 76

Key Findings

• Most Common Task Type: New Features (806 tasks, 45.1%)
• Highest Success Rate: Testing & Test Coverage (84.7%)
• Lowest Success Rate: CI/CD & Workflows (65.1%)
• Average Files Changed: 17.9 files per PR
• Average Code Changes: +1278/-532 lines

Full Analysis Report

Cluster Analysis

Cluster 4: New Features

Size: 806 tasks (45.1% of total)

Success Metrics:

• Merged: 579 (71.8%)
• Closed (not merged): 181
• Open: 46

Complexity Metrics:

• Avg files changed: 21.7
• Avg additions: 1060 lines
• Avg deletions: 578 lines
• Avg commits: 3.8
• Avg comments: 2.2
• Avg reviews: 1.4

Top Keywords: update, add, agent, step, github, firewall, use, make

Characteristics: Tasks in this cluster typically involve new features

Example PRs:

• #2097: Add minimal path format syntax reference to imports documentation
• #2099: Add directory creation for copilot engine --add-dir paths
• #2101: [WIP] Migrate JavaScript memory server to Wasm component

---

Cluster 6: Bug Fixes

Size: 398 tasks (22.2% of total)

Success Metrics:

• Merged: 300 (75.4%)
• Closed (not merged): 86
• Open: 12

Complexity Metrics:

• Avg files changed: 11.6
• Avg additions: 1350 lines
• Avg deletions: 420 lines
• Avg commits: 3.4
• Avg comments: 1.3
• Avg reviews: 1.5

Top Keywords: gh, gh aw, aw, issue, section, githubnext gh aw, githubnext gh, githubnext

Characteristics: Tasks in this cluster typically involve bug fixes

Example PRs:

• #2209: [WIP] Comment on issue [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157 regarding recurrence failure
• #2235: Fix: Raise error when agentic workflow hits max-turns limit
• #2254: [WIP] Update GITHUB_PERSONAL_ACCESS_TOKEN to GITHUB_MCP_SERVER_TOKEN

---

Cluster 3: Updates & Modifications

Size: 216 tasks (12.1% of total)

Success Metrics:

• Merged: 176 (81.5%)
• Closed (not merged): 32
• Open: 8

Complexity Metrics:

• Avg files changed: 10.6
• Avg additions: 1750 lines
• Avg deletions: 335 lines
• Avg commits: 3.7
• Avg comments: 2.0
• Avg reviews: 1.8

Top Keywords: agentic, agentic workflow, workflow, workflows, update, create, shared, use

Characteristics: Tasks in this cluster typically involve updates & modifications

Example PRs:

• #2100: Spread scheduled agentic workflows across 24 hours and add 6-hour schedules to s...
• #2103: Add smoke-outpost workflow for investigating failed smoke test runs
• #2109: Add semantic function refactoring workflow for Go code analysis

---

Cluster 1: Updates & Modifications

Size: 201 tasks (11.2% of total)

Success Metrics:

• Merged: 165 (82.1%)
• Closed (not merged): 34
• Open: 2

Complexity Metrics:

• Avg files changed: 18.2
• Avg additions: 771 lines
• Avg deletions: 381 lines
• Avg commits: 3.1
• Avg comments: 1.3
• Avg reviews: 1.1

Top Keywords: cli, version, code, changes, duplicate, update, analysis, duplicate code

Characteristics: Tasks in this cluster typically involve updates & modifications

Example PRs:

• #2127: Fix Smoke OpenCode workflow failure and update to version 0.15.13
• #2170: Update blog auditor to validate code snippet syntax against latest schema
• #2171: Refactor duplicate MCP code patterns for improved maintainability

---

Cluster 2: CI/CD & Workflows

Size: 109 tasks (6.1% of total)

Success Metrics:

• Merged: 71 (65.1%)
• Closed (not merged): 31
• Open: 7

Complexity Metrics:

• Avg files changed: 29.1
• Avg additions: 3011 lines
• Avg deletions: 1519 lines
• Avg commits: 4.6
• Avg comments: 4.7
• Avg reviews: 1.2

Top Keywords: mcp, server, mcp server, tool, github, safe, tools, json

Characteristics: Tasks in this cluster typically involve ci/cd & workflows

Example PRs:

• #2167: Fix OpenCode MCP server integration - Enable safe-outputs and GitHub tools
• #2219: Add tip about enabling agentic-workflows tool in MCP Server documentation
• #2255: Replace GITHUB_PERSONAL_ACCESS_TOKEN with GITHUB_MCP_SERVER_TOKEN in Copilot eng...

---

Cluster 5: Testing & Test Coverage

Size: 59 tasks (3.3% of total)

Success Metrics:

• Merged: 50 (84.7%)
• Closed (not merged): 8
• Open: 1

Complexity Metrics:

• Avg files changed: 14.7
• Avg additions: 558 lines
• Avg deletions: 95 lines
• Avg commits: 3.3
• Avg comments: 1.0
• Avg reviews: 0.8

Top Keywords: fix, tests, format, javascript, test, workflows, issues, reference

Characteristics: Tasks in this cluster typically involve testing & test coverage

Example PRs:

• #2153: Fix TestStopTimeResolutionIntegration to check for correct environment variable ...
• #2320: Fix test expectation for safe outputs MCP server name
• #2484: Fix nested quoting in awf compiler shell command generation

---

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files	Avg Commits	Top Keywords
5	Testing & Test Coverage	59	84.7%	14.7	3.3	fix, tests, format
1	Updates & Modifications	201	82.1%	18.2	3.1	cli, version, code
3	Updates & Modifications	216	81.5%	10.6	3.7	agentic, agentic workflow, workflow
6	Bug Fixes	398	75.4%	11.6	3.4	gh, gh aw, aw
4	New Features	806	71.8%	21.7	3.8	update, add, agent
2	CI/CD & Workflows	109	65.1%	29.1	4.6	mcp, server, mcp server

Sample Data Table

Sample of analyzed PRs with cluster assignments:

PR #	Title	Cluster	Outcome	Files	Commits	Keywords
#2127	Fix Smoke OpenCode workflow failure and update to ...	1 (Updates & Modifications)	Merged	2	4	cli, version
#2170	Update blog auditor to validate code snippet synta...	1 (Updates & Modifications)	Merged	2	5	cli, version
#2171	Refactor duplicate MCP code patterns for improved ...	1 (Updates & Modifications)	Merged	10	3	cli, version
#2208	Optimize CLI version checker workflow based on per...	1 (Updates & Modifications)	Merged	2	3	cli, version
#2216	Update cli-version-checker workflow: add node ecos...	1 (Updates & Modifications)	Merged	3	2	cli, version
#2167	Fix OpenCode MCP server integration - Enable safe-...	2 (CI/CD & Workflows)	Merged	3	3	mcp, server
#2219	Add tip about enabling agentic-workflows tool in M...	2 (CI/CD & Workflows)	Merged	1	2	mcp, server
#2255	Replace GITHUB_PERSONAL_ACCESS_TOKEN with GITHUB_M...	2 (CI/CD & Workflows)	Merged	32	2	mcp, server
#2257	Use --additional-mcp-config with valid JSON for Co...	2 (CI/CD & Workflows)	CLOSED	36	7	mcp, server
#2264	Pass MCP config as CLI argument instead of file fo...	2 (CI/CD & Workflows)	CLOSED	54	4	mcp, server
#2100	Spread scheduled agentic workflows across 24 hours...	3 (Updates & Modifications)	Merged	30	2	agentic, agentic workflow
#2103	Add smoke-outpost workflow for investigating faile...	3 (Updates & Modifications)	Merged	2	3	agentic, agentic workflow
#2109	Add semantic function refactoring workflow for Go ...	3 (Updates & Modifications)	Merged	3	4	agentic, agentic workflow
#2110	[WIP] Refactor clusters of functions in Go files	3 (Updates & Modifications)	CLOSED	0	1	agentic, agentic workflow
#2115	Add scheduling best practices guidance for daily w...	3 (Updates & Modifications)	Merged	1	3	agentic, agentic workflow
#2097	Add minimal path format syntax reference to import...	4 (New Features)	Merged	1	4	update, add
#2099	Add directory creation for copilot engine --add-di...	4 (New Features)	Merged	25	3	update, add
#2101	[WIP] Migrate JavaScript memory server to Wasm com...	4 (New Features)	CLOSED	0	1	update, add
#2102	Add workflow status badges documentation page	4 (New Features)	Merged	6	4	update, add
#2104	Add edit tool to commit-changes-analyzer workflow	4 (New Features)	Merged	2	2	update, add
#2153	Fix TestStopTimeResolutionIntegration to check for...	5 (Testing & Test Coverage)	Merged	1	2	fix, tests
#2320	Fix test expectation for safe outputs MCP server n...	5 (Testing & Test Coverage)	Merged	2	3	fix, tests
#2484	Fix nested quoting in awf compiler shell command g...	5 (Testing & Test Coverage)	Merged	29	5	fix, tests
#2505	[WIP] Investigate and fix warning about working-di...	5 (Testing & Test Coverage)	CLOSED	0	1	fix, tests
#2660	[WIP] Fix deprecated syntax issues in recompile ta...	5 (Testing & Test Coverage)	Merged	3	2	fix, tests
#2209	[WIP] Comment on issue #2157 regarding recurrence ...	6 (Bug Fixes)	Merged	3	3	gh, gh aw
#2235	Fix: Raise error when agentic workflow hits max-tu...	6 (Bug Fixes)	Merged	54	4	gh, gh aw
#2254	[WIP] Update GITHUB_PERSONAL_ACCESS_TOKEN to GITHU...	6 (Bug Fixes)	CLOSED	64	2	gh, gh aw
#2282	Extract 22 YAML generation functions from compiler...	6 (Bug Fixes)	CLOSED	4	5	gh, gh aw
#2283	Extract extraction functions from compiler.go to f...	6 (Bug Fixes)	Merged	6	4	gh, gh aw

Detailed Insights

1. Task Complexity Patterns

• Most Complex: CI/CD & Workflows (avg 29.1 files changed)
• Least Complex: Updates & Modifications (avg 10.6 files changed)

2. Success Rate Patterns

• Tasks related to testing & test coverage have the highest success rate (84.7%)
• Tasks related to ci/cd & workflows have the lowest success rate (65.1%)

3. Review and Collaboration Patterns

• Most reviewed: Updates & Modifications (avg 1.8 reviews)
• Least reviewed: Testing & Test Coverage (avg 0.8 reviews)

4. Task Distribution

Distribution of tasks across clusters:

• New Features: ██████████████████████ 806 (45.1%)
• Bug Fixes: ███████████ 398 (22.2%)
• Updates & Modifications: ██████ 216 (12.1%)
• Updates & Modifications: █████ 201 (11.2%)
• CI/CD & Workflows: ███ 109 (6.1%)
• Testing & Test Coverage: █ 59 (3.3%)

Recommendations

Based on the clustering analysis, here are actionable recommendations:

1. Leverage High-Success Patterns

Tasks in the Testing & Test Coverage cluster show 84.7% success rate.
Consider applying similar prompt patterns to other task types.

2. Improve Low-Success Clusters

The CI/CD & Workflows cluster has only 65.1% success rate.
Investigate common failure patterns and refine prompts or provide more context for these tasks.

3. Optimize for Complexity

Tasks in the CI/CD & Workflows cluster are most complex (avg 29.1 files).
Consider breaking down complex tasks into smaller, more manageable subtasks.

4. Focus on Dominant Task Types

New Features represents 45.1% of all tasks.
Optimizing prompts for this category will have the largest impact on overall success rates.

Methodology

NLP Techniques Used:

• Text Preprocessing: Cleaned PR bodies to extract original task prompts
• Feature Extraction: TF-IDF vectorization with 200 features, unigrams to trigrams
• Clustering Algorithm: K-means clustering with k=6 (determined via elbow method)
• Evaluation: Silhouette score for cluster quality assessment

Data Sources:

• GitHub PR data from githubnext/gh-aw repository
• PRs created by copilot-swe-agent
• Includes PR metadata, comments, reviews, and commit history

---

Generated by Prompt Clustering Analysis Workflow on 2025-12-10 19:23 UTC

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6165.
Fair winds, matey! 🏴‍☠️