[prompt-clustering] Copilot Agent Prompt Clustering Analysis - 2025-12-11

[github-actions[bot]]

🔬 Copilot Agent Prompt Clustering Analysis

Analysis Date: 2025-12-11

Summary

Performed NLP-based clustering analysis on 986 copilot agent task prompts from the last 30 days using TF-IDF vectorization and K-means clustering. Identified 9 distinct clusters representing different types of tasks with an overall success rate of 77.2%.

Key Findings

• Most Common Task Type: Testing & Quality (287 tasks, 29.1%)
• Highest Success Rate: CI/CD & Workflows (86.0% success rate)
• Lowest Success Rate: Bug Fixes (67.3% success rate)
• Most Complex Tasks: Feature Implementation (avg 1613 lines added)

Full Clustering Analysis Report

Cluster Visualization

2D visualization of task prompts using PCA dimensionality reduction. Each color represents a distinct cluster.

Detailed Cluster Analysis

Cluster 7: Testing & Quality

Size: 287 tasks (29.1% of total)

Success Rate: 75.6% (217 merged)

Top Keywords: update, add, error, comment, pull, job

Average Complexity Metrics:

• Files changed: 18.2
• Lines added: 601
• Comments: 1.5
• Reviews: 1.6

Representative Examples:

• #3917: Replace activation job checkout with GitHub API for timestamp checking
• #2400: Fix patch generation when agent provides non-existent branch name

---

Cluster 1: General Tasks

Size: 140 tasks (14.2% of total)

Success Rate: 72.9% (102 merged)

Top Keywords: cli, firewall, mcp, version, logs, aw

Average Complexity Metrics:

• Files changed: 15.6
• Lines added: 522
• Comments: 1.6
• Reviews: 1.7

Representative Examples:

• #2253: Add OIDC authentication with API key fallback - enabled by default for Claude
• #2318: [WIP] Add daily test coverage improver workflow with firewall enabled

---

Cluster 3: CI/CD & Workflows

Size: 121 tasks (12.3% of total)

Success Rate: 86.0% (104 merged)

Top Keywords: pkg, pkg workflow, functions, workflow, code, function

Average Complexity Metrics:

• Files changed: 10.7
• Lines added: 504
• Comments: 1.5
• Reviews: 2.0

Representative Examples:

• #2879: Implement JavaScript bundler with on-demand bundling and caching for embedded sources
• #3503: Refactor: Extract duplicate agent output handler boilerplate

---

Cluster 6: CI/CD & Workflows

Size: 110 tasks (11.2% of total)

Success Rate: 77.3% (85 merged)

Top Keywords: workflows, github, github workflows, md, gh, workflow

Average Complexity Metrics:

• Files changed: 17.8
• Lines added: 592
• Comments: 0.9
• Reviews: 2.0

Representative Examples:

• #3779: Implement golden file testing for compiler output validation
• #4003: Add /cloclo command workflow with Claude engine and MCP integrations

---

Cluster 8: Feature Implementation

Size: 90 tasks (9.1% of total)

Success Rate: 80.0% (72 merged)

Top Keywords: agentic workflow, agentic, workflow, update, create, add

Average Complexity Metrics:

• Files changed: 5.8
• Lines added: 1613
• Comments: 1.9
• Reviews: 2.0

Representative Examples:

• #4086: Add shared github-context.md import for comprehensive GitHub invocation context
• #2813: Add Ollama Llama Guard 3 threat scanning for safe outputs

---

Cluster 2: CI/CD & Workflows

Size: 73 tasks (7.4% of total)

Success Rate: 76.7% (56 merged)

Top Keywords: agent, agentic workflows, agentic, workflows, github, copilot

Average Complexity Metrics:

• Files changed: 12.7
• Lines added: 1070
• Comments: 1.9
• Reviews: 2.0

Representative Examples:

• #2430: [WIP] Add firewall feature to all agentic workflows
• #3666: Add repository quality improvement workflow with focus area rotation

---

Cluster 5: Bug Fixes

Size: 67 tasks (6.8% of total)

Success Rate: 74.6% (50 merged)

Top Keywords: schema, json, pkg, error, field, validation

Average Complexity Metrics:

• Files changed: 7.5
• Lines added: 274
• Comments: 1.2
• Reviews: 2.1

Representative Examples:

• #3927: Remove action_pins.json and resolve SHAs dynamically at compile time
• #4045: Add validation error summary with category grouping and severity sorting

---

Cluster 4: Documentation Updates

Size: 49 tasks (5.0% of total)

Success Rate: 85.7% (42 merged)

Top Keywords: docs, documentation, md, reference, content, update

Average Complexity Metrics:

• Files changed: 5.4
• Lines added: 291
• Comments: 1.3
• Reviews: 1.8

Representative Examples:

• #4088: Add daily documentation testing workflow with beginner perspective
• #2987: Add troubleshooting documentation structure

---

Cluster 9: Bug Fixes

Size: 49 tasks (5.0% of total)

Success Rate: 67.3% (33 merged)

Top Keywords: comments, issuetitle, issue, section, issuedescription, author

Average Complexity Metrics:

• Files changed: 19.6
• Lines added: 541
• Comments: 2.6
• Reviews: 1.6

Representative Examples:

• #3505: Add create-commit-status safe output type with pending/final status lifecycle
• #2996: Use BurntSushi/toml encoder for Codex engine TOML configuration generation

---

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Lines	Top Keywords
3	CI/CD & Workflows	121	86.0%	504	pkg, pkg workflow, functions
4	Documentation Updates	49	85.7%	291	docs, documentation, md
8	Feature Implementation	90	80.0%	1613	agentic workflow, agentic, workflow
6	CI/CD & Workflows	110	77.3%	592	workflows, github, github workflows
2	CI/CD & Workflows	73	76.7%	1070	agent, agentic workflows, agentic
7	Testing & Quality	287	75.6%	601	update, add, error
5	Bug Fixes	67	74.6%	274	schema, json, pkg
1	General Tasks	140	72.9%	522	cli, firewall, mcp
9	Bug Fixes	49	67.3%	541	comments, issuetitle, issue

Sample Data (50 Most Recent PRs)

PR #	Title	Cluster	Theme	Outcome	Files	Lines	Keywords
#4247	[WIP] Update agentic workflows to use shared agent...	4	Documentation Updates	✅	6	63	docs, documentation
#4244	Extract Copilot PR data fetching into reusable sha...	4	Documentation Updates	✅	3	102	docs, documentation
#4239	Update Node.js version check from 24 to 20 for Git...	7	Testing & Quality	✅	3	7	update, add
#4238	Add git fallback for update command when GitHub AP...	7	Testing & Quality	✅	12	1157	update, add
#4237	Fix nested error rendering with Python-style visua...	2	CI/CD & Workflows	❌	3	260	agent, agentic workflows
#4236	Refactor ALL_TOOLS to separate JSON file with runt...	5	Bug Fixes	✅	80	3061	schema, json
#4235	[WIP] Refactor ALL_TOOLS JSON array into separate ...	5	Bug Fixes	❌	0	0	schema, json
#4234	Preserve YAML formatting in frontmatter field upda...	5	Bug Fixes	✅	2	485	schema, json
#4233	Change update command to override local changes by...	5	Bug Fixes	✅	2	137	schema, json
#4232	Document strict mode enforcement areas and CLI fla...	8	Feature Implementation	✅	3	76	agentic workflow, agentic
#4231	Add documentation for string sanitization vs norma...	3	CI/CD & Workflows	✅	5	444	pkg, pkg workflow
#4230	Remove duplicate formatFileSize() function in pkg/...	3	CI/CD & Workflows	✅	1	8	pkg, pkg workflow
#4224	Add comprehensive strict mode reference documentat...	6	CI/CD & Workflows	✅	3	71	workflows, github
#4223	Eliminate duplicate MCP tool table rendering logic	3	CI/CD & Workflows	✅	4	389	pkg, pkg workflow
#4221	Add dedicated integration test job to CI workflow	5	Bug Fixes	✅	1	36	schema, json
#4220	Add integration tests for GitHub MCP server config...	5	Bug Fixes	✅	1	276	schema, json
#4219	Add integration tests for playwright MCP configura...	5	Bug Fixes	✅	1	188	schema, json
#4218	Fix JavaScript test assertions for loadAgentOutput...	5	Bug Fixes	✅	2	2	schema, json
#4217	Add Node.js 24+ requirement with Makefile validati...	5	Bug Fixes	✅	2	34	schema, json
#4216	Fix make lint to auto-install dependencies	9	Bug Fixes	❌	1	16	comments, issuetitle
#4214	Fix safe-output jobs failing on agent output parse...	7	Testing & Quality	✅	63	172	update, add
#4211	Update github.com/modelcontextprotocol/go-sdk from...	9	Bug Fixes	✅	3	18	comments, issuetitle
#4210	Standardize CLI workflow identifier terminology to...	7	Testing & Quality	✅	5	47	update, add
#4206	Update github.com/stretchr/testify from v1.8.1 to ...	9	Bug Fixes	✅	2	3	comments, issuetitle
#4205	Fix Playwright version confusion between MCP packa...	7	Testing & Quality	✅	16	48	update, add
#4204	Fix help text formatting: remove dashes, align spa...	7	Testing & Quality	✅	2	5	update, add
#4203	Isolate test temp directories to prevent conflicts	9	Bug Fixes	✅	274	1528	comments, issuetitle
#4202	Add deprecated field detection to strict mode vali...	8	Feature Implementation	✅	6	490	agentic workflow, agentic
#4201	Migrate to local prettier installation using npm s...	5	Bug Fixes	✅	3	26	schema, json
#4180	Add dev-deps command and migrate prettier to local...	5	Bug Fixes	❌	7	324	schema, json
#4179	Fix CHANGELOG v0.21.0: discussion field is optiona...	8	Feature Implementation	✅	1	1	agentic workflow, agentic
#4178	Add formal deprecation policy documentation	6	CI/CD & Workflows	❌	2	307	workflows, github
#4169	Add schema versioning infrastructure with v1.0.0	8	Feature Implementation	❌	5	365	agentic workflow, agentic
#4168	Add Node.js prerequisite check to make deps target	9	Bug Fixes	❌	4	35	comments, issuetitle
#4161	Standardize interface{} to any syntax across c...	3	CI/CD & Workflows	✅	17	90	pkg, pkg workflow
#4160	Add semantic types to constants for type safety an...	8	Feature Implementation	✅	18	172	agentic workflow, agentic
#4159	Consolidate duplicate GitHub tools lists into shar...	3	CI/CD & Workflows	✅	2	36	pkg, pkg workflow
#4158	Add unified ToolsConfig struct to replace map[stri...	3	CI/CD & Workflows	✅	12	336	pkg, pkg workflow
#4156	Replace untyped maps with strongly-typed SafeOutpu...	8	Feature Implementation	❌	2	539	agentic workflow, agentic
#4150	Extract DomainBuckets to eliminate duplicate acces...	3	CI/CD & Workflows	✅	9	197	pkg, pkg workflow
#4146	Remove deprecated displayMissingToolsAnalysis func...	8	Feature Implementation	✅	4	83	agentic workflow, agentic
#4144	docs: add copilot instructions file for convention...	5	Bug Fixes	❌	1	181	schema, json
#4142	Use GitHub API for lock file timestamp checks inst...	5	Bug Fixes	✅	87	4709	schema, json
#4141	Update actions/github-script to v8 in dev and test...	2	CI/CD & Workflows	✅	4	4	agent, agentic workflows
#4140	Fix template injection vulnerabilities in cloclo w...	2	CI/CD & Workflows	✅	2	6	agent, agentic workflows
#4139	Add missing issues and pull-requests read permissi...	2	CI/CD & Workflows	✅	2	6	agent, agentic workflows
#4130	Standardize error formatting with console package ...	8	Feature Implementation	❌	9	188	agentic workflow, agentic
#4129	Generate zizmor annotations for workflow_run trigg...	5	Bug Fixes	✅	8	257	schema, json
#4128	Configure Q workflow to skip PR creation when no c...	5	Bug Fixes	✅	2	28	schema, json
#4126	Optimize daily-team-status workflow with data pre-...	1	General Tasks	✅	3	1622	cli, firewall

Insights & Recommendations

1. Documentation Tasks Have Highest Success Rate

Documentation-related tasks achieve 85.7% success rate with relatively low complexity (avg 291 lines). Recommendation: Documentation tasks are ideal candidates for copilot agents.

2. Task Complexity Varies Significantly

Task complexity ranges from 274 lines (Bug Fixes) to 1613 lines (Feature Implementation). Recommendation: Break down complex tasks into smaller, focused subtasks.

3. Testing & Quality Tasks Are Most Common

The largest cluster (Testing & Quality) contains 29.1% of all tasks. Recommendation: Invest in improving prompt templates and best practices for this category.

4. Review Engagement Varies by Task Type

Review engagement varies from 1.6 (Bug Fixes) to 2.1 reviews (Bug Fixes). Recommendation: Standardize review processes across task types.

5. Prompt Engineering Opportunities

3 clusters have success rates below 75%: General Tasks, Bug Fixes, Bug Fixes. Recommendation: Analyze failed PRs in these clusters to identify common issues and improve prompt templates.

---

Methodology: Analyzed 986 copilot-created PRs using NLP techniques (TF-IDF vectorization, K-means clustering with k=9). Prompts extracted from PR bodies, cleaned, and clustered based on semantic similarity.

Analysis Period: Last 30 days

Generated: 2025-12-11 19:25:35 UTC

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6291.
Fair winds, matey! 🏴‍☠️