[prompt-clustering] Copilot Agent Prompt Clustering Analysis - January 28, 2026

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts using machine learning techniques to identify patterns and optimize agent performance.

Summary

Analysis Period: 2025-10-22 to 2026-01-28 (3+ months)
Total Tasks Analyzed: 4,598
Clusters Identified: 7
Overall Success Rate: 70.4%
Merged PRs: 3,237
Closed PRs: 1,361

Key Findings

1. Most Common Task Type: General workflow tasks (1,377 tasks, 29.9% of total)
2. Highest Success Rate: Bug Fixes (Cluster add cli flag to guard dropping a agentic workflow instructinos file #6) with 77.3% success rate
3. Performance Gap: 13.6 percentage point difference between best and worst performing clusters
4. Complexity: Feature implementation tasks involve the most file changes (avg 26-29 files)

Cluster Analysis Results

Success Rate by Task Type

Cluster	Theme	Size	% of Total	Success Rate	Avg Files	Top Keywords
#6	Bug Fixes	1,343	29.2%	77.3%	23.5	fix, reference, update
#1	General	1,377	29.9%	69.3%	15.1	workflow, pkg, gh
#3	Updates	586	12.7%	69.3%	13.8	agentic, workflow, workflows
#2	Documentation	334	7.3%	65.9%	25.2	agent, copilot, docs
#7	Features	324	7.0%	65.1%	28.6	safe, output, create
#5	Features	383	8.3%	64.8%	26.1	mcp, server, configuration
#4	Bug Fixes	251	5.5%	63.7%	14.0	campaign, issue, run

View Detailed Cluster Descriptions

Cluster #1: General Workflow Tasks (29.9%)

Keywords: workflow, pkg, gh, aw, gh aw, files, test, cli, issue, workflows

Tasks related to general workflow management, package maintenance, and CLI operations. This is the largest category, representing nearly 30% of all tasks.

• Success Rate: 69.3% (954 merged, 423 closed)
• Complexity: Medium (avg 15.1 files, +585/-431 lines)
• Characteristics: Broad category covering various workflow-related tasks

Example Prompts:

• "Fix missing OPENAI_API_KEY in GenAIScript smoke test"
• "Optimize CLI version checker workflow based on performance analysis"
• "Investigate workflow compilation failures"

---

Cluster #6: Bug Fixes - High Success (29.2%)

Keywords: fix, reference, update, add, job, github, actions, file, step, use

Bug fix tasks with the highest success rate. These tasks focus on fixing specific issues, updating references, and correcting errors.

• Success Rate: 77.3% ✨ (1,038 merged, 305 closed)
• Complexity: Medium-High (avg 23.5 files, +689/-1,157 lines)
• Characteristics: Clear problem statements, specific fixes, often includes deletions

Example Prompts:

• "Update the frontmatter imports documentation with all supported URL and path formats"
• "Add directory creation for copilot engine --add-dir paths"
• "Create a nodejs JavaScript script that generates documentation with status badges"

---

Cluster #3: Updates & Modifications (12.7%)

Keywords: agentic, workflow, workflows, agent, update, add, create, daily, issue, github

Tasks focused on updating existing functionality, modifying workflows, and creating scheduled jobs.

• Success Rate: 69.3% (406 merged, 180 closed)
• Complexity: Medium (avg 13.8 files, +1,103/-326 lines)
• Characteristics: More additions than deletions, often scheduled/automated tasks

Example Prompts:

• "Review scheduled agentic workflows and spread them the entire day"
• "Add smoke-outpost workflow for investigating failed smoke test runs"
• "Update workflow schedules to run every 6 hours"

---

Cluster #5: Feature Implementation - MCP/Server (8.3%)

Keywords: mcp, server, configuration, github, add, tools, support, client, agent, features

Feature development tasks focusing on MCP (Model Context Protocol) servers, configuration, and new capabilities.

• Success Rate: 64.8% (248 merged, 135 closed)
• Complexity: High (avg 26.1 files, +1,172/-459 lines)
• Characteristics: Complex implementations, many file changes, architectural work

Example Prompts:

• "Add MCP server configuration support"
• "Implement new GitHub tools for agent"
• "Create client support for new features"

---

Cluster #2: Documentation (7.3%)

Keywords: agent, copilot, docs, make, mcp, agentic, workflow, update, reference, add

Documentation-related tasks including guides, references, and explanatory content.

• Success Rate: 65.9% (220 merged, 114 closed)
• Complexity: Medium-High (avg 25.2 files, +927/-437 lines)
• Characteristics: Often touches many files for consistency

Example Prompts:

• "Create documentation for agent features"
• "Update MCP reference documentation"
• "Add guide for agentic workflow setup"

---

Cluster #7: Feature Implementation - Safe Outputs (7.0%)

Keywords: safe, output, create, add, update, workflow, github, issue, data, code

Feature development tasks focused on safe outputs, data handling, and issue management.

• Success Rate: 65.1% (211 merged, 113 closed)
• Complexity: Highest (avg 28.6 files, +1,149/-447 lines)
• Characteristics: Most complex tasks, many architectural changes

Example Prompts:

• "Create safe output mechanisms for workflow data"
• "Add issue creation with safe outputs"
• "Implement data validation for outputs"

---

Cluster #4: Bug Fixes - Campaign/Security (5.5%)

Keywords: campaign, issue, run, security, fix, workflow, github, audit, analysis, create

Bug fixes related to campaigns, security issues, audits, and workflow runs.

• Success Rate: 63.7% (160 merged, 91 closed)
• Complexity: Medium (avg 14.0 files, +490/-300 lines)
• Characteristics: Security-focused, audit-related tasks

Example Prompts:

• "Fix security audit workflow issues"
• "Resolve campaign run failures"
• "Address security analysis findings"

Insights & Patterns

What Works Well ✅

1. Bug Fix Tasks (Cluster add cli flag to guard dropping a agentic workflow instructinos file #6) show the highest success rate at 77.3%

   • Clear problem statements
   • Specific, well-defined scope
   • Often include context from error logs or failed runs

2. Medium Complexity tasks (13-24 files) have good success rates

   • Not too broad, not too narrow
   • Clear boundaries

3. Tasks with Concrete Examples tend to succeed more often

   • Reference to specific files or paths
   • Error messages or logs included
   • Expected behavior clearly stated

What Needs Improvement ⚠️

1. Feature Implementation Tasks (Clusters Add workflow: githubnext/agentics/weekly-research #5, Weekly Research Report: AI Workflow Automation Landscape and Market Opportunities - August 2025 #7) have lower success rates (64-65%)

   • Often too broad or vague
   • Lack architectural context
   • Multiple interpretation possibilities
   • High complexity (26-29 files changed)

2. Campaign/Security Tasks (Cluster Add workflow: githubnext/agentics/weekly-research #4) have the lowest success rate (63.7%)

   • May require domain-specific knowledge
   • Often involve multiple systems
   • Security constraints may limit options

3. Large Tasks with many file changes show lower success

   • Tasks touching 25+ files have ~5% lower success rate
   • May need better decomposition

Recommendations

1. Optimize Prompt Patterns

For Bug Fixes (continue current approach):

• ✅ Include error messages and logs
• ✅ Reference specific files and line numbers
• ✅ Provide clear expected vs actual behavior
• ✅ Include reproduction steps if applicable

For Feature Implementation (needs improvement):

• 📝 Break down large features into smaller, incremental tasks
• 📝 Provide architectural context and design patterns
• 📝 Include examples from similar existing features
• 📝 Specify acceptance criteria clearly
• 📝 Consider proof-of-concept first, then full implementation

For Documentation (moderate improvements needed):

• 📝 Specify target audience and detail level
• 📝 Provide structure/outline for complex docs
• 📝 Reference existing documentation style guide

2. Task Decomposition Strategy

Based on file change analysis:

• Tasks touching 15-20 files: Optimal complexity, keep as-is
• Tasks touching 25+ files: Consider splitting into 2-3 subtasks
• Tasks with high additions (+1000 lines): Break into phases

3. Improve Security/Campaign Tasks

For Cluster #4 (lowest success rate):

• Provide more security context and constraints upfront
• Include links to security documentation
• Specify compliance requirements explicitly
• Consider specialized prompts for audit-related tasks

4. Quality Metrics to Track

Monitor these metrics going forward:

• Success rate by cluster over time
• Average file changes per cluster
• Time to merge (when available)
• Revision count before merge

5. Prompt Engineering Guidelines

Create specific guidelines for each task type:

Bug Fix Template:

Issue: [specific error or problem]
Context: [where/when it occurs]
Expected: [correct behavior]
Actual: [current behavior]
Logs/Errors: [relevant output]
Files: [suspected files]
``````

**Feature Template**:
``````
Goal: [what capability to add]
Use Case: [why/when it's needed]
Example: [similar existing feature]
Scope: [what's in/out of scope]
Architecture: [where it fits]
Acceptance: [how to verify success]

Data Quality Notes

• Coverage: 4,598 PRs analyzed (excellent sample size)
• Time Range: 3+ months of data (October 2025 - January 2026)
• Clustering Quality: Silhouette analysis identified 7 as optimal cluster count
• Feature Engineering: TF-IDF vectorization with 100 features, bigrams included
• Success Definition: Merged PRs considered successful, closed without merge considered unsuccessful

Next Steps

1. Immediate: Apply recommendations to new prompts
2. Short-term: Create prompt templates for each task type
3. Medium-term: A/B test prompt improvements
4. Long-term: Build automated prompt quality checker

Technical Details

Methodology

Data Collection

• Source: All copilot-codeagent[bot] PRs from githubnext/gh-aw
• Period: 2025-10-22 to 2026-01-28
• Sample: 4,598 PRs with valid task prompts
• Extraction: Prompts extracted from PR body "Original prompt" sections

NLP Processing

• Vectorization: TF-IDF with 100 max features
• N-grams: Unigrams and bigrams (1-2 words)
• Min Document Frequency: 2 (term must appear in at least 2 documents)
• Max Document Frequency: 80% (exclude too-common terms)
• Stop Words: English stop words removed

Clustering Algorithm

• Method: K-means clustering
• K Selection: Silhouette score optimization (tested k=2 to k=10)
• Optimal K: 7 clusters
• Initialization: k-means++ with 10 random starts
• Feature Space: 100-dimensional TF-IDF vectors

Cluster Interpretation

• Keywords: Top 5-10 TF-IDF terms per cluster
• Theme Assignment: Automated based on keyword analysis
• Validation: Manual review of sample PRs per cluster

Metrics Calculated

• Success rate (merged / total)
• Average files changed
• Average lines added/deleted
• Cluster size and distribution

---

Full Dataset: Complete analysis with all 4,598 PRs available upon request (643KB markdown file)

Generated: 2026-01-28 by Prompt Clustering Analysis Agent
Repository: githubnext/gh-aw
Workflow Run: §21427537536

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 4, 2026, 6:44 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-04T06:44:36.436Z.