Azure DevOps Copilot CLI Extension

A dual-purpose extension that integrates GitHub Copilot CLI into both Visual Studio Code and Azure DevOps pipelines for automated integration testing and enhanced software quality assurance.

🚀 Features

AI-Powered Integration Testing: Leverage GitHub Copilot CLI to generate comprehensive integration tests
Work Item Assistance: AI-powered work item analysis, rewriting, story point calculation, and backlog structuring
Intelligent Test Review: Automatically reviews and updates existing test cases to keep them current
GitHub Instructions Override: Customize test requirements using repository-specific instruction files
Multi-Framework Support: Intelligent support for Playwright, RestAssured, SpecFlow, pytest, and Go testify
Smart Application Discovery: Automatically analyzes Node.js, ASP.NET, Java, Python, and Docker applications
Dual Integration: Works as both VS Code extension and Azure DevOps pipeline task
Azure DevOps Pipeline Integration: Seamlessly integrate with your existing CI/CD workflows
Quality Gate Automation: Automated quality validation with configurable thresholds
Post-Deployment Validation: Validate deployments on LOCAL/INT/QA environments automatically

📦 Installation Options

Option 1: VS Code Extension

Install from the VS Code marketplace or manually:

From VS Code Marketplace: Search for "Azure DevOps Copilot CLI Extension"
Manual installation: Download the .vsix file and install using code --install-extension

Option 2: Azure DevOps Extension

Install in your Azure DevOps organization:

Go to Organization Settings → Extensions → Browse Marketplace
Search for "Azure DevOps Copilot CLI Extension"
Click Get it free and install to your organization
Or upload the Azure DevOps .vsix file manually

📋 Prerequisites

Node.js (v20 or higher)
PowerShell (v7.0 or higher) - for Azure DevOps tasks
GitHub Copilot CLI (npm install -g @github/copilot)
Azure DevOps account with pipeline access (for Azure DevOps integration)
GitHub Copilot subscription

🚀 Quick Start

VS Code Usage

Install the extension from VS Code marketplace
Install GitHub Copilot CLI:
```
npm install -g @github/copilot
```
Authenticate with GitHub:
```
export GITHUB_TOKEN="your_github_token"
```
Run integration tests:
- Open Command Palette (Ctrl+Shift+P)
- Run Azure DevOps Copilot: Run Integration Tests
- Select target environment

Azure DevOps Pipeline Usage

Add the Copilot CLI task to your Azure DevOps pipeline:

# azure-pipelines.yml
stages:
- stage: IntegrationTests
  jobs:
  - job: CopilotTests
    steps:
    - task: CopilotCLITask@1
      displayName: 'Run AI-Powered Integration Tests'
      inputs:
        testEnvironment: 'QA'
        copilotCliPath: 'npx'
        githubToken: '$(GITHUB_TOKEN)'
        qualityGateThreshold: '85'
        integrationTestTimeout: '300'
        failOnError: true

Advanced Pipeline Configuration

- task: CopilotCLITask@1
  displayName: 'Comprehensive Integration Testing'
  inputs:
    testEnvironment: 'INT'
    testSuite: 'api-tests'
    copilotCliPath: 'npx'
    githubToken: '$(GITHUB_TOKEN)'
    customPrompt: |
      Focus on API endpoint testing and database integration.
      Include performance benchmarks and security validation.
      Test error handling for edge cases.
    integrationTestTimeout: '600'
    qualityGateThreshold: '90'
    mcpServerUrl: 'https://mcp.example.com'
    workingDirectory: '$(System.DefaultWorkingDirectory)'
    failOnError: true

⚙️ Configuration

VS Code Settings

Configure through VS Code settings:

{
  "azdo-copilot.copilotCliPath": "npx",
  "azdo-copilot.githubToken": "your_github_token",
  "azdo-copilot.integrationTestTimeout": 300,
  "azdo-copilot.qualityGateThreshold": 80,
  "azdo-copilot.defaultEnvironment": "LOCAL",
  "azdo-copilot.mcpServerUrl": "https://mcp.example.com",
  "azdo-copilot.useGitHubInstructions": true,
  "azdo-copilot.githubInstructionFiles": [
    ".github/copilot-instructions.md",
    "copilot-instructions.md"
  ]
}

🔍 Intelligent Test Review

The extension automatically reviews and updates existing test cases when generating new integration tests:

Features

Automatic Discovery: Finds test files in common directories (tests, test, spec, __tests__)
Framework Detection: Identifies test frameworks (Jest, Mocha, Vitest, pytest, xUnit, etc.)
Outdated Test Detection: Flags tests that are old, incomplete, or using deprecated patterns
Safe Updates: Creates backup files before making any changes
Smart Enhancement: Adds missing test cases while preserving existing logic

Update Criteria

Tests are updated if they:

Are older than 1 week
Have insufficient coverage (< 3 test cases)
Lack API integration patterns
Target wrong environment
Use outdated code patterns

Supported Frameworks

JavaScript/TypeScript: Jest, Mocha, Vitest, Cypress, Playwright
Python: pytest, unittest
C#: xUnit, NUnit, MSTest
Java: JUnit

📝 GitHub Instructions Override

Customize test requirements using repository-specific instruction files to ensure consistency across your organization:

Setup

Create .github/copilot-instructions.md in your repository
Define your custom test requirements:

# Custom Test Requirements

- Test authentication using OAuth 2.0 flows
- Include performance benchmarks under 100ms
- Add comprehensive error handling scenarios
- Test file upload/download with large files
- Include security validation for all endpoints

Supported File Locations

.github/copilot-instructions.md (recommended)
.github/COPILOT_INSTRUCTIONS.md
copilot-instructions.md
.copilot-instructions.md
Custom paths via configuration

Configuration

{
  "azdo-copilot.useGitHubInstructions": true,
  "azdo-copilot.githubInstructionFiles": [
    ".github/copilot-instructions.md",
    "docs/test-requirements.md"
  ]
}

### Azure DevOps Task Inputs

| Input | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `testEnvironment` | pickList | Yes | LOCAL | Target environment (LOCAL/INT/QA/STAGING/PRODUCTION) |
| `testSuite` | string | No | - | Specific test suite to execute |
| `copilotCliPath` | string | Yes | npx | Path to GitHub Copilot CLI executable |
| `githubToken` | string | No | $(GITHUB_TOKEN) | GitHub personal access token |
| `customPrompt` | multiLine | No | - | Custom prompt to guide test generation |
| `integrationTestTimeout` | string | No | 300 | Test timeout in seconds |
| `qualityGateThreshold` | string | No | 80 | Quality gate threshold percentage |
| `mcpServerUrl` | string | No | - | MCP server URL for enhanced context |
| `workingDirectory` | filePath | No | $(System.DefaultWorkingDirectory) | Working directory for execution |
| `failOnError` | boolean | No | true | Fail build on test failures |

## 🔧 Setup for Azure DevOps

### 1. Install the Extension

1. Download the Azure DevOps extension from releases or build from source
2. Upload to your Azure DevOps organization via **Organization Settings** → **Extensions**
3. Install to your organization

### 2. Configure Authentication

Set up GitHub authentication in your pipeline:

```yaml
variables:
  - name: GITHUB_TOKEN
    value: 'your_github_token' # Use Azure DevOps secret variables

3. Agent Requirements

Ensure your build agents have:

Node.js v20+
PowerShell 7+
Internet access for GitHub Copilot CLI

4. Pipeline Integration Examples

Basic Integration Testing

- stage: Test
  jobs:
  - job: IntegrationTests
    steps:
    - task: CopilotCLITask@1
      inputs:
        testEnvironment: 'INT'
        copilotCliPath: 'npx'
        githubToken: '$(GITHUB_TOKEN)'

Multi-Environment Testing

- stage: Testing
  jobs:
  - job: QATests
    steps:
    - task: CopilotCLITask@1
      inputs:
        testEnvironment: 'QA'
        qualityGateThreshold: '90'
  
  - job: StagingTests
    dependsOn: QATests
    steps:
    - task: CopilotCLITask@1
      inputs:
        testEnvironment: 'STAGING'
        qualityGateThreshold: '95'

🎯 Output Variables

The Azure DevOps task sets the following output variables for use in subsequent tasks:

CopilotTestsPassed - Number of tests that passed
CopilotTestsFailed - Number of tests that failed
CopilotQualityScore - Overall quality score percentage
CopilotTestSuccess - Boolean indicating overall success

Using Output Variables

- task: CopilotCLITask@1
  name: 'CopilotTests'
  inputs:
    testEnvironment: 'QA'

- script: |
    echo "Tests passed: $(CopilotTestsPassed)"
    echo "Quality score: $(CopilotQualityScore)%"
  displayName: 'Display test results'

🧪 How It Works

Project Detection: Automatically analyzes your workspace (Node.js, Java, Python, .NET, Go)
Test Review: Scans existing test files and identifies outdated or incomplete tests
GitHub Instructions: Checks for custom test requirements in repository instruction files
Framework Detection: Identifies test frameworks (Playwright, Jest, MSTest, pytest, etc.)
Test Generation: Uses GitHub Copilot CLI to create environment-specific tests with custom requirements
Test Enhancement: Updates existing tests with missing patterns and modern practices
Environment Configuration: Loads API URLs from OpenAPI specs or config files
Test Execution: Runs tests with real-time progress tracking and timeout management
Quality Validation: Provides quality scores and actionable insights
Result Reporting: Sets Azure DevOps variables and build status

📁 Project Structure

├── vss-extension.json     # Azure DevOps extension manifest
├── package.json          # VS Code extension configuration
├── azure-pipelines.yml   # CI/CD pipeline for the extension
├── task/                 # Azure DevOps task definition
│   ├── task.json         # Task metadata and inputs
│   └── task.ps1          # PowerShell execution script
├── src/                  # VS Code extension source
│   ├── extension.ts      # Core extension implementation
│   ├── mcp-client.ts     # MCP integration
│   └── utils/
│       └── exec.ts       # Command execution utilities
└── config/
    ├── environments.json # Environment configurations
    └── example-openapi.yaml # OpenAPI specification template

🔍 Troubleshooting

Common Issues

GitHub Copilot CLI not found

npm install -g @github/copilot

GitHub authentication failed

# Option 1: Environment variable
export GITHUB_TOKEN="your_token"

# Option 2: GitHub CLI
gh auth login

# Option 3: Azure DevOps variable
# Set GITHUB_TOKEN in pipeline variables

PowerShell execution policy (Windows)

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

Debug Mode

Enable verbose logging by setting the Azure DevOps variable:

variables:
  system.debug: true

🤝 Contributing

Fork the repository
Create a feature branch: git checkout -b feature/amazing-feature
Commit changes: git commit -m 'Add amazing feature'
Push to branch: git push origin feature/amazing-feature
Open a Pull Request

🚀 Feature Enhancement

Have ideas for new features or improvements? We'd love to hear from you!

Contact us for feature enhancements: info@jariyah.us

Whether you need custom integrations, additional test frameworks, or specialized Azure DevOps workflows, our team can help enhance this extension to meet your specific requirements.

⭐ Rate & Review

Don't forget to send rating and review for this extension!

If this extension has helped improve your development workflow and Azure DevOps pipelines, please consider:

⭐ Rating it on VS Code Marketplace - Help others discover this tool
⭐ Rating it on Azure DevOps Marketplace - Share your experience with the community
� Writing a review - Let others know how it's working for you
🐛 Reporting issues - Help us improve by sharing any bugs or suggestions

Your feedback helps us continue to improve and helps other developers find useful tools!

�📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📚 Documentation

VS Code Extension Guide - VS Code specific documentation
Configuration Templates - Environment and OpenAPI examples
Publishing Guide - Extension publishing instructions
Changelog - Version history and updates

🔗 Related Links

🚀 Ready to supercharge your Azure DevOps pipelines with AI-powered testing!

🔧 Key Commands

Run Integration Tests: Execute AI-generated integration tests with automatic test review
Work Item Assistance: AI-powered work item analysis and improvement
- Fetch work items directly from Azure DevOps using work item IDs
- Rewrite work items for clarity and testability
- Auto-calculate story points with detailed reasoning
- Suggest missing subtasks and dependencies
- Convert business requirements to structured backlog items
Test Configuration: Test configuration dialogs and GitHub instruction detection
Setup Pipeline Task: Generate Azure DevOps pipeline configurations
Configure MCP Server: Set up Model Context Protocol integration
Validate Deployment: Perform post-deployment validation