Skip to content

[Epic] 🔍 MCP Compliance Checker - Automated Specification Testing Tool #1286

New issue
New issue
Open
Feature
Open
[Epic] 🔍 MCP Compliance Checker - Automated Specification Testing Tool#1286
Feature
Assignees
crivetimihai
Labels
mcp-2025-06-18MCP 2025-06-18 protocol specificationmcp-protocolAlignment with MCP protocol or specificationtestingTesting (unit, e2e, manual, automated, etc)
Milestone
Release 1.0.0
@crivetimihai

Description

@crivetimihai
crivetimihai
opened on Oct 18, 2025

🔍 MCP Compliance Checker - Automated Specification Testing Tool

Goal

Create a comprehensive automated testing tool that validates MCP (Model Context Protocol) implementations against the official specification. The tool will:

  1. Test MCP servers for full spec compliance with detailed reports
  2. Test MCP clients using a 100% spec-compliant reference server
  3. Support multiple spec versions (2025-06-18 initially, configurable)
  4. Generate actionable reports in multiple formats (console, JSON, HTML, Markdown)
  5. Enable CI/CD integration for continuous compliance monitoring

This ensures MCP implementations remain spec-compliant and makes it easy to identify and fix conformance issues.

Why Now?

MCP is a rapidly evolving protocol with strict requirements for interoperability:

  1. Compliance Gaps Discovered: Manual review found 32+ compliance issues in ContextForge ([Epic]: Fully implement MCP 2025-06-18 compliance across all endpoints #1285)
  2. Interoperability Critical: MCP clients/servers from different vendors must work together
  3. Spec Evolution: New spec versions (2025-06-18) introduce breaking changes
  4. Manual Testing Insufficient: 300+ test points across protocol, types, transports, field naming
  5. Developer Productivity: Automated testing catches issues in seconds vs hours of manual verification
  6. Quality Assurance: Reference server provides gold standard for testing clients

Without automated compliance testing, spec violations go undetected until runtime failures occur with third-party MCP implementations.

📖 User Stories

US-1: Developer - Test MCP Server Compliance

As a MCP Server Developer
I want to automatically test my server against the MCP specification
So that I can ensure compatibility with all MCP clients

Acceptance Criteria:

Given I have an MCP server running on http://localhost:4444/rpc
When I run:
  mcp-compliance-checker server http://localhost:4444/rpc --spec 2025-06-18

Then the tool should:
  - Test protocol lifecycle (initialize, ping)
  - Test all advertised capabilities (tools, resources, prompts)
  - Validate field naming (camelCase vs snake_case)
  - Check type definitions (ImageContent, ToolResult, etc.)
  - Test pagination (cursor, nextCursor)
  - Verify transport compliance (HTTP, SSE)
  - Generate compliance report with pass/fail status

When all tests pass
Then I see:
  "✅ Overall Compliance: 100% (47/47 tests passed)"
  - Exit code 0

When tests fail
Then I see:
  "❌ Overall Compliance: 68% (32/47 tests passed)"
  - List of failed tests with file locations and fix suggestions
  - Exit code 1

Technical Requirements:

  • Test runner that executes all spec-defined tests
  • Authentication support (Bearer token, Basic auth)
  • Timeout handling (configurable per test)
  • Detailed error messages with line numbers
  • Exit codes for CI/CD integration
US-2: Developer - Test Against Specific Spec Version

As a MCP Server Developer
I want to test my server against different spec versions
So that I can verify backward compatibility and plan migrations

Acceptance Criteria:

Given the tool supports spec versions: 2025-06-18, 2025-03-26

When I test against current spec:
  mcp-compliance-checker server http://localhost:4444/rpc --spec 2025-06-18
Then the tool uses requirements from specs/2025-06-18.yaml

When I test against older spec:
  mcp-compliance-checker server http://localhost:4444/rpc --spec 2025-03-26
Then the tool uses requirements from specs/2025-03-26.yaml

When I test without specifying version:
  mcp-compliance-checker server http://localhost:4444/rpc
Then the tool defaults to latest spec (2025-06-18)

When I specify an unsupported spec:
  mcp-compliance-checker server http://localhost:4444/rpc --spec 2024-01-01
Then I see error:
  "❌ Spec version 2024-01-01 not found. Available: 2025-06-18, 2025-03-26"

Technical Requirements:

  • YAML-based spec configuration in specs/ directory
  • Version detection and selection logic
  • Error handling for missing spec versions
  • Documentation of supported spec versions
US-3: Developer - Generate Compliance Reports

As a MCP Server Developer
I want to generate compliance reports in different formats
So that I can share results with my team and stakeholders

Acceptance Criteria:

Given I have tested my server

When I generate a console report:
  mcp-compliance-checker server http://localhost:4444/rpc
Then I see:
  - Color-coded output (green ✅, yellow ⚠️, red ❌)
  - Summary table with pass/fail counts
  - Critical issues highlighted
  - Overall compliance percentage

When I generate a JSON report:
  mcp-compliance-checker server http://localhost:4444/rpc 
    --output-format json --output-file compliance.json
Then I get:
  - Machine-readable JSON with test results
  - Structured issue list with severity levels
  - Metadata (timestamp, target URL, spec version)
  - Suitable for CI/CD parsing

When I generate an HTML report:
  mcp-compliance-checker server http://localhost:4444/rpc 
    --output-format html --output-file report.html
Then I get:
  - Standalone HTML file with embedded CSS
  - Interactive tables (sortable, filterable)
  - Color-coded test results
  - Shareable via email/web

When I generate a Markdown report:
  mcp-compliance-checker server http://localhost:4444/rpc 
    --output-format markdown --output-file compliance.md
Then I get:
  - GitHub-flavored markdown
  - Suitable for README or PR descriptions
  - Includes summary tables
  - Links to spec documentation

Technical Requirements:

  • Console reporter using rich library (colored, formatted)
  • JSON reporter with structured output
  • HTML reporter with Jinja2 templates
  • Markdown reporter for documentation
  • File output support (--output-file)

🏗 Architecture

Directory Structure

tools/
├── mcp-compliance-checker/         # Python compliance checker
│   ├── specs/
│   │   ├── 2025-06-18.yaml        # Current spec definition
│   │   └── 2025-03-26.yaml        # Legacy spec
│   ├── tests/
│   │   ├── server/                 # Server compliance tests
│   │   ├── client/                 # Client compliance tests
│   │   ├── transport/              # Transport tests
│   │   └── types/                  # Type validation tests
│   ├── reporters/
│   │   ├── console.py              # Rich console output
│   │   ├── json.py
│   │   ├── html.py
│   │   └── markdown.py
│   ├── checker.py                  # Main orchestrator
│   └── cli.py                      # Click CLI
│
└── mcp-reference-server/           # Go reference server
    ├── main.go
    └── handlers/

📋 Implementation Tasks

Phase 1: Core Infrastructure ✅

  • Project setup with Poetry/UV
  • Spec loader (YAML parser)
  • CLI framework (Click)
  • Test base classes
  • Console reporter (Rich)

Phase 2: Server Compliance Tests ✅

  • Lifecycle tests (initialize, ping)
  • Tools tests
  • Resources tests
  • Prompts tests
  • Field naming validator
  • Type validator

Phase 3: Extended Server Tests ✅

  • Completion tests
  • Sampling tests
  • Logging tests
  • Roots tests
  • Pagination tests
  • Progress notification tests
  • Cancellation tests

Phase 4: Reference Server (Go) ✅

  • Basic HTTP server
  • JSON-RPC handler
  • All required methods
  • All server capabilities
  • 100% spec-compliant types

Phase 5: Client Compliance Tests ✅

  • Client sampling tests
  • Client logging tests
  • Client roots tests
  • Client error handling tests

Phase 6: Transport Tests ✅

  • HTTP transport tests
  • SSE transport tests
  • WebSocket transport tests
  • STDIO transport tests

Phase 7: Reporting & Polish ✅

  • JSON reporter
  • HTML reporter
  • Markdown reporter
  • CI/CD integration guide
  • Documentation

✅ Success Criteria

  • Detects all 32 issues from [Epic]: Fully implement MCP 2025-06-18 compliance across all endpoints #1285 manual review
  • Supports 2025-06-18 and 2025-03-26 specs via YAML
  • Tests 300+ compliance points (protocol, types, transports, field naming)
  • Generates reports in 4 formats (console, JSON, HTML, Markdown)
  • Reference server passes 100% of its own compliance tests
  • CI/CD integration with exit codes and artifacts
  • Fast execution (< 30 seconds for full test)
  • Clear error messages with file locations and fix suggestions
  • 90%+ test coverage
  • Passes make verify checks

🏁 Definition of Done

  • Python compliance checker implemented in tools/mcp-compliance-checker/
  • YAML spec configs for 2025-06-18 and 2025-03-26
  • Server tests: lifecycle, tools, resources, prompts, sampling, logging, roots, completion, pagination, progress, cancellation
  • Type tests: content types, resource types, annotations, field naming
  • Transport tests: HTTP, SSE, WebSocket, STDIO
  • Go reference server (100% spec-compliant)
  • Client tests: sampling, roots, logging, error handling
  • Four reporters: Console (rich), JSON, HTML, Markdown
  • CLI with --spec, --capabilities, --quick, --output-format, --output-file, --verbose
  • Authentication: Bearer token, Basic auth
  • Detects all 32 issues from [Epic]: Fully implement MCP 2025-06-18 compliance across all endpoints #1285
  • Unit tests with 90%+ coverage
  • Integration tests against ContextForge
  • Documentation: README, examples, CI/CD guide
  • Docker image for CI/CD
  • Passes make verify
  • Performance: < 30 seconds for full test

🔗 Related Issues

📚 References

Metadata

Metadata

Assignees

Labels

mcp-2025-06-18MCP 2025-06-18 protocol specificationmcp-protocolAlignment with MCP protocol or specificationtestingTesting (unit, e2e, manual, automated, etc)

Type

Feature

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions