Implement end-to-end integration testing for complete workflow

[milliondreams]

Problem

While individual components (executor, runner, CLI) will be implemented, we need comprehensive end-to-end integration testing to ensure the complete system works together correctly with real MCP servers.

Solution

Create end-to-end integration tests that validate the complete test harness functionality from CLI invocation through test execution to report generation.

Implementation Details

New Test Infrastructure:

1. End-to-End Integration Tests (crates/mandrel-mcp-th/tests/integration/)

/// Test complete workflow: CLI → Runner → Executor → Reports
#[tokio::test]
async fn test_complete_workflow_with_codeprism_server() {
    // 1. Start CodePrism MCP server
    // 2. Execute test via CLI
    // 3. Validate all reports generated
    // 4. Verify test results accuracy
}

/// Test with all supported YAML specification formats
#[tokio::test]
async fn test_all_specification_formats() {
    // Test comprehensive, tools, and workflow specifications
}

/// Test error handling and recovery scenarios
#[tokio::test]
async fn test_error_scenarios() {
    // Server disconnection, invalid specs, timeouts
}

2. Mock MCP Server for Testing

/// Lightweight mock MCP server for predictable testing
pub struct MockMcpServer {
    pub tools: HashMap<String, Box<dyn Fn(Value) -> Value>>,
}

impl MockMcpServer {
    /// Create server with predefined tool responses
    pub fn with_tools(tools: Vec<(&str, Box<dyn Fn(Value) -> Value>)>) -> Self
    
    /// Start server and return connection info
    pub async fn start(&self) -> Result<ServerInfo>
}

3. Test Data and Fixtures

• Complete test specifications covering all CodePrism tools
• Expected output samples for validation
• Error scenario test cases
• Performance benchmark test cases

Test Coverage Areas:

1. Component Integration

• CLI → TestSuiteRunner integration
• TestSuiteRunner → TestCaseExecutor integration
• TestCaseExecutor → MCP Client integration
• Validation Engine → Reporting integration

2. Real Server Testing

• CodePrism MCP server integration
• All 26 MCP tools execution and validation
• Multi-language test specification execution
• Performance characteristics validation

3. Error Handling

• Server connection failures
• Invalid YAML specifications
• Tool execution timeouts
• Malformed server responses
• Network interruption recovery

4. Report Generation

• HTML report generation and validation
• JSON report format compliance
• JUnit XML format compatibility
• Markdown report readability

Dependencies:

• Blocks on: Issue Implement CLI test command functionality #221 (CLI functionality)
• Uses: All implemented components (executor, runner, CLI)
• Requires: CodePrism MCP server for integration tests

Test Requirements:

• Complete workflow integration tests
• Mock server testing infrastructure
• Real CodePrism server integration tests
• Performance and load testing
• Error recovery and resilience tests

Success Criteria:

• Complete CLI workflow works end-to-end
• All CodePrism test specifications execute successfully
• Report generation works in all formats
• Error handling provides useful feedback
• Performance meets requirements (<10 seconds for comprehensive test)
• Integration tests run reliably in CI/CD

Test Scenarios:

# 1. Basic functionality test
moth test codeprism-python-comprehensive.yaml

# 2. Full report generation test  
moth test codeprism-rust-comprehensive.yaml --output ./reports --format html

# 3. Error handling test
moth test invalid-spec.yaml  # Should show helpful error

# 4. Performance test
moth test large-specification.yaml --verbose  # Should complete in <10s

Validation:

• Compare generated reports against expected outputs
• Verify all test results match expected outcomes
• Validate performance metrics are collected correctly
• Ensure error messages are helpful and actionable

Testing:

• Run against all CodePrism moth specifications
• Test with various MCP server configurations
• Validate across different environments (CI/CD, local)
• Performance testing with large test suites

Estimated Effort:

4-6 hours

Priority:

P0 - Required to validate complete system functionality

[milliondreams]

Issue #222: End-to-End Integration Testing Design Document

Problem Statement

The Mandrel MCP Test Harness has individual components (executor, runner, CLI) but lacks comprehensive end-to-end integration testing to ensure the complete system works together correctly with real MCP servers. Without proper integration testing, we cannot verify that the complete workflow from CLI invocation through test execution to report generation functions correctly.

Proposed Solution

Implement a comprehensive end-to-end integration testing framework that validates the complete test harness functionality using both mock and real MCP servers. This includes testing all component integrations, error scenarios, and report generation capabilities.

High-Level Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Integration Test Suite                       │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐ │
│  │   Mock Server   │  │  Real Server    │  │   Test Data     │ │
│  │   Testing       │  │   Testing       │  │   & Fixtures    │ │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐ │
│  │   Component     │  │   Error         │  │   Performance   │ │
│  │   Integration   │  │   Handling      │  │   Testing       │ │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐ │
│  │   CLI Testing   │  │   Report        │  │   Validation    │ │
│  │                 │  │   Generation    │  │   Engine        │ │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Component Design

1. Mock MCP Server Infrastructure

// Mock server for predictable testing
pub struct MockMcpServer {
    pub tools: HashMap<String, Box<dyn Fn(Value) -> Result<Value, String> + Send + Sync>>,
    pub server_handle: Option<tokio::task::JoinHandle<()>>,
    pub port: u16,
    pub delay_ms: Option<u64>, // For timeout testing
}

impl MockMcpServer {
    pub fn new() -> Self;
    pub fn with_tools(tools: Vec<(&str, Box<dyn Fn(Value) -> Result<Value, String> + Send + Sync>)>) -> Self;
    pub fn with_delay(delay_ms: u64) -> Self;
    pub async fn start(&mut self) -> Result<ServerInfo, MockServerError>;
    pub async fn stop(&mut self) -> Result<(), MockServerError>;
    pub fn get_connection_info(&self) -> ServerInfo;
}

// Predefined tool responses for testing
pub struct MockToolResponses;

impl MockToolResponses {
    pub fn analyze_code_quality() -> Box<dyn Fn(Value) -> Result<Value, String> + Send + Sync>;
    pub fn search_symbols() -> Box<dyn Fn(Value) -> Result<Value, String> + Send + Sync>;
    pub fn error_response() -> Box<dyn Fn(Value) -> Result<Value, String> + Send + Sync>;
    pub fn timeout_response() -> Box<dyn Fn(Value) -> Result<Value, String> + Send + Sync>;
}

2. Integration Test Framework

// Main integration test framework
pub struct IntegrationTestFramework {
    pub mock_server: Option<MockMcpServer>,
    pub test_data_dir: PathBuf,
    pub reports_dir: PathBuf,
    pub temp_dir: PathBuf,
}

impl IntegrationTestFramework {
    pub fn new() -> Self;
    pub async fn setup(&mut self) -> Result<(), IntegrationTestError>;
    pub async fn teardown(&mut self) -> Result<(), IntegrationTestError>;
    pub async fn run_cli_test(&self, spec_file: &str, args: &[&str]) -> Result<TestResult, IntegrationTestError>;
    pub async fn validate_reports(&self, expected_reports: &[ReportExpectation]) -> Result<(), IntegrationTestError>;
}

// Test result structure
pub struct TestResult {
    pub exit_code: i32,
    pub stdout: String,
    pub stderr: String,
    pub execution_time: Duration,
    pub generated_files: Vec<PathBuf>,
}

// Report validation expectations
pub struct ReportExpectation {
    pub format: ReportFormat,
    pub file_path: PathBuf,
    pub expected_content: Vec<ContentExpectation>,
}

pub enum ContentExpectation {
    ContainsText(String),
    JsonPath(String, Value),
    ElementCount(String, usize),
    FileSize(Range<u64>),
}

3. Test Data and Fixtures

// Test data management
pub struct TestFixtures {
    pub spec_files: HashMap<String, PathBuf>,
    pub expected_outputs: HashMap<String, Value>,
    pub error_scenarios: Vec<ErrorScenario>,
}

impl TestFixtures {
    pub fn load_all() -> Result<Self, TestFixturesError>;
    pub fn get_spec(&self, name: &str) -> Option<&PathBuf>;
    pub fn get_expected_output(&self, spec_name: &str) -> Option<&Value>;
    pub fn create_temp_spec(&self, content: &str) -> Result<PathBuf, TestFixturesError>;
}

// Error scenario testing
pub struct ErrorScenario {
    pub name: String,
    pub spec_content: String,
    pub expected_error: ExpectedError,
}

pub enum ExpectedError {
    InvalidYaml,
    ServerConnectionFailure,
    ToolExecutionTimeout,
    MalformedResponse,
    ValidationFailure,
}

Implementation Plan

Phase 1: Mock Server Infrastructure

1. Create Mock MCP Server (crates/mandrel-mcp-th/src/testing/mock_server.rs)

   • Implement basic MCP protocol handling
   • Add tool response simulation
   • Include delay/timeout simulation capabilities
   • Add connection management

2. Implement Predefined Tool Responses (crates/mandrel-mcp-th/src/testing/mock_responses.rs)

   • Create realistic responses for all 26 CodePrism tools
   • Include error response patterns
   • Add response validation helpers

Phase 2: Integration Test Framework

1. Create Integration Test Framework (crates/mandrel-mcp-th/src/testing/integration_framework.rs)

   • Implement test setup/teardown
   • Add CLI execution helpers
   • Include report validation logic

2. Implement Test Data Management (crates/mandrel-mcp-th/src/testing/fixtures.rs)

   • Create test specification files
   • Add expected output samples
   • Include error scenario definitions

Phase 3: Component Integration Tests

1. CLI Integration Tests (crates/mandrel-mcp-th/tests/integration/cli_tests.rs)

   • Test complete CLI workflow
   • Validate argument parsing and execution
   • Test error handling and user feedback

2. Runner Integration Tests (crates/mandrel-mcp-th/tests/integration/runner_tests.rs)

   • Test TestSuiteRunner with real specifications
   • Validate test execution orchestration
   • Test concurrent execution capabilities

3. Executor Integration Tests (crates/mandrel-mcp-th/tests/integration/executor_tests.rs)

   • Test TestCaseExecutor with mock servers
   • Validate MCP protocol handling
   • Test error recovery mechanisms

Phase 4: Real Server Integration Tests

1. CodePrism Server Tests (crates/mandrel-mcp-th/tests/integration/codeprism_tests.rs)

   • Test with real CodePrism MCP server
   • Validate all 26 tools execution
   • Test multi-language support

2. Performance Tests (crates/mandrel-mcp-th/tests/integration/performance_tests.rs)

   • Test execution time requirements
   • Validate memory usage patterns
   • Test concurrent execution limits

Phase 5: Error Handling and Report Generation Tests

1. Error Scenario Tests (crates/mandrel-mcp-th/tests/integration/error_tests.rs)

   • Test server disconnection scenarios
   • Validate timeout handling
   • Test malformed specification handling

2. Report Generation Tests (crates/mandrel-mcp-th/tests/integration/report_tests.rs)

   • Test HTML report generation
   • Validate JSON report format
   • Test JUnit XML compatibility

Testing Strategy

Test Categories

1. Unit Tests (90% coverage target)

   • Mock server components
   • Test framework utilities
   • Report validation logic

2. Integration Tests (80% coverage target)

   • Complete CLI workflow
   • Component integration
   • Real server interactions

3. End-to-End Tests (100% scenario coverage)

   • Full workflow validation
   • Error scenario handling
   • Performance requirements

Test Data Organization

crates/mandrel-mcp-th/tests/
├── integration/
│   ├── cli_tests.rs
│   ├── runner_tests.rs
│   ├── executor_tests.rs
│   ├── codeprism_tests.rs
│   ├── performance_tests.rs
│   ├── error_tests.rs
│   └── report_tests.rs
├── fixtures/
│   ├── specifications/
│   │   ├── comprehensive.yaml
│   │   ├── tools-only.yaml
│   │   ├── workflow.yaml
│   │   └── error-scenarios/
│   ├── expected_outputs/
│   └── test_data/
└── utils/
    ├── mod.rs
    └── test_helpers.rs

Performance Requirements

• CLI Execution: <2 seconds for basic operations
• Comprehensive Test Suite: <10 seconds total execution
• Report Generation: <1 second for all formats
• Memory Usage: <100MB during execution
• Concurrent Tests: Support up to 10 parallel test cases

Success Criteria

Functional Requirements

• Complete CLI workflow executes successfully
• All CodePrism test specifications execute correctly
• Report generation works in all formats (HTML, JSON, JUnit XML)
• Error handling provides actionable feedback
• Mock server supports all required testing scenarios

Quality Requirements

• 90% unit test coverage for new components
• 80% integration test coverage for workflows
• 100% error scenario coverage
• All tests pass consistently in CI/CD

Performance Requirements

• CLI execution completes in <2 seconds
• Comprehensive test suite completes in <10 seconds
• Memory usage remains below 100MB
• Concurrent execution supports 10+ test cases

Reliability Requirements

• Tests pass consistently across environments
• Error recovery mechanisms work correctly
• Report generation is deterministic
• Mock server handles edge cases gracefully

Alternative Approaches Considered

1. External Test Runner Integration

Approach: Use existing test runners like pytest or cargo test
Pros: Leverages existing tooling, familiar patterns
Cons: Doesn't test complete CLI workflow, limited report format control
Decision: Rejected - Need to test complete CLI integration

2. Docker-based Testing

Approach: Run all tests in containerized environments
Pros: Consistent environments, isolation
Cons: Increased complexity, slower execution
Decision: Considered for future enhancement, not initial implementation

3. Property-based Testing

Approach: Use property-based testing frameworks
Pros: Comprehensive edge case coverage
Cons: Complex setup, longer execution times
Decision: Included for specific components, not primary strategy

Breaking Changes

None expected - This is purely additive testing infrastructure that doesn't affect existing APIs or user-facing functionality.

Testing Implementation

Mock Server Testing

#[tokio::test]
async fn test_mock_server_basic_functionality() {
    let mut server = MockMcpServer::new()
        .with_tools(vec![
            ("analyze_code_quality", MockToolResponses::analyze_code_quality()),
            ("search_symbols", MockToolResponses::search_symbols()),
        ]);
    
    let server_info = server.start().await.unwrap();
    
    // Test server responds to basic requests
    let client = McpClient::new(server_info.clone()).await.unwrap();
    let response = client.call_tool("analyze_code_quality", json!({})).await.unwrap();
    
    assert!(response.is_success());
    server.stop().await.unwrap();
}

CLI Integration Testing

#[tokio::test]
async fn test_complete_cli_workflow() {
    let framework = IntegrationTestFramework::new();
    framework.setup().await.unwrap();
    
    // Test basic CLI execution
    let result = framework.run_cli_test(
        "comprehensive.yaml",
        &["--output", "reports", "--format", "html"]
    ).await.unwrap();
    
    assert_eq!(result.exit_code, 0);
    
    // Validate reports were generated
    framework.validate_reports(&[
        ReportExpectation {
            format: ReportFormat::Html,
            file_path: PathBuf::from("reports/test_report.html"),
            expected_content: vec![
                ContentExpectation::ContainsText("Test Results".to_string()),
                ContentExpectation::ElementCount("test-case", 26),
            ],
        }
    ]).await.unwrap();
    
    framework.teardown().await.unwrap();
}

Error Scenario Testing

#[tokio::test]
async fn test_server_disconnection_handling() {
    let mut server = MockMcpServer::new();
    let server_info = server.start().await.unwrap();
    
    // Start test execution
    let framework = IntegrationTestFramework::new();
    
    // Stop server mid-execution to simulate disconnection
    tokio::spawn(async move {
        tokio::time::sleep(Duration::from_millis(100)).await;
        server.stop().await.unwrap();
    });
    
    let result = framework.run_cli_test("comprehensive.yaml", &[]).await.unwrap();
    
    // Should handle gracefully and provide useful error message
    assert_ne!(result.exit_code, 0);
    assert!(result.stderr.contains("Server connection lost"));
}

This comprehensive design ensures complete end-to-end integration testing coverage while maintaining high quality standards and following TDD principles.

[milliondreams]

COMPLETE: End-to-end integration is working. Successfully executed moth against CodePrism MCP server with 25/25 tests passing. Full CLI->Runner->Client->Server integration demonstrated.