Implement CLI test command functionality

[milliondreams]

Problem

The CLI framework is in place with proper command structure, but the test command implementation is missing. Users currently see "Test functionality is not yet implemented" when trying to run tests.

Solution

Implement the test command functionality that integrates with the test suite runner to provide a complete CLI testing experience.

Implementation Details

Update Existing CLI Commands:

1. Implement Test Command Handler (crates/mandrel-mcp-th/src/cli/commands.rs)

impl Commands {
    /// Execute test command with proper error handling and output
    pub async fn execute_test(
        &self,
        spec_file: PathBuf,
        output_dir: Option<PathBuf>,
        format: Option<OutputFormat>,
        server_config: Option<PathBuf>,
    ) -> Result<()> {
        // 1. Initialize test suite runner
        // 2. Connect to MCP server
        // 3. Execute test specification
        // 4. Generate and save reports
        // 5. Display summary to user
    }
    
    /// Display test progress and real-time feedback
    fn display_test_progress(&self, result: &TestResult) {
        // Show real-time test execution progress
    }
}

2. Enhanced CLI Arguments

#[derive(Args)]
pub struct TestArgs {
    /// Path to test specification file
    pub spec_file: PathBuf,
    
    /// Output directory for test reports
    #[arg(short, long)]
    pub output: Option<PathBuf>,
    
    /// Report format (html, json, junit, markdown)
    #[arg(short, long, default_value = "html")]
    pub format: OutputFormat,
    
    /// Server configuration file
    #[arg(short, long)]
    pub server_config: Option<PathBuf>,
    
    /// Run tests in parallel
    #[arg(short, long)]
    pub parallel: bool,
    
    /// Fail fast on first error
    #[arg(long)]
    pub fail_fast: bool,
    
    /// Verbose output
    #[arg(short, long)]
    pub verbose: bool,
}

3. Enhanced Test Output and Progress

• Real-time test progress display
• Colored output for pass/fail status
• Summary statistics
• Report generation and location display

CLI Integration Flow:

1. Parse command: moth test spec.yaml --output ./reports --format html
2. Initialize runner: Create TestSuiteRunner with configuration
3. Execute tests: Run test specification with progress display
4. Generate reports: Create output in specified format and location
5. Display summary: Show test results and report location

Dependencies:

• Blocks on: Issue Implement test suite runner for YAML specification execution #220 (Test suite runner)
• Uses: Existing CLI framework (crates/mandrel-mcp-th/src/cli/)
• Uses: Existing reporting system (crates/mandrel-mcp-th/src/reporting/)

Test Requirements:

• Integration test with real YAML specification
• CLI argument parsing tests
• Error handling and user feedback tests
• Report generation integration tests
• Progress display tests

Success Criteria:

• moth test spec.yaml command works end-to-end
• Proper error messages for invalid inputs
• Progress display during test execution
• Report generation in specified format and location
• Clean summary output with statistics
• Verbose mode shows detailed test information

Usage Examples:

# Basic test execution
moth test codeprism-python-comprehensive.yaml

# With custom output and format
moth test spec.yaml --output ./reports --format json

# Parallel execution with fail-fast
moth test spec.yaml --parallel --fail-fast

# Verbose output
moth test spec.yaml --verbose

Testing:

• Test with actual CodePrism YAML specifications
• Verify all CLI options work correctly
• Test error scenarios (missing files, invalid server config)
• Validate report generation and output formatting

Estimated Effort:

2-3 hours

Priority:

P0 - Makes the CLI actually functional for users

[milliondreams]

Issue #221: CLI run Command Implementation Design Document

1. Problem Statement

The Mandrel Test Harness (moth) has a defined CLI structure with a run command, but the implementation is currently a placeholder. It does not connect to the TestSuiteRunner and cannot execute any tests. This issue is about implementing the logic to make the moth run command fully functional.

2. Proposed Solution

The solution is to implement the handle_run_command function in crates/mandrel-mcp-th/src/cli/mod.rs. This function will:

1. Parse the RunArgs to configure the test run.
2. Initialize the McpClient and TestCaseExecutor.
3. Initialize the TestSuiteRunner with the correct configuration.
4. Execute the test suite specified in the arguments.
5. Pass the results to the TestReporter.
6. Print a summary to the console and return an appropriate exit code.

High-Level Architecture

┌──────────────┐      ┌─────────────────────┐      ┌───────────────────┐
│              │      │                     │      │                   │
│  CLI (`moth`)├─────▶│   `handle_run_command`  ├─────▶│  `TestSuiteRunner`  │
│              │      │      (in cli/mod.rs)    │      │ (in runner/mod.rs)│
└──────────────┘      └─────────────────────┘      └───────────────────┘
      ▲                        │                           │
      │                        │                           ▼
      │                        │                  ┌──────────────────┐
      │                        └─────────────────▶│  `TestReporter`  │
      │                                           │(in reporting/mod.rs)│
      └───────────────────────────────────────────┘

3. API Design

The main changes will be within the handle_run_command and a new display_summary function.

handle_run_command in cli/mod.rs

// in `crates/mandrel-mcp-th/src/cli/mod.rs`
async fn handle_run_command(&self, args: &RunArgs) -> Result<i32> {
    // 1. Load the specification to get server config
    let spec_loader = SpecificationLoader::new();
    let spec = spec_loader.load_from_file(&args.config).await?;

    // 2. Initialize the client and executor
    // NOTE: This uses the ServerConfig from the loaded YAML spec
    let client = McpClient::new(spec.server.into()).await?;
    let executor_config = ExecutorConfig::default();
    let executor = TestCaseExecutor::new(Arc::new(Mutex::new(client)), executor_config);

    // 3. Initialize the TestSuiteRunner
    let runner_config = RunnerConfig::new()
        .with_parallel_execution(args.parallel)
        .with_fail_fast(args.fail_fast);
    let mut runner = TestSuiteRunner::new(executor, runner_config);

    // 4. Execute the test suite
    let suite_result = runner.run_test_suite(&args.config).await?;

    // 5. Generate a report
    // PLANNED(#194): This will be a more robust reporting implementation
    if let Some(output_dir) = &args.output {
        let report_path = output_dir.join("report.json");
        let report_json = serde_json::to_string_pretty(&suite_result)?;
        tokio::fs::write(report_path, report_json).await?;
    }
    
    // 6. Display summary and return exit code
    self.display_summary(&suite_result);
    Ok(if suite_result.failed == 0 { 0 } else { 1 })
}

fn display_summary(&self, result: &TestSuiteResult) {
    // Pretty-print a summary of the results to the console
    println!("\n✅ Test Suite Finished ✅");
    println!("Suite: {}", result.suite_name);
    println!("Total Tests: {}, Passed: {}, Failed: {}", 
        result.total_tests, result.passed, result.failed);
    println!("Duration: {:.2}s", result.total_duration.as_secs_f64());
}

RunArgs in cli/args.rs

The existing RunArgs struct will be used to pass configuration to the handle_run_command function. We need to ensure it has all the necessary fields.

// in `crates/mandrel-mcp-th/src/cli/args.rs`
#[derive(Args, Debug)]
pub struct RunArgs {
    /// Path to the test specification file.
    #[arg()]
    pub config: PathBuf,

    /// Output directory for generated reports.
    #[arg(short, long, default_value = "./reports")]
    pub output: Option<PathBuf>,

    /// Run tests in parallel.
    #[arg(long)]
    pub parallel: bool,

    /// Stop execution on the first test failure.
    #[arg(long)]
    pub fail_fast: bool,
}

4. Implementation Plan

1. Update RunArgs: Add parallel and fail_fast flags to the RunArgs struct in cli/args.rs.
2. Implement handle_run_command:
   • Replace the placeholder println! with the logic described in the API design.
   • Load the TestSpecification from the path in RunArgs.
   • Use the server config from the spec to initialize McpClient.
   • Create a TestCaseExecutor.
   • Create and configure the TestSuiteRunner using arguments from RunArgs.
   • Call runner.run_test_suite().
3. Implement display_summary: Add a private helper function to CliApp to print a formatted summary of TestSuiteResult.
4. Testing:
   • Add a new integration test in crates/mandrel-mcp-th/tests/cli.rs.
   • The test will execute moth run against a simple YAML file.
   • It will assert that the command exits with the correct code (0 for success, 1 for failure).
   • It will check the console output for the summary.

5. Testing Strategy

• Integration Test (cli_run_command_success): Run moth run with simple-test.yaml. Assert exit code is 0 and stdout contains "Passed: 3".
• Integration Test (cli_run_command_failure): Run moth run with a spec where one test is expected to fail. Assert exit code is 1 and stdout contains "Failed: 1".
• Integration Test (cli_run_command_fail_fast): Run moth run --fail-fast on the failing spec. Assert that the number of executed tests is less than the total.
• Argument Parsing Tests: Add unit tests in cli/mod.rs to ensure flags like --parallel and --fail-fast are parsed correctly.

This approach will provide a fully functional run command and set the stage for the final integration and documentation work.

[milliondreams]

Closing this issue as it has been completed. Work was finished in commit 5a7c2f2 which implemented the CLI run command functionality with comprehensive result display and integration testing.