Add comprehensive testing infrastructure#16
Conversation
943f0a6 to
802e04a
Compare
This PR adds automated testing tools to compare client output against direct API calls for all commands. ## New Testing Tool ### test-commands.sh (Bash) - Compare client binary output vs direct API calls - Support for safe-only testing (no mutations) - --show-safe option to list all safe commands - --resume option to continue after session timeout - Automatic exclusion of problematic commands (e.g., logout) - Detailed logging and JSON result summaries - Full JSON comparison with dynamic field filtering ## Documentation ### tools/README.md - Quick start guide - Usage examples for all test modes - Safe command categories (20 commands) - Session timeout handling - Excluded commands documentation ### tools/TESTING.md - Complete testing documentation - Interpreting test results - Debugging guide - CI/CD integration examples ## Features - **Safe Mode**: Test 20 query commands that don't modify game state - **Resume**: Continue testing from specific command after timeout - **Exclusions**: Automatically exclude commands that break testing (logout) - **Logging**: Detailed logs with timestamps and full output - **Results**: JSON summaries for programmatic analysis - **Status Tracking**: Identical, different, errors Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
802e04a to
60ba843
Compare
|
Thanks for the idea here @rsned — comparing client behavior against the live API is exactly the right instinct. We took a similar approach in PR #18, but shifted the comparison point: instead of diffing CLI output against raw API responses (which breaks because the client intentionally formats/transforms data), we compare the client's command list directly against the OpenAPI spec. The result is a We also added a GitHub Actions CI workflow so lint and tests run on every PR — that was a gap worth closing regardless. Appreciate the push in this direction! |
This PR adds automated testing tools to compare client output against direct API calls for all commands.
New Testing Tool
test-commands.sh (Bash)
Documentation
tools/README.md
Features
Co-Authored-By: Claude Sonnet noreply@anthropic.com