v0.3.1

A small follow-up to 0.3.0.

pip install -U mcp-toolproof

Changed

• The GitHub Action is named MCP toolproof, so it can be listed on the GitHub Marketplace. Workflows don't change: they still use uses: NilkanthSuthar/toolproof@v0.3.1.

Fixed

• refresh, override_provider and override_model were accepted as keys in the evals: section of toolproof.yaml. They are command-line options only (--no-cache, --provider, --model) and are now rejected like any unknown key.

Full changelog: https://github.com/NilkanthSuthar/toolproof/blob/main/CHANGELOG.md

v0.3.0

Tool-selection evals, a GitHub Action and an HTML report.

pip install -U mcp-toolproof
pip install -U "mcp-toolproof[anthropic]"   # or [openai], for evals

New

• toolproof eval: tool-selection evals. Each case gives a model your server's tool list and a prompt, and checks which tool and arguments it picks. Providers: anthropic, openai (and compatible APIs through OPENAI_BASE_URL), and command for your own program. Accuracy is gated by min_accuracy; API answers are cached in .toolproof-cache/ so re-runs are free and repeatable, and --no-cache refreshes them. The provider SDKs are optional extras, so the base install still needs no model.
• GitHub Action: uses: NilkanthSuthar/toolproof@v0.3.0 installs toolproof, runs it, uploads JUnit, JSON and HTML reports and writes a job summary.
• --html: a single-file HTML report with a latency chart that works offline. Everything that comes from the server under test is escaped.
• --summary: appends a Markdown summary, for example to $GITHUB_STEP_SUMMARY.
• HTTP headers for servers that need auth, and ${VAR} in the server: section so secrets come from the environment.
• The package ships py.typed, and run_config, RunReport and ConfigError are public API.
• A documentation site: https://nilkanthsuthar.github.io/toolproof/

Fixed

• import toolproof and the pytest plugin no longer load Hypothesis up front.

Notes

• The anthropic and openai adapters follow the current APIs (automatic tool choice, no sampling parameters by default) and are tested against fake clients in CI. The command provider and everything else are tested end to end. If you run into an API error with a provider, please open an issue with the message.

Full changelog: https://github.com/NilkanthSuthar/toolproof/blob/main/CHANGELOG.md

v0.2.0

Fuzzing and benchmarks.

pip install -U mcp-toolproof

New

• toolproof fuzz: generates valid and invalid inputs from each tool's input schema with Hypothesis and reports crashes, hangs, internal errors, leaked tracebacks, outputSchema violations and invalid input that was accepted, each shrunk to the smallest failing input
• Fuzzing restarts a crashed server and keeps going, skips tools marked destructive by default, and prints its seed so a run can be repeated
• toolproof bench: p50/p95/p99 latency, throughput and error rate at a chosen concurrency, with thresholds that fail the run
• bench: and fuzz: sections in toolproof.yaml; toolproof run includes them when present (--skip-bench, --skip-fuzz)
• Fuzz and bench results in the console, JUnit and JSON reports
• fuzz, bench and inspect fall back to ./toolproof.yaml when no server is given

Try it

toolproof fuzz -- python examples/buggy_server.py

This finds all five bugs planted in the example server without any hand-written tests.

See the README for details.

v0.1.0

First release of toolproof: automated tests for MCP servers.

pip install mcp-toolproof

The package is mcp-toolproof on PyPI. The command and the import are both toolproof.

What's in it

• toolproof inspect: list a server's tools, resources and prompts (table or --json)
• Static checks for tool definitions: names, descriptions, input/output JSON Schemas, required fields
• YAML test cases with is_error, equals, contains, regex, jsonpath, max_latency_ms and automatic outputSchema validation
• toolproof run with per-test timeouts, retries, crash detection and automatic server restart
• Console, JUnit XML and JSON reports; exit code 0/1 for CI
• pytest plugin with an mcp_server fixture
• stdio and streamable HTTP transports
• Example weather server and a buggy server with planted bugs

See the README for the quickstart and YAML reference.