Skip to content

v0.2.0

Choose a tag to compare

@NilkanthSuthar NilkanthSuthar released this 07 Oct 00:37
· 21 commits to main since this release

Fuzzing and benchmarks.

pip install -U mcp-toolproof

New

  • toolproof fuzz: generates valid and invalid inputs from each tool's input schema with Hypothesis and reports crashes, hangs, internal errors, leaked tracebacks, outputSchema violations and invalid input that was accepted, each shrunk to the smallest failing input
  • Fuzzing restarts a crashed server and keeps going, skips tools marked destructive by default, and prints its seed so a run can be repeated
  • toolproof bench: p50/p95/p99 latency, throughput and error rate at a chosen concurrency, with thresholds that fail the run
  • bench: and fuzz: sections in toolproof.yaml; toolproof run includes them when present (--skip-bench, --skip-fuzz)
  • Fuzz and bench results in the console, JUnit and JSON reports
  • fuzz, bench and inspect fall back to ./toolproof.yaml when no server is given

Try it

toolproof fuzz -- python examples/buggy_server.py

This finds all five bugs planted in the example server without any hand-written tests.

See the README for details.