Skip to content

v0.3.0

Choose a tag to compare

View all tags
@NilkanthSuthar NilkanthSuthar released this 15 Oct 05:41
· 3 commits to main since this release
v0.3.0
7e18124

Tool-selection evals, a GitHub Action and an HTML report.

pip install -U mcp-toolproof
pip install -U "mcp-toolproof[anthropic]"   # or [openai], for evals

New

  • toolproof eval: tool-selection evals. Each case gives a model your server's tool list and a prompt, and checks which tool and arguments it picks. Providers: anthropic, openai (and compatible APIs through OPENAI_BASE_URL), and command for your own program. Accuracy is gated by min_accuracy; API answers are cached in .toolproof-cache/ so re-runs are free and repeatable, and --no-cache refreshes them. The provider SDKs are optional extras, so the base install still needs no model.
  • GitHub Action: uses: NilkanthSuthar/toolproof@v0.3.0 installs toolproof, runs it, uploads JUnit, JSON and HTML reports and writes a job summary.
  • --html: a single-file HTML report with a latency chart that works offline. Everything that comes from the server under test is escaped.
  • --summary: appends a Markdown summary, for example to $GITHUB_STEP_SUMMARY.
  • HTTP headers for servers that need auth, and ${VAR} in the server: section so secrets come from the environment.
  • The package ships py.typed, and run_config, RunReport and ConfigError are public API.
  • A documentation site: https://nilkanthsuthar.github.io/toolproof/

Fixed

  • import toolproof and the pytest plugin no longer load Hypothesis up front.

Notes

  • The anthropic and openai adapters follow the current APIs (automatic tool choice, no sampling parameters by default) and are tested against fake clients in CI. The command provider and everything else are tested end to end. If you run into an API error with a provider, please open an issue with the message.

Full changelog: https://github.com/NilkanthSuthar/toolproof/blob/main/CHANGELOG.md

Assets 2
Loading