How Verification Works

How Verification Works

toolism slim --verify (needs pip install "toolism[verify]") compares every changed schema with its original, in both directions. The question each time: does every tool call the original accepts also pass the slim schema, and vice versa?

• Identical / annotations-only: proved. If the schemas differ only in keywords that never affect validation (title, description, examples, ...), equivalence is certain.
• Anything else: sampled. hypothesis-jsonschema generates instances from one schema and checks them against the other. This is strong evidence, not proof.
• Skipped. If instances can't be generated, the result is reported as skipped, never as passed.

When a counterexample is found

toolism finds the rules responsible: it adds the rules that changed the tool back one at a time, re-verifying each combination, and reverts only the ones that break equivalence. The other rules' savings are kept. If no single culprit is found, the whole tool is restored to its original. --fail-on-rollback makes slim exit 1 when this happens.

In CI, every schema the default rules change in the benchmark corpus is verified this way, with zero counterexamples. The implementation is in src/toolism/verify.py.