First release. Scores tool-call reliability for agentic coding CLIs — whether the agent makes correct, safe, well-justified tool calls, not whether the prose reads nicely.
pip install toolharness
toolharness suite --adapter claude-codeWhat's in it
- Eight failure modes (M1–M8), each scored 0–100, plus a safety-weighted composite. Deterministic detectors for the clear-cut modes; hybrid heuristic + LLM-judge for the ambiguous ones (wrong tool, ignored output, premature stop).
- Real CLI adapters for Claude Code (SDK stream-json + OTEL), Cursor, and Codex, with parser tests over captured real runs. Gemini is deferred pending auth.
- Bundled task suite —
toolharness suite --adapter <agent>runs curated coding tasks end to end, each seeded into a throwaway sandbox, and aggregates the scores. No task specs to write. - Self-contained HTML dashboard — score bars, SVG radar, tool-call timeline, click-to-expand drill-down. One file, no external requests.
- CI gate —
--fail-underand--fail-under-mode UNSAFE_CALL=90exit non-zero on regressions. - BFCL validation with per-detector P/R/F1 and Cohen's κ against the judge — see BENCHMARKS.md.
Bring your own judge key
The harness ships no API key and no hosted service. Default scoring is heuristic-only and makes no network calls. The optional LLM judge reads your own GROQ_API_KEY / OPENROUTER_API_KEY / NVIDIA_API_KEY, sends it only to the provider you pick, and never writes it to the cache or any report. --judge ollama runs against a local model with no key at all.
The judge is deliberately never Claude, GPT, or Gemini — those are the agents under test, and self-preference bias would undermine the scores.