v0.1.0 — toolharness

First release. Scores tool-call reliability for agentic coding CLIs — whether the agent makes correct, safe, well-justified tool calls, not whether the prose reads nicely.

pip install toolharness
toolharness suite --adapter claude-code

What's in it

• Eight failure modes (M1–M8), each scored 0–100, plus a safety-weighted composite. Deterministic detectors for the clear-cut modes; hybrid heuristic + LLM-judge for the ambiguous ones (wrong tool, ignored output, premature stop).
• Real CLI adapters for Claude Code (SDK stream-json + OTEL), Cursor, and Codex, with parser tests over captured real runs. Gemini is deferred pending auth.
• Bundled task suite — toolharness suite --adapter <agent> runs curated coding tasks end to end, each seeded into a throwaway sandbox, and aggregates the scores. No task specs to write.
• Self-contained HTML dashboard — score bars, SVG radar, tool-call timeline, click-to-expand drill-down. One file, no external requests.
• CI gate — --fail-under and --fail-under-mode UNSAFE_CALL=90 exit non-zero on regressions.
• BFCL validation with per-detector P/R/F1 and Cohen's κ against the judge — see BENCHMARKS.md.

Bring your own judge key

The harness ships no API key and no hosted service. Default scoring is heuristic-only and makes no network calls. The optional LLM judge reads your own GROQ_API_KEY / OPENROUTER_API_KEY / NVIDIA_API_KEY, sends it only to the provider you pick, and never writes it to the cache or any report. --judge ollama runs against a local model with no key at all.

The judge is deliberately never Claude, GPT, or Gemini — those are the agents under test, and self-preference bias would undermine the scores.