whatbroke, a CLI that diffs an agent's behavior between two traced runs

[arthi-arumugam-git]

I made an open source CLI called whatbroke that diffs an AI agent's behavior between two traced runs. You change something (model, prompt, a tool description), run the same job again, and it reports the behavioral diff: tool calls dropped or added, arguments that changed, outputs that flipped, cost and latency moves.

If you trace Haystack pipelines through OpenTelemetry, you can export the spans as OTLP JSON and import them, no code changes. Langfuse exports and LangSmith run dumps work too, plus a plain JSONL format for recording traces directly.

npx whatbroke-cli import spans.json
npx whatbroke-cli diff before.jsonl after.jsonl

Repo: https://github.com/arthi-arumugam-git/whatbroke

MIT, 87 tests, exit code 1 on breaking findings so it drops into CI. The case study in the repo shows a 3x smaller model silently passing a tool's own JSON schema as the tool arguments while the replies still read fine. Feedback welcome.