whatbroke: diff what your agent did between two runs

[arthi-arumugam-git]

I built a small open source CLI called whatbroke. It diffs an agent's behavior between two runs and reports what actually changed: tool calls dropped or added, arguments that drifted, outputs that flipped, cost and latency moves.

If you trace your Agno agents with Langfuse, you can export from there and diff two versions of your agent without touching agent code. It also imports OTLP JSON spans that use the GenAI semantic conventions and LangSmith run dumps, or you can record its plain JSONL format directly.

npx whatbroke-cli import langfuse-export.json
npx whatbroke-cli diff before.jsonl after.jsonl

Repo: https://github.com/arthi-arumugam-git/whatbroke

The reason I built it: I swapped a tool calling agent to a 3x smaller model and the replies kept reading fine while the tool arguments quietly broke. Text diffs see nothing there. MIT, 87 tests, exit code 1 on breaking changes so it can gate a model swap in CI. Feedback welcome.