Diff what actually changed in your agent after a model swap behind LiteLLM

[arthi-arumugam-git]

LiteLLM makes swapping the model behind an agent a one line change, which is great, and also how I broke my own agent without noticing. The replies still looked fine while the tool calls underneath had changed.

So I built whatbroke, an open source CLI that diffs an agent's behavior between two runs: tool calls dropped or added, arguments that drifted, outputs that flipped, cost and latency moves. If you log your LiteLLM calls to Langfuse, you can export from there and diff two model configs directly. It also reads OTLP JSON spans that use the GenAI semantic conventions, and LangSmith run dumps.

npx whatbroke-cli import langfuse-export.json
npx whatbroke-cli diff gpt4o.jsonl cheaper-model.jsonl

Repo: https://github.com/arthi-arumugam-git/whatbroke

MIT, 87 tests, exit code 1 on breaking changes so you can gate a model swap in CI. Would love to hear what people here check today when they move traffic to a cheaper model.