whatbroke: a diff for your agent behavior between two runs

[arthi-arumugam-git]

I built a small CLI that diffs an AI agent behavior between two runs. You record a JSONL trace before a change (model swap, prompt tweak, framework upgrade) and one after, then whatbroke diff before.jsonl after.jsonl shows which tool calls disappeared, which arguments drifted, where cost and latency moved, and which outputs flipped.

The reason it exists: I swapped models on an agent and it kept telling users their subscription was cancelled while it had quietly stopped calling the cancel tool. The replies read fine. The behavior broke.

There is a proxy mode (whatbroke record) so you can capture traces from an Agno agent without touching code, just point your base URL at it and run the agent as usual. Since agents are nondeterministic you can record each scenario a few times and findings come back with a flap rate, so noise your agent was already making does not get blamed on the change.

Deterministic, fully offline, MIT: https://github.com/arthi-arumugam-git/whatbroke

If you try it on an Agno agent I would genuinely love to hear what it catches.