whatbroke: a CLI that diffs your local agent's behavior between two models

[arthi-arumugam-git]

I run small local models (llama3.2:3b, qwen2.5:3b) as the brains of a tool-calling support agent, and I kept hitting the same problem: swap the model, read a few transcripts, everything looks fine, and then days later you notice a tool call quietly vanished or changed shape.

So I built whatbroke, an MIT CLI that diffs an agent's behavior between two runs. You record a JSONL trace before the change and one after (there's a local proxy mode so you can capture traces from any stack by pointing your base URL at it), then whatbroke diff before.jsonl after.jsonl reports which tool calls appeared/disappeared, which args drifted, ordering changes, token/latency movement, and outcome flips. Because local models are nondeterministic, you can record each scenario a few times and findings come back with a flap rate, so anything the baseline already flip-flopped on gets demoted to noise.

The experiment that convinced me it was worth sharing: swapping llama3.2:3b for qwen2.5:3b (same size class) in the support agent. qwen added an account-lookup call llama always skipped, which looks like an upgrade, but it fired the lookup and the cancellation in parallel, so the cancel args were picked before the lookup result existed. It invented a plausible Stripe-shaped account id in 3 of 3 runs while telling the customer the account was cancelled. Also roughly 2x the tokens at the same parameter count. Full traces are in the repo's findings doc.

Everything runs offline, no keys, no accounts: https://github.com/arthi-arumugam-git/whatbroke (npx whatbroke-cli to try it). If you swap GGUFs under an agent and something silently changes, I'd genuinely like to hear about it, I collect these now.