whatbroke: see the behavioral diff between two runs of your agent

[arthi-arumugam-git]

I built an open source CLI called whatbroke that diffs what an agent actually did between two runs. You change something (model, prompt, tool description), run the same job again, and it reports the behavioral diff: tool calls dropped or added, arguments that changed, outputs that flipped, cost and latency moves.

Since ADK emits OpenTelemetry spans, you can export those as OTLP JSON and import them, no agent code changes. It also reads Langfuse exports, LangSmith run dumps, and its own plain JSONL trace format.

npx whatbroke-cli import spans.json
npx whatbroke-cli diff before.jsonl after.jsonl

Repo: https://github.com/arthi-arumugam-git/whatbroke

The thing that pushed me to build it: I swapped a 3b model for the 1b variant in a tool calling agent and the replies stayed plausible while the model started passing the tool's own JSON schema as the tool arguments. A text diff sees nothing. The behavior diff catches it. MIT licensed, feedback very welcome.