Diff what your browser agent actually did between two runs

[arthi-arumugam-git]

Browser agents are especially prone to this: you bump the model or tweak the task prompt, the run still finishes, and only later you notice it took a different path or skipped a step.

I built whatbroke, an open source CLI that diffs an agent's behavior between two recorded runs. It aligns the runs and reports dropped or added tool calls, changed arguments, flipped outputs, and cost and latency moves, each with a severity. Exit code 1 on breaking changes, so it can gate an upgrade in CI.

It reads OTLP JSON spans that use the GenAI semantic conventions, Langfuse exports, LangSmith run dumps, or its own plain JSONL trace format if you want to record runs yourself.

npx whatbroke-cli import spans.json
npx whatbroke-cli diff before.jsonl after.jsonl

Repo: https://github.com/arthi-arumugam-git/whatbroke

MIT, 87 tests. Curious how people here currently check that a model bump didn't change what their browser agent actually does.