backslide

Drop yesterday's eval run and today's. Get back only the rows that got worse, with the reason attached.

Live: https://backslide.benrichardson.dev

---

what it is

You changed one line of a system prompt. You re-ran your eval. The pass rate went from 94% to 91% and you have two 300-row output files and no idea which three per cent moved or why.

Backslide joins the two runs, scores only things that can honestly be called worse, and hands back the subset that broke. It reads promptfoo JSON and JSONL, OpenAI Batch output, OpenAI Evals exports, oaieval logs, raw chat-completion dumps, generic JSONL and CSV. Everything happens in the browser tab: an eval output file holds your prompts, your customers' inputs, your model's answers and often a reference set you paid to have written, which is exactly the material nobody wants to paste into a hosted diff tool.

The part that is genuinely hard, and the part nothing else does, is the trajectory alignment. Every agent-evaluation library scores a run against a hand-written expected sequence. Backslide diffs two real runs against each other.

how it works

The join is the product

If the two rows being compared were never the same test case, every number printed afterwards is fiction. So the join is a cascade, and no stage is trusted blind.

1. The file's own case id — then verified. Every id-based dialect keys on a position: promptfoo's testIdx, OpenAI's data_source_idx, oaieval's trailing ordinal. Insert three test cases at the top of your config and every id points at the wrong row — at 100% coverage, with the tool's highest confidence label. So after the key join, Backslide counts how many inputs the two files share and how many the ids actually lined up. If the files have 17 inputs in common and the ids matched none of them, the ids are wrong and are thrown away.

2. Content hashing, with per-file affix stripping. The changed system-prompt line is by construction common to every row of the second file and absent from the first, so the longest common prefix and suffix are computed within each file and stripped before hashing. Four key functions are scored (full text, affix-stripped, last paragraph, last 200 characters) and the one with the best matched-minus-ambiguity score wins; the winner is named in the UI. Row order is irrelevant — a hash join does not care.

3. Similarity assignment, with abstention. Whatever is left is matched on character 3-gram Jaccard by greedy maximum-weight matching, which cannot double-assign. A pair survives only if it beats, by a factor of 1.8, the best score either row achieves with any other row. Measured on real paraphrase pairs the true partner scores 0.18–0.42 against a best impostor of 0.08–0.14 — so a fixed threshold rejects every genuine paraphrase, and the margin is what carries the signal. Anything below the margin is reported unmatched, never guessed.

Rows with byte-identical inputs are reported as a group with a net change and excluded from the per-row count. Pairing them is a guess by definition — an embedding model would give them identical vectors too — and the regression count would otherwise depend on which arbitrary pairing was chosen.

Only checks that can say "worse"

check	a flip means
errored	the call errored, returned nothing, or was cut off at the token limit
carried-assertion	an assertion your own harness recorded as passing now fails
is-json	it parsed before and does not now
json-schema	both parse; the first satisfies your schema and the second does not
expected-match	it matched your reference column and no longer does
length-bounds	it was inside your bounds and is now outside
latency-regression	slower by both the ratio and the floor
cost-regression	dearer by both the ratio and the floor
trajectory	a tool step was replaced or vanished

Edit distance and length change are shown but marked changed, not judged: a diff of 400 characters is not worse than a diff of 3.

Five candidates were cut against that test, and the reasons are recorded in CONTRIBUTING.md so nobody re-adds them. The sharpest is ROUGE-L against a reference: measured on a harmless paraphrase it drops 0.50, five times any sensible threshold, so it flags precisely the thing this tool exists not to flag.

Each row is counted once

The checks are not independent. An errored row has an empty output, so JSON.parse('') throws and a naive implementation reports a run that hit a rate limit as "300 of them stopped returning valid JSON" — sending you to debug a JSON-formatting bug that does not exist. The first check that explains a row owns it; everything below it is marked not-counted. That is what makes the per-check column sum to the regression total, which is what allows the headline to be generated from the summary rather than transcribed.

Errors are also split out of the regression headline. A 429 storm did not happen because of your prompt, and the fix is to re-run.

Trajectories are aligned, not zipped

Tool-call sequences are compared with Needleman–Wunsch using affine gap penalties (Gotoh's three-matrix form) over an alphabet of toolName:hash(normalisedArguments).

15 steps, one inserted at index 2

  aligned:  ==+=============     1 gap
  zipped:                        14 "differences", 13 of them fiction

An inserted step is a divergence, not a regression: a longer path is not a worse path. Only a step that was replaced or that vanished counts. Scoring is exact +3, same-tool-different-args +1, mismatch −2, gap open −4, gap extend −1 — and the relationship between gap-open and mismatch is load-bearing: two gaps (−8) must cost more than one substitution (−2), or a tool swap is absorbed as a deletion plus an insertion and reported as two structural changes instead of one.

Argument normalisation is exposed as editable rules with a live symbol alphabet and a collision count, because the two failure directions are not symmetric: under-normalising fails loudly (you see 300 regressions, disbelieve them, and tighten the rules) while over-normalising fails silently (you see zero and ship the bug). That asymmetry is why the defaults are conservative and why sorting arrays is off by default.

browser APIs used

• Web Workers (module) — parse, join, score, align and export off the main thread. The worker retains the parsed rows, so flipping a normalisation rule recomputes over memory rather than re-reading the files.
• File API + drag-and-drop + <input type=file> — both, because iOS Safari cannot drag and drop.
• File System Access / anchor download, Web Share (level 2), Clipboard
• Service Worker (vite-plugin-pwa) — installable, and the samples are precached so the demo works offline too.

No WASM. No SharedArrayBuffer. No model. No tokenizer. No runtime network call on the data path.

security / privacy model

Protected

• Both files are read, joined, scored and exported inside the tab. No code path in Backslide puts file content into a network request.
• Nothing about your files is stored. The only persisted values are your theme and your thresholds.

Not protected

• GitHub Pages logs the request for the page itself, as any host would.
• The artefacts you download quote your model's output verbatim — that is the point of them, and it means they are as sensitive as the files you started with.

Trust model

• The static bundle served by GitHub Pages and the TLS chain to it.
• A Cloudflare Web Analytics beacon (anonymous page views, no cookies, no fingerprinting) and the feedback widget on feedback.benrichardson.dev, which sends only what you type into it.
• Those two are the only third-party origins the page may contact, and the Content-Security-Policy in the page header is where that is enforced rather than promised. Backslide does not claim to make no network requests at all; it claims that none of them can carry your files, and the policy is how you check that yourself.

honest limits

Stated permanently in the interface, not only here.

• It does not read. No language model. If the second run is worse prose, Backslide says nothing changed and is wrong.
• Token counts come from your file. Backslide ships no tokenizer; a blank cell means your export did not carry the number.
• The join can be wrong, and it tells you how each pair was made so you can check.
• Rows with identical inputs cannot be told apart by any method, and are reported as a group.
• promptfoo eval --filter-failing already re-runs everything failing in your second run. What Backslide adds is the narrower set — what passed before and fails now — plus the trajectory alignment.
• This is a snapshot, not CI. It does not gate your pull request.

stack

• Vite 7 + vanilla TypeScript
• @cfworker/json-schema (eval-free, so it survives a strict CSP where ajv cannot), diff, fastest-levenshtein — three bundled dependencies, ~13 kB gzipped combined
• Vitest, 70 tests
• GitHub Pages, deployed by GitHub Actions

No cookies, no fingerprinting, no third-party fonts. Anonymous, cookie-less page-view counts via Cloudflare Web Analytics — no personal data, no cross-site tracking.

local development

npm install
npm run dev      # vite dev server on :5173
npm test         # run the vitest suite
npm run build    # produce dist/ for deploy
npm run preview  # serve dist/ locally

npm run build and npm test both regenerate public/samples/ from a seeded, committed script, so the counts in the tests and the counts in the files cannot drift.

deploying

A push to main triggers .github/workflows/deploy.yml, which runs tests, builds, and deploys dist/ to GitHub Pages. The custom domain is set via public/CNAME — point a CNAME DNS record for backslide.benrichardson.dev at ben-gy.github.io.

license

GNU Affero General Public License v3.0 or later, with an attribution requirement added under section 7(b) — see ADDITIONAL-TERMS.md.

In short: you may run, modify, redistribute and even sell this, but if you distribute it — or run a modified version where other people can reach it — you have to publish your source under the same licence and keep the attribution. A separate commercial licence without those obligations is available on request: hi@ben.gy.

Third-party components keep their own licences — see THIRD-PARTY-NOTICES.md.