What Is Compared

You record what a reference implementation does for a set of scenarios: every response, every LLM prompt, every call to an outside service, every database row and every file it writes. Then you check another implementation against those recordings: a rewrite in another language, a port, or the next version of the same code.

• Black box. An implementation is a command. parityrun starts it with a fresh data folder (or a fresh copy of its own folder) and talks to it only over HTTP and its command line, so the language and framework don't matter.
• Web pages as a browser reads them. With html=True, HTML is compared tag by tag: attribute order, entity spelling, indentation and comments don't count; text, tags and attribute values do. Redirects, forms, uploads and cookies work as in a browser.
• LLM calls are part of the contract. A built-in stub speaks the OpenAI chat completions API, the Anthropic Messages API and the claude command line. Each step queues the replies its calls need, and the requests themselves (model, messages, schema) are recorded and compared, so a changed prompt fails the check.
• Outside services are stubbed, and recorded. Point the app's API base at the stub and answer for any host with a small handler; every call (method, path, query, body) is compared.
• Noise is normalised, not ignored. Ids are numbered by first appearance, times, dates and unix times become placeholders, the run's own paths and ports become labels, captured tokens become <name>. Everything else must match exactly.
• Databases by schema and rows (row order is free): SQLite files, or MySQL and PostgreSQL through a hook. See Databases.
• Pictures and videos are compared by eye. Images by structural similarity (SSIM), videos by duration, size, sound and five frames.
• No flaky goldens. record runs every scenario twice and refuses to save one whose runs differ.

Reading a check

parityrun check --impl NAME prints each failing comparison and the parity %, and exits 0 only at 100 %. It writes out/report-<impl>.json and keeps what the implementation did as out/<impl>/<name>/recording.json (with its pictures), to set beside golden/<name>/recording.json.

Review the diff of golden/ before committing a new recording: from then on a recording is only as right as the run that made it.