Repository navigation
What is Hairspring
Hairspring is a testing tool for bugs that depend on timing and ordering. It runs a system many times, each time with a different but repeatable sequence of events, looks for a run that breaks a rule you care about, and hands you a small file (a bundle) that replays that run.
This page explains the problem it addresses, the idea behind it, and the three ways it can run your code. For the words in bold, see Concepts-and-Vocabulary.
Most tests check one path through a program. Give the function this input, expect that output. That works well when the program does the same thing every time.
Many real systems do not. As soon as you have more than one thread, more than one process, a network, retries or timers, the program's behaviour depends on the order in which things happen, and that order is decided by things outside your code: the operating system's thread scheduler, how busy the machine is, how long a message takes to arrive. Run the same test a thousand times and you may get several different orders. If one rare order triggers a bug, you get a test that fails once a week on the CI server and never on your laptop.
Two difficulties come together here:
- Finding the bad order. It may need two threads to be interrupted at exactly the wrong instruction. Waiting for it to happen by chance can take a very long time.
- Reproducing it once found. Even if it happens, you usually cannot make it happen again, because the next run takes a different order. A bug you cannot reproduce is very hard to fix, and harder still to prove fixed.
Deterministic simulation testing turns the order of events into something the test controls. Instead of letting the operating system or the network decide what happens next, a controller makes every such choice, and it makes those choices from a seed (a number that drives a pseudo-random sequence).
That gives you two properties that ordinary tests lack:
- Different seeds explore different orders. Seed 1 might deliver message A before B; seed 2, B before A. Running hundreds of seeds is a systematic way to try hundreds of orders.
- The same seed gives the same order. If seed 7 fails, running seed 7 again takes the same path and fails the same way. The failure is no longer a rumour: it has a name and can be replayed.
The approach is not new. Hairspring's README credits FoundationDB's simulation testing, TigerBeetle's VOPR,
Antithesis, and the record/replay work of rr and Hermit as inspiration. What Hairspring adds is a single open-source
tool that does this on your own machine, with a command line, a REST API and web UI, an MCP server for AI agents, and a
GitHub Action.
According to the README, Hairspring:
- finds: explores many executions from a seed;
- shrinks: cuts a failing run down to a minimal trace that still produces the same failure (for the simulation backend, a 1-minimal trace by delta debugging; "1-minimal" means removing any single remaining step makes the failure go away);
- replays: writes a one-file text bundle that replays the run, checked against the program version;
-
checks determinism:
hairspring doctorruns the same seed twice and reports the first point where the two runs differ.
Here is the README's example, using a built-in demo of a message consumer that can apply the same message twice:
$ hairspring test pubsub-buggy --seed 5
hairspring: exploring pubsub-buggy (seed 5)
FAILED pubsub-buggy (sim)
invariant violated; shrunk from 23 to 8 steps
steps (8):
1. Deliver(1, 0)
2. Tick
...
bundle written: .hairspring/bundles/pubsub-buggy-d3858727.hairspring
$ hairspring replay .hairspring/bundles/pubsub-buggy-d3858727.hairspring # exits 1 while the bug exists
The failing run took 23 steps; Hairspring shrank it to 8 that still break the rule, and wrote a bundle you can replay.
How much control Hairspring has over the order of events depends on how your code is run. It has three backends. The table is from the README.
| Backend | You provide | Determinism | macOS | Linux x86-64 | Status |
|---|---|---|---|---|---|
sim |
A state machine implementing the Sim trait (Rust) |
Total: the state is plain data | Yes | Yes | Stable core |
wasm |
A WebAssembly module using hairspring.choose/rand/fail imports |
Total for the guest (NaN canonicalisation, fuel, memory limits); snapshot/fork of memory | Yes | Yes | Working |
process |
An unmodified Linux program (threads, sockets) | Exploration by scheduler seed under Hermit; failures reproduce exactly | No | Yes (bare metal or PMU-capable VM) | Experimental |
In everyday terms:
-
sim: you write a model of your system in Rust as a state machine: "here are the things that could happen next; here is the state after one of them happens; here is the rule that must hold". Hairspring picks the next event each time. Because the whole state is ordinary data, it can be copied and explored from any point. This gives the most control, at the cost of writing a model. -
wasm: your code is compiled to WebAssembly and asks Hairspring whenever it needs a decision (choose), a random number (rand) or to report a failure (fail). The sequence of answers is the trace. -
process: you hand Hairspring a real, unmodified Linux program. It runs the program under Hermit, Meta's deterministic-execution tool, which runs one thread at a time under a seeded scheduler with virtual time. You do not change your program, but this backend needs Linux on x86-64 with hardware performance counters, and it is experimental.
How-Deterministic-Simulation-Works explains what "controlling the scheduler" means and why it needs those counters.
Hairspring needs a rule to check, called a property. How you state it depends on the backend:
- a
simtarget implementsviolated(), which says whether the current state breaks the rule; - a
processtarget fails if the program exits non-zero, prints a line startingHAIRSPRING_VIOLATION:, or (with a guest SDK) reports a named property as false. SDKs exist for C (also used for C++), Go, Rust, Python, Java and .NET; see Guest-SDKs-by-Language and, for which of those have passed the acceptance test under Hermit, Limits-and-Honesty.
A program that times out is reported as inconclusive, never as a pass, unless you set hang_is_violation = true.
From the README:
-
CLI:
hairspring test,replay,doctor,confirm,show,resultsand more. -
Server:
hairspring servegives a REST API with live events, a web UI and MCP athttp://127.0.0.1:7878. -
AI agents:
hairspring mcp(stdio) orPOST /mcpexposes 13 tools, among themlist_targets,start_run,get_failure,replay_bundle,get_failure_checksandcancel_run, plus read tools such asget_run,list_runs,get_propertiesandlist_trials. Everything that comes from the system under test is sanitised and framed as untrusted data for the agent. - Teams: SQLite locally, or PostgreSQL with remote workers; a Docker Compose file is included.
-
Dapr services: a simulated Dapr sidecar (
hairspring-dapr) with seeded duplicates, reordering, drops, crashes and conflicts, for an implemented subset of Dapr only.
hairspring serve also serves a web UI, which is the easiest way to look at runs, failures and workers once you have a few. It lists runs and failures with filters, shows each run's trials and seeds, lets someone with a run token start a search or a sweep, and shows how far each failure has been checked (its confirmation). It has a Light, Dark and System colour theme and works on a phone-width window.

A failure page for a built-in simulation target. "Candidate" means no check has repeated the failure yet; confirming one needs the process backend with Hermit on Linux. Your-First-Run has the full tour.
The README is direct about this. Hairspring does not claim:
- bit-exact replay of arbitrary programs on macOS (there is no supported way to do this);
- to be a general-purpose deterministic hypervisor;
- that Hermit, a young open-source project, is production-hardened.
See Limits-and-Honesty for the full list.
- Concepts-and-Vocabulary for the vocabulary.
- Installation and Your-First-Run to try it.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference