Skip to content

What is Hairspring

Tom Barber edited this page Nov 5, 2026 · 3 revisions

What is Hairspring?

Hairspring is a testing tool for bugs that depend on timing and ordering. It runs a system many times, each time with a different but repeatable sequence of events, looks for a run that breaks a rule you care about, and hands you a small file (a bundle) that replays that run.

This page explains the problem it addresses, the idea behind it, and the three ways it can run your code. For the words in bold, see Concepts-and-Vocabulary.

The problem: bugs that only happen sometimes

Most tests check one path through a program. Give the function this input, expect that output. That works well when the program does the same thing every time.

Many real systems do not. As soon as you have more than one thread, more than one process, a network, retries or timers, the program's behaviour depends on the order in which things happen, and that order is decided by things outside your code: the operating system's thread scheduler, how busy the machine is, how long a message takes to arrive. Run the same test a thousand times and you may get several different orders. If one rare order triggers a bug, you get a test that fails once a week on the CI server and never on your laptop.

Two difficulties come together here:

  • Finding the bad order. It may need two threads to be interrupted at exactly the wrong instruction. Waiting for it to happen by chance can take a very long time.
  • Reproducing it once found. Even if it happens, you usually cannot make it happen again, because the next run takes a different order. A bug you cannot reproduce is very hard to fix, and harder still to prove fixed.

The idea: make the order of events an input

Deterministic simulation testing turns the order of events into something the test controls. Instead of letting the operating system or the network decide what happens next, a controller makes every such choice, and it makes those choices from a seed (a number that drives a pseudo-random sequence).

That gives you two properties that ordinary tests lack:

  1. Different seeds explore different orders. Seed 1 might deliver message A before B; seed 2, B before A. Running hundreds of seeds is a systematic way to try hundreds of orders.
  2. The same seed gives the same order. If seed 7 fails, running seed 7 again takes the same path and fails the same way. The failure is no longer a rumour: it has a name and can be replayed.

The approach is not new. Hairspring's README credits FoundationDB's simulation testing, TigerBeetle's VOPR, Antithesis, and the record/replay work of rr and Hermit as inspiration. What Hairspring adds is a single open-source tool that does this on your own machine, with a command line, a REST API and web UI, an MCP server for AI agents, and a GitHub Action.

What Hairspring does with a failure

According to the README, Hairspring:

  • finds: explores many executions from a seed;
  • shrinks: cuts a failing run down to a minimal trace that still produces the same failure (for the simulation backend, a 1-minimal trace by delta debugging; "1-minimal" means removing any single remaining step makes the failure go away);
  • replays: writes a one-file text bundle that replays the run, checked against the program version;
  • checks determinism: hairspring doctor runs the same seed twice and reports the first point where the two runs differ.

Here is the README's example, using a built-in demo of a message consumer that can apply the same message twice:

$ hairspring test pubsub-buggy --seed 5
hairspring: exploring pubsub-buggy (seed 5)
FAILED  pubsub-buggy (sim)
  invariant violated; shrunk from 23 to 8 steps
  steps (8):
      1. Deliver(1, 0)
      2. Tick
      ...
  bundle written: .hairspring/bundles/pubsub-buggy-d3858727.hairspring
$ hairspring replay .hairspring/bundles/pubsub-buggy-d3858727.hairspring   # exits 1 while the bug exists

The failing run took 23 steps; Hairspring shrank it to 8 that still break the rule, and wrote a bundle you can replay.

Three ways to run your code

How much control Hairspring has over the order of events depends on how your code is run. It has three backends. The table is from the README.

Backend You provide Determinism macOS Linux x86-64 Status
sim A state machine implementing the Sim trait (Rust) Total: the state is plain data Yes Yes Stable core
wasm A WebAssembly module using hairspring.choose/rand/fail imports Total for the guest (NaN canonicalisation, fuel, memory limits); snapshot/fork of memory Yes Yes Working
process An unmodified Linux program (threads, sockets) Exploration by scheduler seed under Hermit; failures reproduce exactly No Yes (bare metal or PMU-capable VM) Experimental

In everyday terms:

  • sim: you write a model of your system in Rust as a state machine: "here are the things that could happen next; here is the state after one of them happens; here is the rule that must hold". Hairspring picks the next event each time. Because the whole state is ordinary data, it can be copied and explored from any point. This gives the most control, at the cost of writing a model.
  • wasm: your code is compiled to WebAssembly and asks Hairspring whenever it needs a decision (choose), a random number (rand) or to report a failure (fail). The sequence of answers is the trace.
  • process: you hand Hairspring a real, unmodified Linux program. It runs the program under Hermit, Meta's deterministic-execution tool, which runs one thread at a time under a seeded scheduler with virtual time. You do not change your program, but this backend needs Linux on x86-64 with hardware performance counters, and it is experimental.

How-Deterministic-Simulation-Works explains what "controlling the scheduler" means and why it needs those counters.

How you tell Hairspring what "wrong" means

Hairspring needs a rule to check, called a property. How you state it depends on the backend:

  • a sim target implements violated(), which says whether the current state breaks the rule;
  • a process target fails if the program exits non-zero, prints a line starting HAIRSPRING_VIOLATION:, or (with a guest SDK) reports a named property as false. SDKs exist for C (also used for C++), Go, Rust, Python, Java and .NET; see Guest-SDKs-by-Language and, for which of those have passed the acceptance test under Hermit, Limits-and-Honesty.

A program that times out is reported as inconclusive, never as a pass, unless you set hang_is_violation = true.

Ways to use it

From the README:

  • CLI: hairspring test, replay, doctor, confirm, show, results and more.
  • Server: hairspring serve gives a REST API with live events, a web UI and MCP at http://127.0.0.1:7878.
  • AI agents: hairspring mcp (stdio) or POST /mcp exposes 13 tools, among them list_targets, start_run, get_failure, replay_bundle, get_failure_checks and cancel_run, plus read tools such as get_run, list_runs, get_properties and list_trials. Everything that comes from the system under test is sanitised and framed as untrusted data for the agent.
  • Teams: SQLite locally, or PostgreSQL with remote workers; a Docker Compose file is included.
  • Dapr services: a simulated Dapr sidecar (hairspring-dapr) with seeded duplicates, reordering, drops, crashes and conflicts, for an implemented subset of Dapr only.

The web UI

hairspring serve also serves a web UI, which is the easiest way to look at runs, failures and workers once you have a few. It lists runs and failures with filters, shows each run's trials and seeds, lets someone with a run token start a search or a sweep, and shows how far each failure has been checked (its confirmation). It has a Light, Dark and System colour theme and works on a phone-width window.

A failure page: the shortest failing steps for pubsub-buggy, with Replay and Download bundle buttons and a Confirmation section that reads Candidate, with no checks recorded.

A failure page for a built-in simulation target. "Candidate" means no check has repeated the failure yet; confirming one needs the process backend with Hermit on Linux. Your-First-Run has the full tour.

What Hairspring is not

The README is direct about this. Hairspring does not claim:

  • bit-exact replay of arbitrary programs on macOS (there is no supported way to do this);
  • to be a general-purpose deterministic hypervisor;
  • that Hermit, a young open-source project, is production-hardened.

See Limits-and-Honesty for the full list.

Next

Clone this wiki locally