Repository navigation
Concepts and Vocabulary
Hairspring uses a small set of words consistently. This page defines each one and gives one concrete example. The examples use the built-in demo targets and the C language fixture from the repository, so you can try them yourself once you have followed Installation and Your-First-Run.
If you only want one-line definitions, see Glossary.
flowchart LR
T[Target<br/>your system + a property] --> R[Run]
R --> T1[Trial, seed 1]
R --> T2[Trial, seed 2]
R --> T3[Trial, seed 3 ...]
T2 -->|violation| B[Bundle<br/>seed + recorded inputs]
B --> RP[Replay<br/>same seed, same schedule]
B --> C[Confirmation<br/>recorded checks]
A target is tested by a run, which is made of many trials. Each trial uses a seed, which decides the schedule. A trial ends with a verdict. A failing trial gets a bundle, which you can replay or put through confirmation. A sweep is a run that keeps going after failures so it can count them.
What it is. The thing you are testing, plus the rule it must obey, registered under a name. A target has a kind
(sim, wasm or process, matching the three backends) and a budget that limits how much exploring a run may do.
Example. The built-in target pubsub-buggy is described as "At-least-once consumer; check, apply and record are
separate steps (duplicate effect)". Its fixed twin, pubsub-atomic, does the three steps atomically and should pass.
hairspring targets lists them.
A process target for your own program is a few lines in hairspring.toml, as in the quick start:
[target.at-c-buggy]
kind = "process"
command = ["/path/to/hairspring/examples/language-fixtures/c/build/gccO2/at-c", "buggy"]
trial_timeout_secs = 120
[target.at-c-buggy.budget]
max_trials = 20What it is. The rule that must hold. A trial that breaks it is a failure. Properties are how you tell Hairspring what "wrong" means.
Example. The C fixture at-c reports a property named c.counter_exact through the C SDK, which says the final
counter equals 4,000. In buggy mode, a lost-update race between threads can make the counter come out lower, and the
property is then false.
Properties come in kinds (from the guest protocol): always (must hold every time it is checked), sometimes (must be
true at least once across the run), reachable (must be reached at least once), unreachable (must never be reached)
and always_or_unreachable.
What it is. One attempt to make the target fail. On the process and wasm backends a trial is one seed: the
program runs once from start to finish. On a sim target a trial is one exploration round.
Example. hairspring test at-c-buggy --seed 1 runs the C fixture under Hermit once per trial, each trial with a
different scheduler seed, up to the max_trials = 20 budget.
What it is. One invocation of testing a target: a job that runs many trials. A normal run (a search) stops at its first confirmed failure. A sweep runs its whole budget.
Example. hairspring test lock-lease --seed 7 starts one run of the lock-lease demo. hairspring results lists
stored runs and hairspring show <run id> shows one.
What it is. For each run, Hairspring can keep a fingerprint (a SHA-256 hash) of each input the run executed with: the program under test, the target name and, where the backend reports them, its settings. Only fingerprints are kept, never the values, so environment values and paths are not stored. A fingerprint can tell you that an input is the same or different between two runs; it cannot tell you what it was, and Hairspring never shows one. It also records whether the worker that reported them was trusted.
What it is not. It is not a comparison. Nothing compares runs by these fingerprints yet.
Which runs have it. Runs stored before this was added have none, and Hairspring treats them as "not recorded" rather
than guessing. Only built-in simulated targets record them, whether run by the server, hairspring test or a remote
worker of this release. Hermit and WASM runs, and runs by workers from older releases, record none.
If a job is run again. When a job is given to another worker after the first one was lost, the new worker's trials replace the first worker's, and its fingerprints replace the first worker's too.
What it is. A number that drives every choice Hairspring makes during a trial. The same seed on the same setup makes the same choices. Different seeds make different choices, which is how Hairspring explores.
Example. On the process backend the seed is passed to Hermit as --sched-seed=<seed>, which decides which thread
runs next and when it is interrupted. In the repository's determinism measurements, 20 Hermit seeds on a 4-thread racy
C program gave 8 distinct interleavings, and each seed gave the same result in 5 of 5 runs.
What it is. The order in which things actually happened in a trial: which thread ran when, which message was
delivered first. On the process backend Hairspring compares schedules by a schedule fingerprint, which the
showcase pages describe as a hash of five fields of Hermit's run summary.
Example. In the README's pubsub-buggy failure, the schedule is the list of steps:
Deliver(1, 0), Tick, Tick, Deliver(0, 0), Step(1), Step(0), Step(1), Step(0). Change one step and you may
get a different outcome.
What it is. How a trial ended. In a sweep's stored rows the verdicts are:
| Verdict | Meaning |
|---|---|
pass |
the trial ran and no property failed |
violation |
a property failed, the program printed HAIRSPRING_VIOLATION:, Hermit reported a deadlock, or the program exited non-zero |
inconclusive |
it timed out, or was cut short by the wall-clock budget; not a pass |
unusable |
its reported events could not be used (lost, cut or contradictory); not a pass |
infra |
the host or Hermit failed, not the program ("failed in the host") |
Example. A trial of the C fixture that runs past trial_timeout_secs is inconclusive. The quick start is firm
about this: such a trial is "never a pass". Only pass and violation count as rated trials when a rate is stated.
What it is. A small text file that records what is needed to replay one failing trial: the seed, the build, the
target, the backend, backend-specific parameters (param.* lines) and the path (the recorded steps or choices).
Bundles from the process backend also record the execution environment (kernel, CPU, loader file hashes, Hermit
executable hash and so on) so a replay can tell you if the machine has changed.
Example. The README's run wrote .hairspring/bundles/pubsub-buggy-d3858727.hairspring. The layout looks like this
(values shortened for illustration):
hairspring-bundle v2
seed=5
build=...
target=pubsub-buggy
backend=sim
path=...,...,...
A process bundle carries param.sched_seed=... and, when it records the environment, starts with
hairspring-bundle v3.
What it is. Running a bundle again to see whether the failure comes back. Replay refuses a bundle recorded against different code where it can tell.
Example. hairspring replay .hairspring/bundles/<bundle>.hairspring. For a process bundle the exit code says what
happened:
| Exit | Meaning |
|---|---|
| 0 | not reproduced, on the environment it was recorded in |
| 1 | reproduced: the property is still violated |
| 2 | error, including a replay refused because the dynamic loader changed |
| 3 | not reproduced on a changed environment: inconclusive, not evidence the bug is fixed |
So after you fix a bug, replaying its bundle and getting exit 0 on the same machine is good evidence the fix works for that schedule. Exit 3 tells you nothing either way.
What it is. A run that does not stop at the first failure. It runs every trial in its budget, stores every trial, and reports how many were violations out of the rated trials, with a 95% interval.
Example. hairspring test checkout --sweep --trials 1000 --seed 1 runs trials at seeds 1 to 1000. The report
gives a line such as sweep: <n> trials: <a> pass, <b> violation, <c> inconclusive, <d> unusable, <e> infra, names the
lowest failing seeds, and states the rate. See Sweeps-and-Confirmation.
What it is. A set of explicit, recorded checks that a failure is real and repeatable, run by
hairspring confirm <run-id | failure-id | bundle> on the process backend. The checks are: rerunning the same seed,
replaying the bundle with the environment check, optionally running a fixed build at the same seed (a control), and
optionally Hermit's strict determinism verify. The failure then gets a state such as candidate, rerun_repeated,
confirmed or verified.
Example. hairspring confirm <failure-id> --control at-c-fixed would rerun the failing seed, replay the bundle,
and run a second target that uses the fixed build at the same seed and flags (the target name here is your own;
the --control flag is documented in the README). See Sweeps-and-Confirmation.
What it is. A small library you add to a program run on the process backend, so the program can declare named
properties and report them to Hairspring. "Guest" means the program under test; Hairspring is the host. Without an SDK,
Hairspring still works: a non-zero exit is a failure. With one, you get named properties tracked across trials.
Example. With the C SDK:
#include "hairspring.h"
HS_ALWAYS(nbackfill <= mxframe, "wal.ckpt.nbackfill_le_mxframe", "nBackfill <= live mxFrame", nbackfill, mxframe);
HS_SOMETIMES(waited, "wal.writer.waited", "a writer waited for the lock");SDKs exist for C (also used for C++), Go, Rust, Python, Java and .NET. See Guest-SDKs-by-Language, and Limits-and-Honesty for which have passed the acceptance test under Hermit.
What it is. The messages a guest SDK writes so Hairspring can read the program's properties. Each one is a line on
standard output or standard error that starts with HAIRSPRING_EVENT (one space) followed by a JSON object. Other
lines are ordinary program output and are ignored. The format is the frozen guest event protocol v1.
Example. A process says hello first, sends a catalog line for each property id at startup, an assert line with
its counts at exit, and an end line last. A line such as
HAIRSPRING_EVENT {"v":1,"kind":"work","proc":"main","ops":4000}
reports work done (the field names are from the protocol's message table; the values here are illustrative). You do not write these by hand: the SDK does. If the events are lost or contradictory, the trial is marked degraded and is not counted as a pass.
The server also streams live events about runs (progress, finished) to the web UI and API clients; that is a separate thing from guest events.
What it is. The part of Hairspring that actually runs a trial. There are three: sim (a Rust state machine),
wasm (a WebAssembly module) and process (an unmodified Linux program under Hermit). See What-is-Hairspring.
Example. The C fixture target above has kind = "process", so it uses the process backend and needs Linux on
x86-64.
What it is. A failure Hairspring recorded, together with the evidence about it. Its state is derived from the stored confirmation checks, never stored by hand, so it cannot drift from them.
Example. A failure whose confirmation reruns all repeated it, but whose bundle replay has not (yet) repeated it, is
rerun_repeated. Hairspring's own wording is careful: that state "is not 'reproduced'", and only confirmed and
verified reach the generated claim text.
In the web UI. A failure's page shows its state as a ladder: Candidate (no check has repeated the failure), Rerun repeated, Confirmed and Verified, with two ends that are not rungs, Not reproducible and Control failed. A failure found on a simulation target has no checks, so it stays a candidate.

The ladder for a simulation failure. None of the states says the failure is fixed.
What it is. A process that takes queued jobs and runs them. hairspring serve runs one inside the server unless you give it --workers 0, and hairspring worker registers a remote one. The Workers page lists each worker's state (Busy, Idle or Not seen recently), whether it is Trusted or Untrusted (an administrator sets this), and what it said about itself when it registered.
Example. The Workers page of a fresh hairspring serve lists one worker, embedded-0, Idle and Trusted. Data from an untrusted worker is shown, labelled as unverified, and claims need trusted workers.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference