Repository navigation
Running Your Own Software
This is the heart of the guide. It shows how to point Hairspring at a program of your own, how it decides the program has failed, how to add the optional guest SDK, how to read what comes back, and what to keep when you find something. It ends with a small program you write yourself, start to finish.
Read Installation and Your-First-Run first. Words such as target, trial, seed, bundle and sweep are explained in Concepts-and-Vocabulary.
Please read this before you point Hairspring at anything you care about.
Hairspring is experimental. APIs and the bundle format will change.
Testing your own program as an unmodified
processtarget needs Linux on x86-64 with hardware performance counters (bare metal, or a virtual machine that passes them through), and a separately built Hermit. macOS cannot run it.Languages, as recorded in the project's support matrix. The status cells are copied exactly:
Language Status C Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere. C++ (clang++ and g++) Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere. Go Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere. Rust Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere. Python Not supported on this build: the probe program did not complete under Hermit. Java Supported on a Java-specific configuration: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build) with skid_margin = 35(the other languages: 25), JVM flags-Xint -XX:+UseSerialGC -XX:ActiveProcessorCount=1 -XX:-UsePerfData -Xshare:auto, Temurin 21.0.12.1, and a reduced fixture (50 iterations per worker instead of 1,000); not tested elsewhere..NET SDK and fixture available. Not supported on this build: the probe program did not complete under Hermit. "Supported" means passed on one configuration and no other: Ubuntu 24.04.5 LTS, kernel 6.17.0-42-generic, one bare-metal host with an Intel Xeon E5-1410 v2 (Ivy Bridge) CPU, a pinned Hermit build run as root,
hairspring 0.1.0, chaos mode withmax_timeslice_ns = 100000andskid_margin = 25. The results apply only to that configuration. The project makes no claim about other hosts, kernels or Hermit builds.The Java notes in the matrix: Trials take about 210 s as run (up to three at once; about 105 s alone); suited to nightly runs. Replaying a recorded run matched only from the same paths as the recording (JDK install, program jar and working directory). At
skid_margin = 25the probe program did not complete under Hermit. Java usedskid_margin = 35and the JVM flags above, and was measured on 2026-10-26 and 2026-10-27.For Go, 3 of the 400 trials in that acceptance run (1 in the failing arm, 2 in the passing arm) gave no verdict and are not counted.
So: if your program is written in C, C++, Go or Rust and you have a Linux x86-64 machine like the one above, you are in tested territory, with the caveat that it was tested on one machine. If it is written in Java, it is supported only on the Java-specific configuration in the table above (skid_margin = 35, the interpreter-only JVM flags, a reduced fixture, slow trials); see "Java and other JVM programs" below. If it is written in Python or .NET, the SDKs exist and work natively, but the programs did not complete under Hermit on that build, so do not expect the process backend to work for them. A program that is not written in any of those languages is untested.
Also read Limits-and-Honesty. In particular: a trial that times out is inconclusive, never a pass, and a pass with a small budget says only that those seeds did not fail.
A JVM reads the text of its own paths: the JDK install, the class path and the working directory. A bundle records hashes of the command line and the working directory (not the paths themselves), and replay runs whatever the current configuration says. If either differs from the recording, replay prints command line or working directory differs from the recording and a miss exits 3 (inconclusive). Keep the JDK install, the jar and the working directory at the same absolute paths for recording and for replay. In CI, use a fixed workspace path: a fixed checkout directory, a container with the JDK at a fixed path, or a path without per-job names. Re-record if the paths must change.
The Java configuration in the support matrix used skid_margin = 35 and the JVM flags -Xint -XX:+UseSerialGC -XX:ActiveProcessorCount=1 -XX:-UsePerfData -Xshare:auto. These are set in the target in hairspring.toml: skid_margin with the other scheduler keys, and the flags on the java command line. Nothing in this section was run here; it needs Linux x86-64 and Hermit.
A target is a named thing Hairspring can explore. You describe targets in a TOML file, hairspring.toml by default (--config PATH for another). hairspring init writes a starter file with comments; hairspring targets lists what is defined.
There are three kinds:
kind |
You provide | Runs on |
|---|---|---|
sim |
a state machine you write in Rust against the Sim trait, registered in your own binary (hairspring_cli::main_with); the built-in demos are of this kind |
macOS and Linux |
wasm |
a WebAssembly module that uses the hairspring.choose, rand and fail imports; the config key is module
|
macOS and Linux |
process |
an existing Linux program and the command that starts it | Linux x86-64 with Hermit |
The rest of this page is about process, because that is the one that tests software you already have. For sim, the Rust example in the project README and the worked example in crates/hairspring-pubsub are the place to start. For wasm targets, see Test-Logic-with-Wasm.
Unknown keys are rejected, so a typo is an error, not a silent no-op: unknown field `events_file`, expected one of `kind`, `description`, `sim`, `command`, `env`, `cwd`, `hermit_args`, `trial_timeout_secs`, `heuristic`, `sticky_permille`, `chaos`, `max_timeslice_ns`, `skid_margin`, `hang_is_violation`, `parallelism`, `pass_env`, `backend`, `module`, `budget` . That is the complete list of keys. There is no events_file key in this release; the backend has an optional events-file transport internally, but it is not switchable from the config.
[target.checkout]
kind = "process"
command = ["/opt/app/bin/checkout", "--self-test"]
trial_timeout_secs = 120
[target.checkout.budget]
max_trials = 100
wall_secs = 600The target name (checkout) is 1 to 64 characters of letters, digits, _, - and .. Names with other characters are refused: target name `my seats` must be 1-64 chars of [A-Za-z0-9_.-].
| Key | Meaning |
|---|---|
kind |
"process" |
command |
Required. The program and its arguments, as an array. Use an absolute path to a Linux x86-64 executable. The first element must not be empty |
description |
Free text shown by hairspring targets
|
cwd |
The working directory to start the program in |
env |
Extra environment variables for the program, as a table, for example env = { GOMAXPROCS = "2" }
|
trial_timeout_secs |
The longest one trial may run before it is killed and counted as timed out. If you leave it out, the backend default applies (60 seconds in the code we read). Hermit is slow, so raise it for heavier programs |
hang_is_violation |
false by default. Set it to true only when a hang or deadlock is the bug you are looking for. Then a timeout is a violation. Otherwise a timeout is inconclusive |
parallelism |
Trials run at once, 1 to 64 (default 1). Keep it at or below your core count |
pass_env |
true by default. Under --base-env=minimal or empty in hermit_args, Hairspring passes env, HAIRSPRING_SEED and its other variables to the program with --env=NAME (names only; Hermit copies the values). Set false to get the previous behaviour, where Hermit dropped them. Process targets only |
backend |
Optional. Which backend runs the target. "hermit" is the only value today, and it is what a process target uses when the key is left out, so existing configs mean what they did. Any other string is refused when the config is read: unknown `backend` for a process target; known: hermit. The message does not repeat the value you wrote. A value that is not a string gets the TOML parser's error instead. A sim or wasm target with a backend key is refused |
chaos, heuristic, sticky_permille, max_timeslice_ns, skid_margin
|
How threads are scheduled. See below |
hermit_args |
Extra flags passed to Hermit. Flags Hairspring sets itself (--chaos, --sched-seed, --epoch, --skid-margin, --summary-json, --tmp and a few more) are refused |
Scheduler options, all optional. The default is seeded random scheduling.
-
chaos = truerandomises thread priorities each timeslice. It cannot be combined withheuristic = "random". -
heuristicis"random"(the default whenchaosis off),"sticky-random"or"none"."none"needschaos = true.sticky_permille(0 to 1000) only goes with"sticky-random". -
max_timeslice_ns(at least 10): preempt after this much virtual time. Smaller values can reach finer races and cost more. -
skid_margin(1 to 10000): a performance-counter skid margin. If the counter lands a preemption late, Hermit aborts the run, and Hairspring treats that as an infrastructure error, not a verdict. See Troubleshooting.
Races inside a few instructions are the case for chaos with a short max_timeslice_ns. The setting used for the tested configuration above was chaos = true, max_timeslice_ns = 100000, skid_margin = 25. That is a starting point, not a recommendation for your program.
Scheduler options, hang_is_violation, parallelism, pass_env and backend are valid only for process targets; on a sim target they are refused.
GET /api/targets (and the MCP list_targets tool) give each target a backend, the id of the backend that runs it (sim, wasm or hermit), and capabilities, a set of true/false flags for replay, confirm, shrink, strict_verify, snapshot_branch, fault_injection and multi_process. A flag is true only if the backend really has the feature. They describe the backend, not the machine the server runs on. snapshot_branch means the feature is exposed to callers, so it is false for every backend today. capabilities is null when the server has no backend registered for the target; a worker that has one can still run it. The older fields are unchanged.
Hermit passes the host's files and environment through as inputs, and inputs change schedules. So Hairspring starts the program from a minimal pinned environment, not yours: PATH=/usr/local/bin:/usr/bin:/bin, LANG=C, LC_ALL=C, TZ=UTC, plus whatever you put in env. For Python programs it also freezes bytecode caches. It sets HAIRSPRING_SEED to the trial's seed, which you may read if your program derives its own randomness from it. Standard input is empty. If your program needs a variable, put it in env.
This holds under every --base-env that Hairspring can add to. With --base-env=minimal or empty, Hermit starts the program from almost nothing (under minimal it still adds its own HOSTNAME and HOME), so Hairspring passes each of these variables with --env=NAME. The values are not put on the command line. A variable you pass yourself with --env in hermit_args keeps your value. When Hairspring adds the variables (--base-env=minimal or empty, with pass_env on), names in env must be printable ASCII without =. With a Hermit --config, --from or --image, Hairspring cannot add them, and prints a warning on standard error once per target when the run starts.
[target.checkout.budget]
max_trials = 100 # trials per run (default 200; the engine caps it at 10,000)
wall_secs = 600 # wall-clock limit for the run (default 60; the engine caps it at 600)max_steps (default 200,000) is for sim targets and does not apply to a process target.
Each trial runs your program once under Hermit with a different scheduler seed. A run starts at --seed N (random if you leave it out; it is always printed) and trial i uses seed N + i. So --seed 1 with 100 trials covers seeds 1 to 100.
Mind the wall-clock limit. The default wall_secs is 60. Because Hermit runs are slower than native, 60 seconds may allow only a handful of trials, and the run will stop there. If a run seems to end early, raise wall_secs in the config or pass --budget-secs. The command-line options override the config for one run:
| Option | Effect |
|---|---|
--seed N |
the first seed |
--budget-secs S, --budget-steps N
|
override the wall-clock or step budget |
--sweep |
run the whole budget past every failure and store every trial; see below |
--trials N |
with --sweep: how many trials (at most 10,000) |
--seeds A..B |
with --sweep: seeds A up to but not including B |
--format text|json|junit|sarif |
the same result for tools |
--bundle-dir DIR |
where bundles are written (default .hairspring/bundles) |
For each trial, Hairspring looks at what the program did. The first rule that applies decides, in this order (from the guest protocol document):
- Hermit itself failed, or the setup was misconfigured: the run is an error and no trial is spent.
-
The trial timed out: inconclusive, or a violation if the target sets
hang_is_violation. - The host failed to collect the program's reports (for example scratch I/O): inconclusive, never a pass.
-
A property reported through the SDK failed (an
alwayswas false, or anunreachablewas hit): the trial fails. -
The program printed a line starting
HAIRSPRING_VIOLATION:, on stdout or stderr. Whatever follows the colon becomes the reason. -
Hermit reported a deadlock (a stderr line beginning
Deadlock detected:together with a non-zero exit): a deadlock finding, whether or nothang_is_violationis set. - A non-zero exit status.
- A clean exit is a pass, unless the SDK's reports are unusable, in which case the trial is inconclusive.
In practice you have three ways to say "this is a failure", from least to most work:
| How | What you do | What you get |
|---|---|---|
| Exit status | make your program exit non-zero when something is wrong (a self-test, assert, a panic) |
pass or fail per trial, nothing else |
| A violation line | print HAIRSPRING_VIOLATION: <reason>
|
the same, with a reason in the report |
| The guest SDK | declare named properties in the source and evaluate them | per-property counts across trials, witness values for the first failure, and coverage claims such as "this was never reached" |
A program does not need an SDK to be run. The SDK is optional.
An SDK lets the program declare properties in its own source and tell Hairspring how each one fared. There are five kinds: always (must hold every time it is evaluated), always_or_unreachable, sometimes (must hold at least once over the run), reachable (must be reached at least once over the run) and unreachable (must never be reached). Evaluating one only updates counters in memory. The program reports once, at exit, as lines beginning HAIRSPRING_EVENT followed by JSON, on standard error.
Two practical points:
- The SDK needs no configuration in
hairspring.toml. It writes to standard error, and Hairspring reads it. (TheHAIRSPRING_EVENTS_FILEandHAIRSPRING_EVENTS=offvariables described in the SDK documents exist, but the file transport is not switchable from the config in this release, so leave them alone.) - Every SDK reports through the same protocol, described in the guest protocol document. The SDKs for each language are on Guest-SDKs-by-Language.
You can see the lines your program writes by running it natively, on any machine, with no Hairspring involved. That is the quickest way to check that your instrumentation is wired up.
We will write a tiny Go program with a classic check-then-act race, run it natively, describe it to Hairspring, add the SDK, and read the results. Every step through "run it natively" and "build for Linux" was run on a Mac. The Hairspring runs themselves need Linux and Hermit, so for those steps we show only the shape of the output, taken from the source, and say so.
Make a directory and two files.
mkdir seats && cd seatsgo.mod:
module example.com/seats
go 1.24
main.go:
package main
import (
"fmt"
"os"
"runtime"
"sync"
"sync/atomic"
)
const buyers = 3
func main() {
var seatsLeft atomic.Int64
var sold atomic.Int64
seatsLeft.Store(1) // one seat exists
var wg sync.WaitGroup
for i := 0; i < buyers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if seatsLeft.Load() > 0 { // check ...
runtime.Gosched() // (a pause, as a network call would be)
seatsLeft.Add(-1) // ... then act
sold.Add(1)
}
}()
}
wg.Wait()
fmt.Printf("sold %d seat(s); 1 existed\n", sold.Load())
if sold.Load() > 1 {
fmt.Fprintln(os.Stderr, "oversold")
os.Exit(1)
}
}Three buyers race for one seat. Each checks that a seat is free, pauses briefly, then takes it. If two buyers both check before either takes, the seat is sold twice. The atomics keep every individual operation safe; the bug is in the logic, as in most real races. The runtime.Gosched() stands in for any real delay between checking and acting, and it makes the window easy for any scheduler to hit.
go run .On the Mac used for this guide, ten runs printed sold 3 seat(s); 1 existed nine times and sold 2 seat(s); 1 existed once, each followed by oversold and exit status 1. Yours will vary. Native scheduling is not repeatable, and that is the point: a native run is a fresh roll of the dice each time, and a failure cannot be replayed. Hairspring's job is to make it replayable.
Hermit runs Linux x86-64 programs. Build a static one (this works from a Mac):
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GOAMD64=v1 \
go build -trimpath -buildvcs=false -ldflags=-buildid= -o seats-linux .Those flags are the ones the repository's Go example records for its Linux binaries. file seats-linux reports ELF 64-bit LSB executable, x86-64, ... statically linked. Copy it to your Linux machine.
On the Linux machine, in a directory of your choice, write hairspring.toml:
[target.seats]
kind = "process"
command = ["/home/you/seats/seats-linux"]
trial_timeout_secs = 120
[target.seats.budget]
max_trials = 50
wall_secs = 300Check that the file parses and the target is listed (we ran this on the Mac; it needs no Hermit):
hairspring targets...
seats process
A mistake in the file gives a plain message instead, for example error: invalid config: target `seats` requires `command` .
hairspring doctor seatsFor a process target this runs each of seeds 1 to 8 (change it with --seeds) twice and compares the program's exit status and output. It reports seats: deterministic over 8 seeds or the first seed where two runs differed, and says if a trial timed out (a doctor trial timed out after 120s; raise trial_timeout_secs so both runs can finish). On a Mac, where there is no Hermit, the command exits 2 with error: backend error: hermit executable not found at hermit. Hermit runs only on x86-64 Linux: install it there, or set HAIRSPRING_HERMIT to its path (we ran this). The rest of this paragraph comes from the source, not from a run here. A program that prints its own timestamps or pids will differ between runs for reasons unrelated to scheduling, so fix that first.
hairspring test seats --seed 1The command prints what it is exploring, then one of three things. We could not run this step here, so these are shapes taken from the source, not captured output:
hairspring: exploring seats (seed 1)
FAILED seats (hermit)
property violated at sched-seed N (exit code 1); reproduced 3/3
steps (...):
1. run `/home/you/seats/seats-linux` under hermit with sched-seed N
2. exit code 1
...
bundle: ...
bundle written: .hairspring/bundles/seats-<8 characters>.hairspring (replay with `hairspring replay ...`)
hairspring: exploring seats (seed 1)
PASSED seats (... trials, ... ms)
hits: 0/N rated trials, exact one-sided 95% upper bound ...
hairspring: exploring seats (seed 1)
ERROR seats: <what went wrong>
The reason in parentheses is exit code 1 for the version without the SDK, or the name of the failed property for the version with it. A failing seed is re-run three times, and the report says how many of those reruns repeated the failure (the message ends reproduced 3/3 when all did). If they do not all repeat it, the message adds that the program or its environment is non-deterministic under Hermit. The step list is a short readable trace of the run and the tail of the program's output, not a list of scheduling decisions.
The command's exit status is 0 if every target passed, 1 if a property was violated, 2 on an error.
On a Mac you will get the error below, which is the honest answer for the platform:
hairspring: exploring seats (seed 1)
ERROR seats: hermit executable not found at `hermit`. Hermit runs only on x86-64 Linux: install it there, or set HAIRSPRING_HERMIT to its path
Add the module to the program's directory:
go get github.com/spiculedata/hairspring/sdks/go@main(That resolves to a pseudo-version of the repository's main branch; we ran it. The SDK is not published as a tagged module.) Then make four changes to main.go:
import hs "github.com/spiculedata/hairspring/sdks/go" // 1. import the SDK (alongside the other imports)
var noOversell = hs.Always("seats.no_oversell", "never sell more seats than exist") // 2. declare the property once
func main() { os.Exit(hs.Main(run)) } // 3. hs.Main runs `run` and sends the report when it returns
func run() int { // the old body of main, ending with:
// ...
fmt.Printf("sold %d seat(s); 1 existed\n", sold.Load())
noOversell.Check(sold.Load() <= 1, hs.Vals{sold.Load(), 1}) // 4. evaluate it, with two witness values
return 0
}Delete the old if sold.Load() > 1 { ... os.Exit(1) }. The program now exits 0 whatever happens, and the verdict comes from the property. Run it natively to see the report (we ran this, with the full program):
HAIRSPRING_EVENT {"v":1,"kind":"hello","sdk":"go", ... }
HAIRSPRING_EVENT {"v":1,"kind":"catalog", ... "id":"seats.no_oversell","type":"always","must_hit":true, ...}
sold 3 seat(s); 1 existed
HAIRSPRING_EVENT {"v":1,"kind":"assert", ... "id":"seats.no_oversell","hits":1,"trues":0,"falses":1,"first_false":{"thread":"go","ord":1,"vals":[3,1,0,0]} ...}
HAIRSPRING_EVENT {"v":1,"kind":"work", ... }
HAIRSPRING_EVENT {"v":1,"kind":"end","proc":"p1","reason":"exit","status":0, ... }
(Padding spaces inside the numbers are removed here for readability.) "falses":1 and "vals":[3,1,0,0] mean the property was evaluated once, was false, and the witness values were 3 sold and 1 existing. Rebuild for Linux as in step 3; hairspring.toml stays the same. Then hairspring test seats --seed 1 again. The report now has a properties table with one row per property, in the form VIOLATION always 'seats.no_oversell' ... with a rate and the first failing seed. A property that was never satisfied over the whole run is listed as UNMET under a separate heading.
The fix makes check and act one step: if seatsLeft.Add(-1) >= 0 { runtime.Gosched(); sold.Add(1) }. We built and ran that natively five times and it printed sold 1 seat(s); 1 existed every time. Build it as seats-fixed-linux, give it a second target ([target.seats-fixed], same keys, a different command), and run hairspring test seats-fixed --seed 1. A pass means those seeds did not fail, nothing more.
A normal run stops at its first confirmed failure. A sweep runs the whole budget past every failure and stores every trial, so the report can give a rate:
hairspring test seats --sweep --trials 1000 --seed 1 # seeds 1 to 1000
hairspring test seats --sweep --seeds 5000..5500 # seeds 5000 to 5499Here is a real sweep summary, from a built-in simulation target:
sweep: 20 trials: 0 pass, 20 violation, 0 inconclusive, 0 unusable, 0 infra
failing seeds: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 (first 10 of 20 violations)
hits: 20/20 rated trials (100.00%), Wilson 95% interval [83.88%, 100.00%]
- Timeouts (
inconclusive), trials whose SDK report was unusable (unusable) and host failures (infra) are listed apart. They are never in the denominator and never counted as passes. - The rate is
hitsover rated trials with a Wilson 95% interval. With no hits you get an upper bound instead. - The failing seeds listed are the lowest ones. Up to the first ten violations are each confirmed by re-running their seed; the row says how many reruns gave the same output (
same-output reruns k/n), or that the confirmation was skipped. - A sweep exits 1 if any trial was a violation, 130 if Ctrl-C stopped it, and also ends at
--budget-secs; if it ends before its trial budget the report says so. - A rate describes the setup it was measured in: your program, your Hermit build, your scheduler settings, your seed range. It is not a rate of anything else.
More in Sweeps-and-Confirmation.
With SQLite, the server uses one writer connection and four read-only connections, so a slow read (a heavy failures filter, say) no longer makes every other request wait behind it. Each read sees one committed snapshot that includes every write that had already returned. Because reads can overlap without a gap, the store keeps the write-ahead log (the -wal file beside the database) bounded: when it reaches 16 MiB it briefly stops lending read connections, waits up to 250 ms for reads in progress to finish and truncates it. During that pause reads and writes from the same process can wait up to a few hundred milliseconds. A read that keeps running past every attempt (a heavy filter looped by several clients) can still let the file grow until it ends, and during a long hairspring gc --apply the limit does not act until the apply releases the database. With PostgreSQL, connections are pooled by the server as before. We measured this on a Mac with generated stores, not on a long-lived server.
A server records the inputs of the runs it executes itself for built-in simulated targets. A remote worker of this release
reports the same for the job it holds, once per job, using its own secret, before it runs. If the server is an older
release, or a proxy in front of it does not pass the path, the server answers 404 or 405; the worker then stops sending
inputs for 10 minutes, says so once, and asks again afterwards. The run is not held up: it just has no recorded inputs.
A report must name every input its kind of target has and be for the run's own target; otherwise the server refuses it,
in fixed words that do not repeat what was sent. On PostgreSQL, the roles that run the server need SELECT, INSERT, DELETE on run_inputs (see Installation). Hermit and WASM runs record no inputs yet. We checked this with tests,
including a worker against a stand-in server and against a real one, not on a long-lived deployment.
Hairspring has no built-in TLS, so a server reachable from other machines usually sits behind a reverse proxy such as nginx or Caddy. By default the server counts every request under the address it connects from, which behind a proxy is the proxy's own address. Tell it which proxy to believe with hairspring serve --trusted-proxy 127.0.0.1 (or the proxy's address on your private network; several entries and networks such as 10.0.0.0/24 are allowed, comma-separated), or set HAIRSPRING_TRUSTED_PROXY. The server then reads the client address from X-Forwarded-For, but only on connections from a listed proxy, and prints the list it trusts when it starts. It reads the header from the right, stopping at the first address that is not a listed proxy; text further left is never decoded. It falls back to the connecting address when an entry it reads is not a plain address, when more than 16 entries or 1024 bytes are needed, or when every entry is a listed proxy. The Forwarded header is not read. Configure the proxy to overwrite any X-Forwarded-For the client sent: in nginx proxy_set_header X-Forwarded-For $remote_addr;, in Caddy header_up X-Forwarded-For {remote_host} inside reverse_proxy. Hairspring refuses to start with a network wider than /8 (IPv4) or /32 (IPv6), the unspecified address or any range starting at it, IPv4-mapped or IPv4-compatible IPv6 entries, or malformed entries, because trusting a range that broad would let a client choose the address it is counted under; a repeated entry is listed once. A client inside a listed network can still choose its counted address through a proxy that appends to the header. SECURITY.md in the repository has full examples and the remaining risks. We ran the server with the flag on a Mac and checked the lockout case there; we did not test the nginx and Caddy settings against a real proxy.
hairspring serve limits what one client can do. --max-connections N (default 512, 1 to 1000000) is how many connections may be open at once; a connection beyond it is answered 503 with Retry-After: 1 and closed (while 128 are being refused at once, a further one is closed without a reply). --max-replays N (default 2, 1 to 64) is how many replays may run at once, REST and MCP together; a replay beyond it is refused straight away, with 429 and Retry-After: 5 on REST and an error with the code replay_busy on MCP, and nothing is queued. --max-confirmations N (default 0, which is off; 0 to 4) is how many confirmations started from the web UI or POST /api/failures/{id}/confirm may run at once: each runs a failure's process target on the server for up to two minutes, a failure has at most one at a time (409), a request beyond N is refused with 429 and Retry-After: 30, and nothing is queued. The server closes a connection that has not sent a complete request head within 10 seconds, including one left idle between requests, and answers 408 if a request body has not arrived within 30 seconds; live event streams and long replays are not affected. One token (or, without tokens, one client address) may hold at most 16 of the 64 event-stream slots. The server speaks HTTP/1.1 only. Keep the file-descriptor limit (LimitNOFILE under systemd) well above --max-connections plus 128. A client that keeps sending requests on its kept-alive connections can hold them indefinitely; only per-client limits in front of the server prevent that. We checked these limits in the server tests and by running the real binary on a Mac, not under load on a long-lived server.
Stored runs accumulate. hairspring gc reports what could be removed and removes nothing unless you ask:
hairspring gc --older-than 30dThe report lists the runs (by state), failures, stored check results, trials, events, jobs, trial trees, stored blobs with their size, and the bundle files in .hairspring/bundles that it would delete. It also says what it is leaving alone and why. Nothing has changed after this command. Its last line gives the arguments that remove exactly what it showed: --apply --cutoff T --expect-runs N --expect-digest D. Run the command again with those arguments to remove those runs. If any of the three is missing, or the runs found are not the runs the report showed, nothing is removed and the exit status is 2. A run that stops qualifying while it is being removed (a job was queued for it, say) is kept, the output says why, and the exit status is 2.
Options: --older-than is required (a whole number and h, d or w, measured from when the run finished, by this computer's clock); --state chooses the states to remove (the default is passed, error and cancelled, and failed runs are kept unless you add failed, because they hold the evidence); --bundle-dir says where bundle files are; --json prints one JSON document. Queued and running runs, and runs with a queued or running job, are never removed, and a trial tree is removed whole or kept whole. It works the same on SQLite and PostgreSQL.
What it does not do: the database file does not shrink (SQLite reuses the space; run VACUUM to give it back), files outside the bundle directory are never touched, and a symbolic link is never followed. Run --apply when no worker is running jobs, because a failure found while it runs, whose bundle text is identical to one being removed, can be left without its stored bundle. The report takes no lock that a server or worker waits for, and its counts are an estimate as of the moment it read them; --apply checks everything again, in short batches, and each batch reads the whole trials table, so on a large store run it off-peak. Only one gc runs against a database at a time. Removal is permanent, and nothing records that a run existed. We ran the report and the pinned apply against scratch stores on SQLite and PostgreSQL, not against a long-lived store.
hairspring serve serves the web UI at / on the same address as the API (http://127.0.0.1:7878 by default). With no tokens configured the server accepts only loopback connections and the UI opens at once; it treats every caller as an administrator, so keep it on your own machine. To let other machines in you must configure tokens: the server refuses a non-loopback address without them.
-
Making tokens.
hairspring token --role read|run|admincreates one and prints it once. Or give the server a file with--tokens-file(onetoken:roleper line; theHAIRSPRING_TOKENSenvironment variable is the alternative). Tokens must be at least 16 characters. An entry may carry a label after the role, separated by exactly one space (token:run ci bot), so that the server's log names who did something. A label is 1 to 64 ASCII letters, digits, spaces,.,_or-, differs from every other label (ignoring case) and does not contain a token. Only whoever writes the tokens file can set one. The server's log lines for creating a run, starting a replay, registering a worker and changing a worker's trust show it aslabel (t:<key>), never the token, andGET /api/meshows it only to the holder of that token. A label is not a secret and not proof of who is calling: people who share a token share its label. Use tokens made byhairspring token, because the key in a log line is a prefix of a digest of the token. Actions taken through MCP over HTTP are not yet logged with the label. Never put a token on a command line you share, and keep the file readable only by you. -
Roles.
readcan look at runs, failures, trials, targets and workers.runcan also start and cancel runs and sweeps, replay bundles and act as a worker, and sees worker ids.admincan also change worker trust. Each role includes the ones before it. -
Signing in. With tokens configured, the UI shows a "Sign in" page and asks for an API token. It keeps the token for that browser tab only, so a new tab asks again. The top bar then shows your organisation's name if the server was started with
--organisation. -
What a
readtoken sees. Everything except worker ids; the Start run and Start sweep buttons are disabled, with a line saying your role cannot start runs. Ask whoever runs the server for aruntoken.

The Runs page with a read token, seen on a live server with simulation targets.
The pages are described, with pictures, in Your-First-Run (section 9). The failure page can start a confirmation only on a server started with --max-confirmations, for a run token; otherwise it shows the command to run. Against a simulation target the property table is empty and a failure stays a candidate: property tables and the coverage charts are filled by process targets that use a guest SDK, and checks are recorded by hairspring confirm, which needs Hermit on Linux x86-64. We have not captured those pages on a Mac.
Open the Workers page (/workers). Above the table it says how many jobs are waiting and how many are held by workers. Each row shows the worker's state, trust, capabilities, the runs it holds (as links) and when it was last seen. The state is Busy (it holds a job lease that has not ended), Idle (it holds none and the server heard from it within one lease length, two minutes by default) or Not seen recently. A run token or above also sees the worker's id; a read token does not, because the id is a credential for the worker endpoints, and the page says so. The page only reads. To change trust, an administrator calls POST /api/workers/{id}/trust with {"trusted": true}. The same data is available from GET /api/workers, which needs only the read role. A worker's capabilities are what it said about itself when it registered; the server does not check them, and the page shows them as text. Against a server older than this page, state and holding say "Not reported by this server", while trust and capabilities still show from what that server sends. We checked the page against a live server on a Mac with the server's own worker (Idle, Trusted) and a worker started by hairspring worker (Idle, Untrusted), and the API with a read, a run and an admin token. We did not see a Busy worker or a Not seen recently one on a live server, and we did not check the page at phone width with a read token on a live server.

The Workers page with a read token (no Id column), with one worker inside the server and one started with hairspring worker.
The search re-runs a failing seed three times. That is weaker than what it takes to trust a finding, so hairspring confirm runs explicit, recorded checks:
hairspring confirm <run id or failure id>
hairspring confirm .hairspring/bundles/seats-xxxxxxxx.hairspring # a bundle: printed, not stored
hairspring confirm <failure id> --control seats-fixed --strict-verify| Check | What it does |
|---|---|
same_seed_rerun |
the seed run again --reruns times (default 3, at most 10); passes when every rerun fails the same way (same property, same behaviour hash, same schedule fingerprint) |
bundle_replay |
the bundle replayed with the environment check; passes when the failure repeats |
fixed_control (--control TARGET) |
the same seed and flags on a second target that runs the fixed build; passes when the fixed build does not fail. It never says "the same interleaving": a different build is a different schedule |
strict_verify (--strict-verify) |
Hermit's own strict determinism verify (two more runs); either exit status of your program is accepted; it passes only if the runs matched and your program failed in the verify the way the recording did |
Each check ends as pass, fail, inconclusive or skipped. The finding moves from candidate to rerun_repeated to confirmed to verified, or ends as not_reproducible or control_failed. Exit status: 1 confirmed or verified (the failure repeats); 0 not reproducible on the environment it was recorded in; 2 error; 3 inconclusive (a check could not run, the environment differs, or the control failed; none of these shows the bug is fixed).
Also: --wall-secs (1 to 600, default 120) limits the whole confirmation, --json prints one JSON document, and --allow-environment-change goes on when the dynamic loader differs from the recording. Do not put credentials in a target's command: it is stored and served as written. Ctrl-C on hairspring confirm is handled: it stores the checks that finished, records the rest as skipped, kills the Hermit processes still running and exits 130 (Troubleshooting). This section is from the project documentation and hairspring confirm --help; we could not run confirm on a process target.
-
Commit the bundle file. It is small text.
hairspring replay <bundle>re-runs the recorded seed under Hermit and compares the host with the environment recorded in the bundle (kernel, CPU flags, loader files, Hermit executable hash, your program's hash). Exit 1 means the failure repeated. It is the evidence for the bug on the build that failed. - Write down the failing seeds from the sweep report next to it.
- Keep a target for the fixed build and run a sweep that includes those seeds.
Be clear about what replay can tell you after a fix. Rebuilding your program changes its hash, so the environment check reports a difference, and a replay that no longer fails exits 3: inconclusive, not evidence the bug is fixed. A different build is a different schedule, so the same seed on the fixed build is a different interleaving. What you can honestly say is "these seeds no longer fail on the fixed build". So run a range of seeds, including the old failing ones, in CI:
hairspring test seats-fixed --sweep --seeds 1..200Treat a clean result as a bound on the rate, not a proof (Limits-and-Honesty). A package upgrade that changes the dynamic loader can stop an old bundle replaying; replay refuses and says so, and --allow-environment-change overrides it.
- It builds as a static Linux x86-64 executable and runs from an empty environment with stdin closed.
- Its result does not depend on the clock, the pid or the host's files (or you accept that
doctorwill complain). -
hairspring targetslists it andhairspring doctor <target>says deterministic. - A short
hairspring testruns; raisewall_secsandtrial_timeout_secsif it ends early. - Add SDK properties if you want counts and witnesses; run a sweep for a rate; confirm any failure; keep the bundle and seeds; sweep the fixed build.
- Machine, build and service questions: Is-My-Machine-Suitable, Build-a-Linux-Executable, Testing-a-Service; to test your own logic on a Mac without Hermit, Test-Logic-with-Wasm.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference