Repository navigation
Operate a Server
This page is for the person who runs hairspring serve for other people. It is a short version of the repository's
Operate a Hairspring server guide, which has the
full detail and is the authority if the two differ. For the threat model and the limits, read
SECURITY.md.
Hairspring is experimental. Commands below marked "not run here" were written from the code and the documentation and have not been run by the author. Options marked "newer than v0.1-RC1" are not in the pre-release binary; see Upgrading-and-Versions.
A server that only runs the built-in or wasm targets runs on macOS or Linux and needs nothing special. Runs of process
targets need Linux on x86-64 with hardware performance counters and Hermit, and only one bare-metal host has been measured
(see Limits-and-Honesty and the repository's
machine suitability guide). The
server and the machines that run process targets can be different machines, with one exception: a replay of a process-target
bundle runs on the server itself with the server's hairspring.toml, so for those replays the server host must also qualify
(Linux x86-64, Hermit, HAIRSPRING_HERMIT set). Install hairspring on every host as in
Installation.
$ hairspring token --role admin
New API token (shown once, store it safely):
<64 hexadecimal characters>
Give it to the server with: HAIRSPRING_TOKENS=<token>:admin
Give it to a worker with: HAIRSPRING_TOKEN=<token>
Roles are read (look at results), run (start runs, replay, and act as a worker) and admin (also change worker trust).
The command only prints the token; the server keeps a SHA-256 digest of each token it is given at start. A token must be at
least 16 characters.
The server and a worker read tokens in different formats:
| Who | Where | Format |
|---|---|---|
| The server |
--tokens-file FILE (mode 600) |
One token:role per line, optionally token:role label
|
| The server | HAIRSPRING_TOKENS |
The same entries, comma-separated: token1:admin alice,token2:run ci bot
|
| A worker |
HAIRSPRING_TOKEN, or --token-file FILE
|
One bare token and nothing else |
Create the files owned by the service user and readable by nobody else, for example (as root; checked in a Ubuntu 24.04 container):
useradd --system --home-dir /var/lib/hairspring --shell /usr/sbin/nologin hairspring
install -d -m 755 /etc/hairspring # on a worker host, run the useradd line there first too
install -m 600 -o hairspring -g hairspring /dev/null /etc/hairspring/tokens
printf '%s:admin\n' "$ADMIN" > /etc/hairspring/tokensA server tokens file readable by other users starts with a warning (run chmod 600). Giving a worker the server's file is
refused with error: /etc/hairspring/tokens: the worker token file must contain one bare token ... (the path you gave). A worker with no token or a read token stops with
error: the server rejected this worker's token.
hairspring serve --addr 0.0.0.0:7878 --tokens-file /etc/hairspring/tokens --workers 0A non-loopback address without tokens is refused:
error: cannot start the server: refusing to listen on a non-loopback address without API tokens; configure at least one token.
--workers 0 means only remote workers run queued jobs; it does not stop the server running target code, because a replay
(POST /api/replay or the MCP replay tool, run role) runs the target on the server with the server's hairspring.toml.
There is no built-in TLS: put a TLS-terminating reverse proxy in front.
| Flag | What it does |
|---|---|
--max-confirmations N |
How many confirmations a run token may start from the web UI (or POST /api/failures/{id}/confirm) at once, from 0 to 4. The default is 0, which turns it off. Each one reruns and replays a failure's process target on this server for up to two minutes, and the checks it stores are trusted like the server's own workers'. A request beyond the limit is refused with 429 and Retry-After: 30, not queued, and a failure has at most one at a time. Turn it on only where every run token holder may change where a failure stands. Ctrl-C or SIGTERM on the server ends a running confirmation between checks. The run in progress finishes: a matched rerun is stored inconclusive (stopped: cancelled), a replay that reproduced is stored pass, a run that did not match or reproduce (for example because the stop also killed the Hermit process) is stored skipped, as hairspring confirm stores it on Ctrl-C, and checks that did not run are skipped. A confirmation with a skipped check does not change the finding. Newer than v0.1-RC2 |
The nginx and Caddy settings, and --trusted-proxy (newer than v0.1-RC1), are in the repository's
SECURITY.md.
hairspring worker --server https://hairspring.example.com --token-file /etc/hairspring/worker-token --name host-aUse https:// through your TLS proxy. With a plain http:// address that is not loopback the worker prints
hairspring-worker: warning: the API token is sent over plain http to a non-local server; use https or a private network
(from the code; not seen here).
A worker that has registered prints hairspring: worker `host-a` serving http://.... A worker for process targets
needs, on its own host (the Linux parts are not run here): its own hairspring.toml passed with --config, defining the
same target names; the program at the same absolute path; Hermit with HAIRSPRING_HERMIT set; and the same loader and
kernel behaviour if it will replay bundles.
A remote worker starts untrusted: its results are labelled as unverified. Trusting a worker changes only how its results
are labelled and whether they can back a claim; it sends the worker no code. An admin marks it trusted through the API (there
is no command for it). Find its id with a run or admin token (read tokens are not shown ids), then:
curl -s -X POST -H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
-d '{"trusted":true}' http://127.0.0.1:7878/api/workers/<worker id>/trustwhich answers {"ok":true} (run against a local server).
-
PostgreSQL: set
HAIRSPRING_DATABASE_URLin the server's environment.serve --helpdoes not list it. Not run here against a live PostgreSQL; a URL that cannot be reached giveserror: cannot use the PostgreSQL database: storage error: error connecting to server. There is no TLS on that connection: keep it on a private network. -
systemd: the repository guide has a server unit (with sandboxing options) and a worker unit (deliberately without
them, because Hermit may need what they restrict: untested) that
systemd-analyze verifyaccepted; starting them was not run here. - Logs: startup lines go to standard output, warnings and errors to standard error (the journal, under systemd).
-
Backups:
(umask 077; sqlite3 .hairspring/hairspring.db ".backup 'hairspring-backup.db'")(the sqlite3 command was run, and the copy read back), orpg_dump -h db.internal -U hairspring hairspring > hairspring-backup.sqlwith the password in~/.pgpass(mode 600) or aPGPASSFILE, not on the command line (not run here). -
Retention:
hairspring gc --older-than 30d(newer than v0.1-RC1) reports what it would remove and removes nothing until you repeat it with the arguments its last line prints.
The documentation says two things. The quickstart says the measurements ran Hermit as root and unprivileged use was not
tested. SECURITY.md says to run the server and workers as unprivileged users. For a worker that runs process targets the
unprivileged case is untested, and the choice is the machine owner's. Know the consequence: a worker run as root runs every
command in its hairspring.toml as root whenever anyone holding a run token queues a run of that target. The server itself
also runs target code (its own worker threads, and replays), so run it unprivileged in every case.
docker compose up --build -d (after cp .env.example .env and replacing every value) starts PostgreSQL, a server and two
workers (not run here). The image has no Hermit and no hairspring.toml, so out of the box it runs only the built-in
simulation targets. Compose workers are untrusted until an admin trusts them.
Read the security checklist. For upgrades and uninstalling see Upgrading-and-Versions; for problems see Troubleshooting.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference