Skip to content

Operate a Server

Tom Barber edited this page Nov 6, 2026 · 3 revisions

Operate a server

This page is for the person who runs hairspring serve for other people. It is a short version of the repository's Operate a Hairspring server guide, which has the full detail and is the authority if the two differ. For the threat model and the limits, read SECURITY.md.

Hairspring is experimental. Commands below marked "not run here" were written from the code and the documentation and have not been run by the author. Options marked "newer than v0.1-RC1" are not in the pre-release binary; see Upgrading-and-Versions.

1. Pick a machine and install

A server that only runs the built-in or wasm targets runs on macOS or Linux and needs nothing special. Runs of process targets need Linux on x86-64 with hardware performance counters and Hermit, and only one bare-metal host has been measured (see Limits-and-Honesty and the repository's machine suitability guide). The server and the machines that run process targets can be different machines, with one exception: a replay of a process-target bundle runs on the server itself with the server's hairspring.toml, so for those replays the server host must also qualify (Linux x86-64, Hermit, HAIRSPRING_HERMIT set). Install hairspring on every host as in Installation.

2. Create tokens

$ hairspring token --role admin
New API token (shown once, store it safely):
  <64 hexadecimal characters>
Give it to the server with:  HAIRSPRING_TOKENS=<token>:admin
Give it to a worker with:    HAIRSPRING_TOKEN=<token>

Roles are read (look at results), run (start runs, replay, and act as a worker) and admin (also change worker trust). The command only prints the token; the server keeps a SHA-256 digest of each token it is given at start. A token must be at least 16 characters.

The server and a worker read tokens in different formats:

Who Where Format
The server --tokens-file FILE (mode 600) One token:role per line, optionally token:role label
The server HAIRSPRING_TOKENS The same entries, comma-separated: token1:admin alice,token2:run ci bot
A worker HAIRSPRING_TOKEN, or --token-file FILE One bare token and nothing else

Create the files owned by the service user and readable by nobody else, for example (as root; checked in a Ubuntu 24.04 container):

useradd --system --home-dir /var/lib/hairspring --shell /usr/sbin/nologin hairspring
install -d -m 755 /etc/hairspring   # on a worker host, run the useradd line there first too
install -m 600 -o hairspring -g hairspring /dev/null /etc/hairspring/tokens
printf '%s:admin\n' "$ADMIN" > /etc/hairspring/tokens

A server tokens file readable by other users starts with a warning (run chmod 600). Giving a worker the server's file is refused with error: /etc/hairspring/tokens: the worker token file must contain one bare token ... (the path you gave). A worker with no token or a read token stops with error: the server rejected this worker's token.

3. Start the server

hairspring serve --addr 0.0.0.0:7878 --tokens-file /etc/hairspring/tokens --workers 0

A non-loopback address without tokens is refused: error: cannot start the server: refusing to listen on a non-loopback address without API tokens; configure at least one token. --workers 0 means only remote workers run queued jobs; it does not stop the server running target code, because a replay (POST /api/replay or the MCP replay tool, run role) runs the target on the server with the server's hairspring.toml. There is no built-in TLS: put a TLS-terminating reverse proxy in front.

Flag What it does
--max-confirmations N How many confirmations a run token may start from the web UI (or POST /api/failures/{id}/confirm) at once, from 0 to 4. The default is 0, which turns it off. Each one reruns and replays a failure's process target on this server for up to two minutes, and the checks it stores are trusted like the server's own workers'. A request beyond the limit is refused with 429 and Retry-After: 30, not queued, and a failure has at most one at a time. Turn it on only where every run token holder may change where a failure stands. Ctrl-C or SIGTERM on the server ends a running confirmation between checks. The run in progress finishes: a matched rerun is stored inconclusive (stopped: cancelled), a replay that reproduced is stored pass, a run that did not match or reproduce (for example because the stop also killed the Hermit process) is stored skipped, as hairspring confirm stores it on Ctrl-C, and checks that did not run are skipped. A confirmation with a skipped check does not change the finding. Newer than v0.1-RC2

The nginx and Caddy settings, and --trusted-proxy (newer than v0.1-RC1), are in the repository's SECURITY.md.

4. Remote workers

hairspring worker --server https://hairspring.example.com --token-file /etc/hairspring/worker-token --name host-a

Use https:// through your TLS proxy. With a plain http:// address that is not loopback the worker prints hairspring-worker: warning: the API token is sent over plain http to a non-local server; use https or a private network (from the code; not seen here).

A worker that has registered prints hairspring: worker `host-a` serving http://.... A worker for process targets needs, on its own host (the Linux parts are not run here): its own hairspring.toml passed with --config, defining the same target names; the program at the same absolute path; Hermit with HAIRSPRING_HERMIT set; and the same loader and kernel behaviour if it will replay bundles.

A remote worker starts untrusted: its results are labelled as unverified. Trusting a worker changes only how its results are labelled and whether they can back a claim; it sends the worker no code. An admin marks it trusted through the API (there is no command for it). Find its id with a run or admin token (read tokens are not shown ids), then:

curl -s -X POST -H "Authorization: Bearer $ADMIN" -H 'content-type: application/json' \
  -d '{"trusted":true}' http://127.0.0.1:7878/api/workers/<worker id>/trust

which answers {"ok":true} (run against a local server).

5. PostgreSQL, systemd, backups, retention

  • PostgreSQL: set HAIRSPRING_DATABASE_URL in the server's environment. serve --help does not list it. Not run here against a live PostgreSQL; a URL that cannot be reached gives error: cannot use the PostgreSQL database: storage error: error connecting to server. There is no TLS on that connection: keep it on a private network.
  • systemd: the repository guide has a server unit (with sandboxing options) and a worker unit (deliberately without them, because Hermit may need what they restrict: untested) that systemd-analyze verify accepted; starting them was not run here.
  • Logs: startup lines go to standard output, warnings and errors to standard error (the journal, under systemd).
  • Backups: (umask 077; sqlite3 .hairspring/hairspring.db ".backup 'hairspring-backup.db'") (the sqlite3 command was run, and the copy read back), or pg_dump -h db.internal -U hairspring hairspring > hairspring-backup.sql with the password in ~/.pgpass (mode 600) or a PGPASSFILE, not on the command line (not run here).
  • Retention: hairspring gc --older-than 30d (newer than v0.1-RC1) reports what it would remove and removes nothing until you repeat it with the arguments its last line prints.

6. Root or not, for process workers

The documentation says two things. The quickstart says the measurements ran Hermit as root and unprivileged use was not tested. SECURITY.md says to run the server and workers as unprivileged users. For a worker that runs process targets the unprivileged case is untested, and the choice is the machine owner's. Know the consequence: a worker run as root runs every command in its hairspring.toml as root whenever anyone holding a run token queues a run of that target. The server itself also runs target code (its own worker threads, and replays), so run it unprivileged in every case.

7. Docker Compose

docker compose up --build -d (after cp .env.example .env and replacing every value) starts PostgreSQL, a server and two workers (not run here). The image has no Hermit and no hairspring.toml, so out of the box it runs only the built-in simulation targets. Compose workers are untrusted until an admin trusts them.

Before you expose a server

Read the security checklist. For upgrades and uninstalling see Upgrading-and-Versions; for problems see Troubleshooting.

Clone this wiki locally