Skip to content

Troubleshooting

Tom Barber edited this page Nov 6, 2026 · 56 revisions

Troubleshooting

Real messages, what they mean, and what to try. Every message here was either produced by us (marked ran) or traced to the repository's source or documents (marked source). Where we are inferring something, we say so. If you hit a message that is not here, search the repository for its text; most are plain strings in the code.

Exit statuses first, because they are the same everywhere:

Status Meaning
0 OK: everything passed (or, for replay and confirm, the failure did not repeat on the environment it was recorded in)
1 a property was violated (or, for replay and confirm, the failure repeated)
2 an error
3 inconclusive (for replay and confirm: not repeated on a changed environment, or a check could not run)
4 the machine cannot run the process backend (doctor --process-backend)

Inconclusive is never a pass, and never evidence that a bug is fixed.

Installing

Message Meaning and what to do
zsh: permission denied: scripts/install.sh (ran) In the checkout we used, the scripts are stored without the executable bit. Run bash scripts/install.sh ..., or chmod +x scripts/*.sh
error: --version must be a release tag such as v0.1-RC1 (ran) You gave --version 0.1. Tags start with v. The current pre-release is v0.1-RC2
hairspring --version prints hairspring 0.1.0 (unknown) (source) The binary was built without git information: from a source archive, or a Docker image built without the HAIRSPRING_BUILD_COMMIT and HAIRSPRING_BUILD_DATE build arguments. It runs normally; quote the line as it is in a bug report
error: no release binary for <os> <arch> (available: Linux x86-64, macOS arm64). Build from source instead: scripts/install.sh --from-source (source) There is no release asset for your platform
error: sha256 mismatch for <asset>: downloaded A, SHA256SUMS says B. Nothing was installed. (source) The download is corrupt or truncated, or the checksum file and the binary disagree. Download again. Nothing was installed
error: sha256 of <asset> is A, not the --sha256 you gave (B). Nothing was installed. (source) The pinned hash you supplied from another source does not match. Treat this seriously
error: cargo is required to build from source (https://rustup.rs), or pass --version TAG to download a release (source) Install Rust, or download a release
error: Hermit runs only on Linux; this is Darwin. / result: CANNOT RUN Hermit on this machine (ran, from install-hermit.sh check on a Mac) Expected on macOS. See Installation

hairspring doctor --process-backend

This command (exit 0 or 4) prints one row per prerequisite and, if any is FAIL, one reason per blocker. See Installation for the full report. The reasons, from the source, with what to do:

Reason What it means
the process backend needs Linux: Hermit does not run on this operating system. The sim and wasm backends work here. (ran) You are on macOS or another non-Linux system
the process backend needs Linux on x86-64; this machine is <os> <arch>. The sim and wasm backends work here. (source) Linux on a different CPU, such as aarch64
Hermit was not found: HAIRSPRING_HERMIT is unset and hermit is not on PATH. Build it with scripts/install-hermit.sh and point HAIRSPRING_HERMIT at the result. (ran) Install Hermit and export HAIRSPRING_HERMIT=/path/to/hermit
... A copy exists at <path>; use it with: export HAIRSPRING_HERMIT=<path> (source) The installer's default location has a Hermit; export that path
Hermit at <path> did not answer --version: the file is missing, not executable, or broken. Rebuild it with scripts/install-hermit.sh. (source) HAIRSPRING_HERMIT points at something that does not run
the kernel refused a hardware performance counter (perf_event_paranoid is N; a container profile or security module can also do this). Hairspring never changes kernel settings: if this is your machine, lowering the setting is your decision (source) Your user may not open a hardware counter. Run as root, or change the setting if your policy allows. Hairspring does not
this machine exposes no hardware performance counters (PMU): common in virtual machines and containers. Hermit needs them; use bare metal or a VM that passes the PMU through. (source) The counters are hidden. The project tested only bare metal
opening a hardware performance counter failed with errno N (source) Look up the errno; the doctor shows the number
no hardware performance counter was opened on this machine (the probe did not run, which happens off Linux x86-64), so the process backend cannot be said to be runnable. (ran) The probe did not run at all, as on a Mac
kernel.yama.ptrace_scope is N, which stops Hermit from tracing the program ... Hairspring never changes this setting. (source) Scope 2 needs root; 3 disables tracing until reboot
running inside docker: counters and ptrace are often blocked here (source, a container row) A hint, not a blocker by itself
kernel ... older than 6.9 as a note (source) Hermit's tracing is slower; schedules are unaffected, timeouts may be

A passing doctor says the prerequisites are met. It does not run a program under Hermit.

Privileges. The project's measurements ran Hermit as root. In one write-up, ten probe runs as an unprivileged user stopped at the user-namespace uid mapping. A summary-file directory owned by the unprivileged user at mode 775 made a run as root stop with a permission-denied panic, while mode 1777 or a root-owned directory did not. If Hermit fails to start for an unprivileged user, try root, and read what you are running first.

Running

Message Meaning and what to do
ERROR <target>: hermit executable not found at hermit. Hermit runs only on x86-64 Linux: install it there, or set HAIRSPRING_HERMIT to its path (ran, on a Mac) hermit is not on PATH and HAIRSPRING_HERMIT is unset. Exit status 2. On a Mac there is no fix: use Linux
error: unknown target X`` (ran) The name is not in hairspring.toml or the built-in list. Check hairspring targets
error: config parse error: TOML parse error at line N ... unknown field ..., expected one of ... (ran) A misspelt or unsupported key. The message lists the valid keys
error: invalid config: target `X` requires `command` (ran) A process target needs a non-empty command array
error: invalid config: target `X`: `hermit_args` must not contain `--chaos`: Hairspring sets it from the scheduler options and the run's seed (ran) Remove that flag from hermit_args; use the scheduler keys instead
error: invalid config: ... chaoscannot be combined with therandom heuristic (ran) Drop heuristic = "random", or drop chaos
... `skid_margin` must be 1-10000 (ran) 0 or too large
error: invalid config: target `X`: `env` must not contain `HAIRSPRING_PROC`: Hairspring sets it for the guest (ran) Hairspring sets HAIRSPRING_PROC, HAIRSPRING_PROC_PID and HAIRSPRING_EVENTS_FILE itself. Remove the key from env
error: invalid config: target `X`: `hermit_args` must not pass `HAIRSPRING_PROC` with `--env`: Hairspring sets it for the guest (ran) The same three names are refused as --env in hermit_args. Remove it
error: invalid config: target `X`: an `env` key cannot be passed to the guest by Hermit (names must be printable ASCII without '='); rename it or set pass_env = false (ran) Under --base-env=minimal or empty, an env key is empty, contains = or has a character outside printable ASCII. Rename it
... scheduler options (heuristic, chaos, ...), hang_is_violation, parallelism and pass_env are only valid for kind = "process" (ran) You put process-only keys on a sim target
error: no stored run run_...`` / error: ...` is not a readable bundle file or a stored bundle hash` (ran) Wrong id or path, or a different --db / working directory. Results live in .hairspring/hairspring.db in the directory you ran in
error: no backend available for target kind confirm is not supported by the sim backend `` (ran) confirm is for process targets only: a simulation (sim) target cannot be confirmed. The stray backticks are in the message
error: cannot start the server: refusing to listen on a non-loopback address without API tokens; configure at least one token (ran) hairspring serve --addr something other than loopback needs tokens (SECURITY.md)
entry N: the label holds a token; labels are not secret, so they must never hold one A token label contains one of the configured tokens (any letter case). Change the label. N counts entries, not file lines: blank and comment lines are not counted, and a line with commas counts as several. Checked in tests, not on a live server
entry N: the label repeats an earlier label (labels must differ, ignoring case) Two entries have the same label, ignoring letter case. Give each token its own label. N counts entries, not file lines. Checked in tests, not on a live server
entry N: a label may use only ASCII letters, digits, space, ., _and-`` The label has another character (a comma, a colon, an accented letter, a tab). A label is also limited to 64 characters, with no leading, trailing or doubled spaces, and exactly one space separates it from the role. N counts entries, not file lines. Checked in tests, not on a live server
A worker, or a client call to register one, fails with 429 Too Many Requests and "worker registration limit reached for this token" The server limits how many workers one token can register: by default 100 registration attempts per token in a fixed one-hour window, and attempts that later fail (for example "caps is too large") still count. Wait for the Retry-After period (always 3600 seconds, the full window, not the time left) and try again, or ask whoever runs the server to raise --register-limit (0 removes the limit). The count is kept in server memory, so restarting the server resets it. The limit slows one token filling the server's worker table, which holds at most 1000 workers and prunes a worker only after 24 hours; to keep a single token from filling it, set --register-limit to 41 or fewer. Checked in tests, not on a live server
A request with a valid token gets 429 and "rate limit exceeded" Either the token's own allowance for the minute is used up, or the allowance for its client address is used up by other traffic, such as requests with no token or a wrong one from callers sharing a NAT, a reverse proxy or an IPv6 /64. Wait and retry (the Retry-After header says 60 seconds). While that traffic continues, valid tokens from the address stay refused, and for up to a minute after it stops. Operators: SECURITY.md in the repository describes this trade-off and the option of putting a rate limiter in front of the server. We checked this in the server tests, not on a live server

Timeouts

Message Meaning and what to do
timed out after Ns (possible hang or deadlock) (source) A trial ran past trial_timeout_secs and was killed. By default that is inconclusive: not a pass, not a failure, and not counted in the rate's denominator. It is a violation only if the target sets hang_is_violation = true
a doctor trial timed out after Ns; raise trial_timeout_secs so both runs can finish (source) hairspring doctor on a process target compares two runs; two kills at deadlines are not comparable, so it stops
a run ends after only a few trials (source) wall_secs defaults to 60 and covers the whole run. Hermit is slower than native, so raise wall_secs (the engine caps it at 600) or pass --budget-secs. When the wall budget, not the target's own limit, cuts a trial short, that trial is inconclusive
note: ended after <n> trials, before its trial budget: it reached its step or wall-clock limit (raise --budget-steps or --budget-secs to run more) (source) A sweep ended early; the note says why as far as it can tell

Hermit runs are about an order of magnitude slower than native, by an amount that depends on the program and the kernel. Set trial_timeout_secs accordingly. A timed-out trial's output decides nothing: a failure it shows is not a finding and is reported as "seen in an inconclusive trial".

When a trial times out or ends, Hairspring kills the guest's whole process group, so processes your program started do not outlive the trial.

Skid overshoot

With skid_margin set, Hairspring passes --panic-on-rbc-overshoot to Hermit. If the hardware counter lets a preemption land later than intended (an overshoot), Hermit panics. The message Hairspring reports is:

hermit panicked on a PMU skid overshoot: the preemption landed late, so this schedule is not reproducible; raise `skid_margin`

What it means, neutrally: that run is infrastructure, not a verdict. It is not a pass and not a failure, and the schedule it ran is not one that could be replayed. In a search, a Hermit failure ends the run as an error. In a sweep it is stored as an infra row and is outside the denominator. A confirmation retries it once at the same flags, then once at double the margin (up to Hermit's limit); a run at a higher margin is a different schedule, so it cannot count toward the reruns (the check is then inconclusive with a caveat, skid_margin_doubled: true) but it can count for the control. Raising skid_margin is the suggested response; the config accepts 1 to 10000. A larger margin is a different schedule and may be slower: one of the project's logs noted that a margin of 5000 made runs exceed 900 seconds. In its experiments the effective default was 100 and 25 was the fastest with no overshoot. That is one machine's finding, not a rule.

Failures that will not repeat

Message Meaning
property violated at sched-seed N (...); reproduced R/3 - not reproducible: the program or its environment is non-deterministic under hermit (source) A failing seed did not fail the same way on all reruns. Something in the program or its surroundings is a hidden input: a timestamp, a pid, a file on the host, a cache. Run hairspring doctor <target> and read "hidden inputs" in docs/determinism.md
hermit rejected its command line (usage error): ... (source) A Hermit flag problem, reported as infrastructure, not as the guest failing
hermit failed to run the program: ... (source) Hermit reported an internal failure. Often the program could not be started inside Hermit
incomplete capture: an output stream was still open Ns after the guest ended (something it started kept it), so output and events may be missing (source) A child of your program kept the output pipe open
the guest's events could not be collected: ... (source) A host-side problem collecting the SDK report; the trial is inconclusive
the verify run timed out after N s, so it says nothing (source) The strict verify (confirm --strict-verify) was stopped at its time limit, which is twice a single run's time, or what is left of wall_secs when that is less (never under 1 s). The check is inconclusive. Raise trial_timeout_secs or wall_secs
Hermit could not run the verify: <reason> (source) The strict verify is inconclusive: Hermit did not compare two runs. The reason is Hermit's own, written as kind(field=value, ..) with null fields left out, for example first_run_rejected(exit_code=0, stderr_bytes=3003, stdout_bytes=128), run_failed(detail=.., failure=.., run=run1) or skid_overshoot(count=2). It ends with Hermit said: and the first line of Hermit's standard error that contains Error:, when there is one. Hermit's --verify-allow is the exit status your program is required to have; the strict verify accepts either, so a program that exits 0 can be verified. An older build passed failure and could not verify a program that exits 0. For a skid overshoot, see "Skid overshoot" above
the two runs differed, but a host file the guest opened changed during run N: <path>, ... (source) The strict verify is inconclusive: Hermit compared the runs, they differed, and a file on the host that your program opened changed while it ran, so the difference is not counted against the program. Stop whatever is rewriting that file and run again
a strict verify is fail with mismatch: the guest did not fail in the verify (source) The two runs matched, but your program did not fail in them the way the recording did (it passed, or failed differently). A failure that depends on how many runs came before it will not verify. A recorded deadlock, or a target whose bug is a hang, shows as an inconclusive verify (Hermit records the verify as no_result: container_failed on a backend whose scheduler runs in a container child, such as ptrace, or run_failed with deadlock on DBT; a hang always reaches the time limit); neither can be verified. A failed or inconclusive verify never moves a finding below where the other checks put it. On an unchanged environment this is a fail; on a changed one it is inconclusive
Inherited descriptor cleanup failed ... close_range (source, quickstart) Your environment (an emulated or restricted container) does not implement close_range. Run on a native Linux host

My env setting or HAIRSPRING_SEED is missing in the program

With an older Hairspring and --base-env=minimal or empty in hermit_args, Hermit dropped the target's env, HAIRSPRING_SEED and the events-file path, so SDK events went to standard error instead of the events file. Upgrade, or pass your own env entries yourself with --env=NAME=VALUE in hermit_args (values must be ASCII and appear on the command line; the seed changes every trial and the events-file path is set by Hairspring, so those two cannot be passed this way). With a Hermit --config or --from file, list the variables in the file's env; Hairspring warns about these targets on standard error, once per target. If a replay says env.guest.passed (recorded none, now hairspring-1), the bundle was recorded before the change. Set pass_env = false on the target to replay it as recorded.

Replay and environment

Message Meaning
warning: environment differs: <field> (recorded X, now Y), ... (source) The host changed since the bundle was recorded. The replay still runs. If it does not repeat the failure, the trace says the environment differs, and the exit status is 3: not evidence the bug is fixed
error: environment mismatch: <details> (exit 2) (source) A host.loader.* hash changed (for example a package upgrade regenerated /etc/ld.so.cache), so the schedule is unreliable and the replay is refused. hairspring replay --allow-environment-change replays anyway and still prints the differences
warning: command line or working directory differs from the recording (<fields>): ... (source) The program's command line (target.command.sha256) or working directory (target.cwd.sha256) is not the recorded one. The replay still runs from the current configuration. If it misses, the trace names this as a possible cause and the exit status is 3
warning: target.command.sha256 and target.cwd.sha256 were not recorded: ... (source) A bundle recorded before these hashes existed. It replays as before; a changed command line or directory would not be noticed
warning: environment differs: this bundle records environment fields this build does not know (...) (source) The bundle may have been written by a newer Hairspring. The replay still runs. If it repeats the failure, that stands (exit 1). If it does not, the exit status is 3, not 0. Upgrade Hairspring and replay again. Builds from before this check do not warn
warning: no environment recorded in this bundle (source) An older bundle. It replays as before, with nothing to compare
warning: hermit.path.sha256 was not recorded: a different Hermit build would not be noticed (source) The bundle has no Hermit hash, so a different Hermit build would go unnoticed

Hermit's replay depends on the machine: its CPU, kernel and dynamic loader. Moving to another machine means recording again.

A Java bundle does not replay

A replay that does not reproduce the failure can be a path problem, because a JVM reads its own paths. A bundle records hashes of the command line and working directory, and replay warns command line or working directory differs from the recording when either changed. A bundle recorded before those hashes existed cannot warn, so the same check applies by hand.

  • Check that the JDK install, the jar and the working directory are at the same absolute paths as when the bundle was recorded.
  • In CI, use a fixed workspace path: a fixed checkout directory, a container with the JDK at a fixed path, or a path without per-job names.
  • If the paths must change, record again.

SQLite busy

Hairspring keeps results in a SQLite file (.hairspring/hairspring.db), in WAL mode. SQLite allows one writer at a time. A write takes the write lock when its transaction begins and waits up to 5 seconds for another writer to finish. If the file is still locked after that, the error reads storage error: database is locked (SQLite's SQLITE_BUSY) and the command should be retried. The store waits for a busy writer for a fixed timeout before it gives up, so you should see this only when several hairspring processes write to the same --db at once. Run them one after another, or give each its own --db.

Ctrl-C and stray processes

Ctrl-C on hairspring test (also with --confirm) and hairspring confirm is handled. The first Ctrl-C cancels the run in progress, kills the processes still in the process group of every trial that is running, as a timeout does (a child that calls setsid leaves the group and is not killed), starts no further trial, prints a line beginning hairspring: interrupted, and exits with status 130. It exits 130 also when Ctrl-C lands just after the last run finished on its own; that run stands as stored. A command runs only the job of the run it started, and that includes the run started by the MCP start_run tool when the server runs runs inline: a queued job of another run (one left behind by an earlier hairspring test that was killed hard, say) is not run, started or marked as running by it. It is left for a worker, so start hairspring worker if that job should finish. Only the command's own run is cancelled. A test run is left in the state cancelled in hairspring results, with the trials already judged kept. An interrupted confirm stores the checks that finished, and records the check that was running and the rest as skipped, so it never changes an earlier finding. A second Ctrl-C ends hairspring straight away, as before. hairspring worker does not change: Ctrl-C ends it in the usual way. Ctrl-C or SIGTERM (what systemd sends) on hairspring serve ends a confirmation started from the browser between checks: the run in progress finishes: a matched rerun is stored inconclusive (cancelled), a replay that reproduced is stored pass (and the confirmation counts as finished), and a run that did not match or reproduce is stored skipped, as hairspring confirm stores it on Ctrl-C (systemd's default KillMode=control-group signals the Hermit process too, so a run can die with the stop). Checks that did not run are skipped and the server exits. A confirmation with a skipped check leaves the finding where the last finished one put it. The server waits up to half a second for the stop to be heard before it treats a run killed by SIGTERM, or a replay that did not reproduce, as a miss, so a stop that reaches it later than that is not noticed; the strict verify has no such handling. The systemd unit in the repository's deploy guide sets KillMode=mixed so SIGTERM goes to the server alone; we checked that against a fake Hermit on a Mac, not real Hermit.

We tested this with a fake hermit script on a Mac (sending SIGINT to the command and checking that the script's process and its child were gone). We did not run it against real Hermit, which needs Linux x86-64.

If your fan is still spinning after an interrupted run, for example after a kill -9 of hairspring (which cannot be handled) or after a second Ctrl-C:

pgrep -af hermit      # list Hermit processes and their command lines
pkill -f hermit       # stop them (check the list first: this matches every Hermit on the machine)

Those two commands are ordinary Linux tools, not part of Hairspring; use whatever you normally use to find and stop processes. Cancelling through the server (REST, MCP or the UI) uses the progress callback and records skipped.

Leftovers in temporary directories

Hairspring keeps Hermit's per-trial summary files in a private directory below /dev/shm when that exists (mode 0700, named hairspring-<uid>-<pid>, with files named <seed>-<hex>.json). In the code we read, a trial removes its summary file whether it finished, failed to start or timed out, and the directory is removed when the run ends normally. The directory must be outside /tmp, because Hermit gives the guest its own /tmp.

Each Hermit trial also gets its own private directory, made in /tmp (the place Hermit made its files) and removed by Hairspring itself. A trial that runs past trial_timeout_secs is stopped with SIGKILL, so Hermit cannot clean up after itself, but this directory is removed anyway. Hairspring points Hermit's own temporary files at it. That is safe only where Hermit does not hand its environment to the guest, so it depends on the target's --base-env:

--base-env After a timeout
minimal or empty nothing is left in the host temporary directory
default (host) Hermit's strict-verify logs are cleaned up, but its private guest /tmp, its copy of /etc/group and, when /var/run/nscd exists, its empty nscd directory (names like .tmpXXXXXX) can still be left in /tmp; ls -a /tmp shows them, and they are safe to delete once no hermit is running

If a strict verify fails, Hermit prints where it kept its two run logs; those now sit inside the trial's directory, which Hairspring removes, so the printed paths no longer exist (the verdict does not depend on them). A directory that cannot be cleaned is reported on Hairspring's standard error and does not change the trial's result. A strict verify (Hermit's own --verify) also stages a file named .hermit-verify-summary-XXXXXX in the directory Hermit runs in: the target's working directory (its cwd setting, or the directory hairspring was started from when none is set). If the working directory sits inside a git work tree whose top-level .gitignore has an ignored/ line (the nearest enclosing work tree is used), Hermit puts the file in that tree's ignored/ directory instead of the working directory. Hermit removes the file when it exits normally. If Hermit is killed, for example when the verify runs past its time limit, or on Ctrl-C, Hairspring removes the staging files (regular files with that name that were not there before the verify) from the working directory; one it cannot remove is named in the check's detail. A file in ignored/ of the enclosing work tree stays, and so does a file left when Hermit crashes on its own or when Hairspring itself is killed; those are safe to delete once no hermit is running. Another verify running in the same directory at the same time can lose its file to this clean-up and become inconclusive. Run hairspring confirm from a working directory that is not shared and does not hold evidence, and look for .hermit-verify-summary-* (in ignored/ of the enclosing work tree, if there is one) after a killed verify. This was checked with a stand-in for Hermit, not run against Hermit here. The failed-verify logs are separate: they live in the per-trial directory and are deleted with it.

The guest sees the same /tmp and environment as before. Hairspring does not pass --tmp for this. If a trial's directory cannot be made, the trial fails with cannot start hermit: cannot create ... instead of running. The directory of a trial is not removed if Hairspring itself is killed with SIGKILL; it is named hairspring-trials-<uid>-<pid>-<hex>. This was checked with a stand-in for Hermit on macOS and by reading Hermit's source; it has not been run against Hermit here.

We did not test what remains after Ctrl-C or a kill. It is plausible that a directory named for a dead process can be left behind, particularly after a kill -9; that is an inference, not something we have seen. If you want to tidy, make sure no hairspring is running first, and do not touch a directory whose pid belongs to a live process. Bundles and the results database are different: they live in .hairspring/ in the directory you ran in, and are kept on purpose.

SDK output not showing up

Symptom Likely cause
No HAIRSPRING_EVENT lines when you run the program natively Java and .NET start the SDK on first use; a program that never touches it reports nothing. Go: os.Exit and a panic bypass the report (use hs.Main). Or HAIRSPRING_EVENTS=off is set
A compatible .NET SDK was not found when you build the .NET SDK or its fixture in a clone, or the .NET conformance tests are skipped The clone's global.json asks for .NET SDK 10.0.100 or later in 10.0. Install a 10.0 SDK of that version or newer (a 9.x or 11.x SDK does not satisfy it). Only the Linux build record needs exactly 10.0.402
A property is missing from the report In Python, Java and .NET a site that never runs is invisible, not unmet; declare it up front (Guest-SDKs-by-Language)
Evaluations from a child process are missing in Python forked children's evaluations are silently lost; use spawn
hairspring: incomplete catalog: <n> evaluation(s) ran at call sites that are missing from the catalog section on stderr (source) The C SDK saw call sites outside its catalogue section. With g++ on x86-64 and aarch64 Linux this no longer happens for C++ templates once the SDK includes the inline-assembly change; on other targets or older SDK copies, use clang for C++, or keep call sites in ordinary functions

The web UI: signing in, empty tables and missing pages

Symptom Likely cause
The page shows "Sign in" and asks for an API token The server has tokens configured. Enter a token (the read role can look, the run role can also start runs). The browser keeps it for that tab only, so a new tab asks again
Sign-in keeps coming back The token is wrong or has been removed, or the server was restarted with a different token list. Ask whoever runs the server for a token
A failure page says "Candidate" and "No checks recorded" No hairspring confirm has run for it. A failure found on a simulation target cannot be confirmed, and confirming needs the process backend with Hermit on Linux x86-64. The page shows the command it would need
A run page's property table is empty and the coverage section says every count stayed at zero The target declared no named properties. This is normal for a simulation target. A process target that uses a guest SDK fills both
A run page of a search on a simulation target has no trials, or the failure page has no Open trial link A search on a simulation target stores no trials; a sweep stores one per trial, and its failures link to a trial
The Workers page has no Id column Worker ids are shown from the run role up, because an id is a credential for the worker endpoints. A read token does not see them, and the page says so
The search box is missing from the top bar The server is older than the release that added GET /api/search; an older server hides the box
"This run does not exist here. It may have been removed." The address names a run the server does not hold, or hairspring gc removed it. The page does not say which
hairspring serve shows a placeholder page, not the UI The binary was built without the UI. Run make ui, then rebuild; a release binary includes it

We saw the open loopback case (no tokens), the disabled Start run button for a read token, the Candidate state, the empty property table and the Workers page without an Id column on a live server on a Mac, with simulation targets. The Sign in page itself and the other rows come from the source and the web tests.

The web UI's colour theme

Symptom Likely cause
The colour theme goes back to System each time you reload or reopen the page The browser is blocking local storage (some private windows do), so the Colour theme choice lasts only for that page. Allow storage for the page, or choose the theme again
The theme is not the one you chose on another browser or machine The choice is saved per browser and is not shared
The Colour theme switch is missing or the navigation looks old The binary may carry an older UI build. A release binary includes the UI built with it; a source build includes it only if you ran make ui first

The coverage chart on a run page

Symptom Likely cause
The coverage section says "Downsampled" The series had more than 2,000 points, or the server had already thinned a very long one. The chart keeps the first and last point and spaces the rest evenly; the table lists the points drawn. The figures under each heading still come from every point, so the trial they name may not be in the table
The coverage section says no trial has been judged yet The run has no judged trial so far. A run that is still going is refreshed every few seconds
The coverage section says it could not load coverage The page says why. A run that is going tries again by itself; for a finished run, reload the page
A line is flat for a long stretch Nothing new was seen in that stretch of trials. That is not a claim that nothing is left to find

The trial page and the failure page in the web UI

These notes apply to builds made from the current source. They were checked by the web test suite, not by eye in a browser on every screen size. We also opened the failure page and a trial page on a live server on a Mac, with simulation data.

Symptom Likely cause
The trial page shows one highlighted caveat and the rest are far down the page The page leads with the caveat that most changes how to read the result (a run that stopped at its first failure, then a perturbing trial, then missing or cut-down detail). The others, and the note that the page re-runs nothing, are under About this page. Nothing was removed
The list of sections at the top of the trial page is a drop-down on a phone On a narrow window the list of section buttons is replaced by a Jump to section drop-down. Choosing a section scrolls to it and moves keyboard focus to its heading. Checked in the source and tests only, not in a browser here
Thread lanes are different lengths, or a longer lane looks like more time Each lane is drawn to its own thread's evaluation count and says so underneath ("0 to N evaluations"). Lanes are not aligned in time and no order across threads is recorded
Only one of stdout and stderr is visible in Output Output shows one stream at a time. Use the stdout and stderr tabs (with the keyboard, the arrow keys, Home and End move between them). Find marks matching lines in every stream and counts them for the one you are looking at; Only matching lines hides the rest. Line numbers count from the first line shown, not from the start of the program's output
A timeslice field shows text instead of numbers The scheduler summary is shown as fields only when it is in the expected shape. Otherwise the page shows the text as the worker reported it
The failure page has no Open trial link, or the trial page has no Shortest failing steps for this run link The link appears only when the run has a stored failing trial, or a bundle failure, to go to. A search on a simulation target stores no trials. If the lookup fails, the link is left out and the page is otherwise unchanged
Copy link or Copy summary says it could not copy The browser refused clipboard access (some require https or a click on the page first). Copy the address from the address bar instead
What is in Copy summary The seed or failure, run and trial numbers, how many properties failed, the first failing property id (escaped and cut short), and the link. It never includes the program's output, failure messages, steps, witness values or the bundle

Counts and filters in the web explorer

Symptom Likely cause
A rate shows 0/1000 while another count on the page shows 1,000 Counts inside a rate are printed as plain digits, matching the server and the command line report. Other counts in the explorer, such as trial totals, use thousands separators
A Failures page with a state chosen shows fewer than 50 rows, or none, but Load more is offered A failure's state is worked out from its stored checks, so the server reads at most 1000 failures per request. If none of those match, the page is short or empty. Press Load more to read further back
A run I was watching under a state filter has gone When a run stops matching the chosen state it is dropped at the next refresh. Rows added with Load more show their state as it was when they were read; reload the page to refresh them
The target box says the name is not valid A target name has 1 to 64 bytes and no control characters. A name with characters outside ASCII may be shorter than 64 characters
"That is as far back as this list can be read" The list has reached position 100000. Add a filter, such as a state or a target, to see older rows
A trial row lists a property as failing but the property table says it was not violated The Failed in this trial column is judged by the kind each trial declared. The property table judges each property by the kind the run saw first, so the two can differ when a property's declared kind changed
The seed list returns a 400 error when filtered by schedule fingerprint The fingerprint must be 16 to 64 lowercase hexadecimal digits. Anything else is refused instead of returning an empty list

These notes were checked by the test suites, not on a live run.

The organisation name, the run page tabs and the Start run button

Symptom Likely cause
The server refuses to start and mentions the organisation The name must be at most 80 characters with no line break, other control character or invisible formatting character (such as a right-to-left override). The message says whether the value came from the flag or HAIRSPRING_ORGANISATION but does not print it. Shorten or retype it
The organisation name is missing from the top bar The name is shown only when the browser has a valid token, and not while its requests are being rate limited. Check the token, or wait a minute
Start run or Start sweep is greyed out, with a line saying your role (read) cannot start runs The token saved in this browser has the read role, which can look at runs but not start them. Ask whoever runs the server for a token with the run role and save it in the browser; the page checks your role again when the token changes
The role line says it is checking and does not go away The server did not answer. After 5 seconds the buttons become available and the server decides; a token that may not start runs still gets a 403
A section says "Could not load ... until the connection returns" The page cannot reach the server and the banner at the top says so. Sections whose requests got no answer show this short line instead of the browser's own message. The page tries again by itself and the line goes when the connection returns. If a section instead shows a message such as "Could not load workers: HTTP 500", the server did answer with an error; that is not a connection problem, so check the server's own output
A run's coverage section says "Every count stayed at zero over N trials" Every count in the coverage section (property ids reached, property ids seen true, distinct schedules) stayed at zero across the trials judged, so the page shows one sentence instead of three flat charts. It records that none were counted, not that none exist
A wide table or a long log needs sideways scrolling on a phone Wide tables, the bundle text and long logs scroll inside their own labelled box, which takes the keyboard focus; the page itself should not move sideways
The Workers page lists no workers Start the server without --workers 0 to get one inside the server, or run hairspring worker against it. Queued runs do not start until a worker holds them
Jobs are queued but every worker is Idle or Not seen recently Check that a worker can reach the server and uses a token with the run role. A worker that is Not seen recently has not contacted the server within one lease length (two minutes by default)
A worker is Untrusted Its data is shown labelled as unverified and cannot back a claim. An administrator can mark a worker trusted through the API
No Id column on the Workers page Your token has the read role. Ids are shown from the run role
Cells on the Workers page say "Not reported by this server" The server is older than the page and does not send worker state or held runs. Update the server
The search box says "No matches." for an id I know exists The search looks in the 1000 newest runs (10 for property ids) plus a direct lookup of a whole run or failure id; older ids and partial text of older runs are not found. Paste the whole id, or open it from a list
The search box says "Too many searches just now" or "Try again in N seconds" The server's rate limit applies to search like any other request. Wait for the time shown; the box stops searching until then
The search box is missing The server is older than the release that added GET /api/search; the box hides itself when the server answers 404
The search box says "Some results could not be shown" The server returned ids the page cannot link to; open them from the lists or search the whole id
An error message ends in "..." Error text that came from the request, the program under test or a worker is cleaned (control and invisible formatting characters become spaces) and cut to 500 characters, ending in "..." when it was cut. The full text is not kept. For a bundle or request that is too long to read this way, check the file itself
Registering a worker fails with "worker name must not contain control or invisible formatting characters" Worker names are plain text of at most 64 bytes with no line breaks, other control characters or invisible formatting characters. Pick a plain name
hairspring gc says "another hairspring gc is running against this database" and exits 2 Only one gc runs against a database at a time. Wait for the other to finish. On SQLite a leftover <database>-gc.lock file beside the database does not block by itself
hairspring gc --apply removed nothing and exited 2 --apply needs --cutoff, --expect-runs and --expect-digest copied from a report, and the runs found must be the runs the report showed. Run the report again and use its last line
no stored run, or a 404 for a run you had a link to The run may have been removed by hairspring gc. The web UI says "This run does not exist here. It may have been removed." and does not yet say that retention removed it
503 "too many open connections" The server is at --max-connections. Wait a second and retry, raise the limit (and the file-descriptor limit), or look for a client holding many connections open
403 "confirmation from the browser is not enabled on this server" ("code":"confirm_disabled") The server was started without --max-confirmations. Run hairspring confirm <failure id> instead, or ask whoever runs the server whether it should be turned on
409 "a confirmation of this failure is already running" ("code":"confirm_running") Someone has already started one for this failure. Wait for it to finish; the page shows its checks when they are stored
429 "too many confirmations are running; try again shortly" Every confirmation slot (--max-confirmations) is in use. Retry after the Retry-After time (30 seconds). Nothing was queued
400 "this server has no configuration for the failure's target, so it cannot confirm it" The failure was stored by a target that this server's hairspring.toml does not have. Confirm it on a machine that has the target, or add the target
400 "only a failure of a process target can be confirmed, and this one is not" The failure came from a simulation or WebAssembly target, which cannot be confirmed
400 "the failure's bundle is for a different target, so it cannot be confirmed" The stored bundle names another target than the failure does. The server refuses such a failure from a worker, and neither the server nor hairspring confirm (by failure id or run id) will confirm one that is already stored; hairspring confirm prints the same words and exits 2. For a run id it goes on to the run's other failures, names each refused one on stderr and exits with an error at the end
400 "the failure's bundle cannot be read, so it cannot be confirmed" The failure's stored bundle is missing or is not a readable bundle. hairspring confirm <failure id> prints the same words (it used to print an internal error for a missing bundle)
404 "not found" on confirm The text is not a failure id, or the server does not know that failure
Every check of a browser confirmation is "Inconclusive" and says Hermit could not be started The server has no Hermit (not Linux on x86-64, or HAIRSPRING_HERMIT is wrong). A confirmation whose every check is inconclusive (for example because Hermit could not be started) still counts as finished. A failure's state is read from the newest finished confirmation, so such a confirmation replaces the state from the one before it and can move a confirmed failure back to candidate. The earlier checks stay stored and listed.
429 "too many replays are running", or the MCP tool error replay_busy Other replays are running. Retry after the Retry-After time (5 seconds), or raise --max-replays (1 to 64)
429 "too many open event streams for this caller" One token (or one client address without tokens) already has 16 live streams open. Close some browser tabs or dashboards
408 "the request body did not arrive in time" The body took longer than 30 seconds after the request head. Check the network path or proxy; the connection is closed
400 {"error": "invalid query string"} A query parameter has the wrong type, for example ?after=abc
Valid tokens or workers get 429 behind a reverse proxy Without --trusted-proxy, every caller behind the proxy shares one address, so requests through the proxy without a valid token can use up that address's allowance, and valid tokens arriving the same way are then refused for up to a minute. Start the server with --trusted-proxy <proxy address> and make the proxy overwrite X-Forwarded-For
error: --trusted-proxy: ... at start Entries wider than /8 (IPv4) or /32 (IPv6), the unspecified address (0.0.0.0, ::) and ranges starting at it, and IPv4-mapped or IPv4-compatible IPv6 entries are refused. Name the proxy's own address or network. The message shows at most the first 16 characters of the entry, with control characters escaped. "has bits set past its /24 prefix; did you mean 10.0.0.0/24?" means the network was written with a host address; use the suggested form
no hardware PMU registered: ... lowering perf_event_paranoid will not help The kernel lists no hardware performance monitoring unit: a virtual machine or container that does not pass the counters through, or an emulated machine. Changing perf_event_paranoid does not add one. Use bare metal or a VM that exposes the PMU. If the report says the cause cannot be determined, /sys is not visible (a chroot or sandbox): run the check where /sys is mounted See Is-My-Machine-Suitable
target NAME: program PATH not found or not executable The target's command[0] is not an executable file where Hermit looks: relative to the target's cwd for a path, on the guest PATH (/usr/local/bin:/usr/bin:/bin unless the target sets PATH) for a bare name. Fix the path or build the program first. Exit 2; nothing was tested. When the message ends in (Hermit: ...), Hermit itself could not find the program inside the guest. If hermit_args change what the guest sees (--mount, --bind, --env, --base-env, --image and similar), a program missing on the host only gives the warning could not find PROGRAM on the host; Hermit's arguments change what the guest sees, so continuing, and Hermit decides
go: go.mod file not found ... 'go get' is no longer supported outside a module Run go mod init example.com/demo first. See Guest-SDKs-by-Language
error[E0463]: can't find crate for `std` on a musl build The target is not installed: rustup target add x86_64-unknown-linux-musl. Homebrew's Rust has no rustup, so build in a linux/amd64 container. See Build-a-Linux-Executable
The service is still running after a driver test The driver exited without stopping its child, or the child called setsid and left the process group. See Testing-a-Service
doctor says two runs of a service test differ The output contains the clock, a pid or an ephemeral port. See Testing-a-Service
the worker token file must contain one bare token (the server's tokens file has token:role lines) The file given to hairspring worker --token-file holds a token:role line or several lines, as the server's tokens file does. Put only the token itself in the worker's file
A run page opens on Properties when the address names another tab Only seeds, failures and activity are recognised after ?tab=; anything else opens Properties
storage error: database schema version N is not supported (expected M) and exit 2 A newer hairspring opened this results database and migrated it; older builds refuse it. Keep using the newer build, or point the older one at a new --db. Back up before upgrading: (umask 077; sqlite3 .hairspring/hairspring.db ".backup 'hairspring-backup.db'"). See Upgrading-and-Versions
storage error: database schema version 8 is not supported (expected 7) A release that uses schema version 8 has upgraded this database, and the release you ran expects 7. Use the newer release, or restore the backup you took before upgrading. See Installation
must be owner of table ... when starting against PostgreSQL after an upgrade The one-time upgrade to schema version 7 (and again to version 8, which adds an index on runs) creates indexes, and PostgreSQL lets only the table's owner do that (it has no separate grant for creating indexes). Run the first open as the role that owns the schema's tables, a member of that role, or a superuser. The schema stays at its old version until that succeeds. After a version 8 upgrade, other roles also need a grant: see the next row
error: cannot use the PostgreSQL database: storage error: role X cannot use table run_inputs: ask the role that owns the schema's tables to run GRANT SELECT, INSERT, DELETE ON <schema>.run_inputs TO X and open the database again (nothing was changed) A role that only reads and writes rows opened a schema 8 database, and the owner has not yet granted it the new table run_inputs. Hairspring refuses at open, before changing anything, because every first batch of trials deletes inputs an earlier attempt left, and the role could not do that. As the role that owns the schema's tables, run the GRANT shown in the message (with your schema and role names), then start again. The owner and superusers do not need it
index NAME exists but is invalid or has a different definition: drop it and open the database again An index with that name exists but is not the one Hairspring needs; often a failed CREATE INDEX CONCURRENTLY left an invalid one. Run DROP INDEX NAME (for example DROP INDEX failures_created), then start Hairspring again

These notes were checked by the test suites; the Workers page wording was also checked on a live server on a Mac.

Still stuck

Read Limits-and-Honesty and Concepts-and-Vocabulary, and check Installation for the prerequisites. The repository's own docs are the authority for anything this page paraphrases.

If that does not answer it, open an issue. Please include:

  • the output of hairspring --version (the whole line; a bare hairspring 0.1.0 with no commit is the v0.1-RC1 pre-release, see Upgrading-and-Versions);
  • the output of uname -a;
  • the output of hairspring doctor --process-backend if you use the process backend, and of hairspring doctor;
  • what you ran, what you expected and what happened, with the seed if there is one, and the relevant part of hairspring.toml with secrets removed.

Pre-release binaries are on the releases page. Do not report a security problem in a public issue: follow SECURITY.md.

Clone this wiki locally