Skip to content

Limits and Honesty

Tom Barber edited this page Nov 6, 2026 · 12 revisions

Limits and honesty

Hairspring's documentation tries to state only what was measured, and to say where something was not. This page gathers those statements in one place so you can judge whether Hairspring fits your situation before you invest time in it.

If anything here disagrees with the repository, the repository is right. The key sources are the README, docs/support-matrix.md and docs/determinism.md.

Release status

  • Hairspring is experimental. The README says APIs and the bundle format will change, and that while hairspring-core has a small, tested surface, "the rest is young".
  • v0.1-RC2 is a release candidate. The tag reads "Hairspring 0.1 release candidate 2". It is a release candidate, not a general release, and it should be treated as such.
  • The process backend is marked Experimental in the README's backend table. The sim core is marked stable and the wasm backend working.

Language support under Hermit

This is copied from docs/support-matrix.md. That page "says which languages have passed our acceptance test under Hermit, and on what. It states only what was measured."

Language Status
C Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere.
C++ (clang++ and g++) Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere.
Go Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere.
Rust Supported: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build); not tested elsewhere.
Python Not supported on this build: the probe program did not complete under Hermit.
Java Supported on a Java-specific configuration: passed our acceptance test on one configuration (Ubuntu 24.04, one Xeon host, pinned Hermit build) with skid_margin = 35 (the other languages: 25), JVM flags -Xint -XX:+UseSerialGC -XX:ActiveProcessorCount=1 -XX:-UsePerfData -Xshare:auto, Temurin 21.0.12.1, and a reduced fixture (50 iterations per worker instead of 1,000); not tested elsewhere.
.NET SDK and fixture available. Not supported on this build: the probe program did not complete under Hermit.

Notes from the matrix:

  • C++: passed separately with clang++ and with g++.
  • Go: 3 of the 400 trials (1 in the failing arm, 2 in the passing arm) gave no verdict and are not counted.
  • Rust: the probe program passed, and the acceptance test passed.
  • Java: Trials take about 210 s as run (up to three at once; about 105 s alone); suited to nightly runs. Replaying a recorded run matched only from the same paths as the recording (JDK install, program jar and working directory). At skid_margin = 25 the probe program did not complete under Hermit.

The one configuration

From the support matrix, exactly: every result above except Java's was measured on this configuration and no other (Java's exceptions are the last two bullets).

  • Ubuntu 24.04.5 LTS, kernel 6.17.0-42-generic.
  • One bare-metal host with an Intel Xeon E5-1410 v2 (Ivy Bridge) CPU.
  • Hermit executable with SHA-256 666a8713d510bcd57cf2ea4c419c23368f8e7f5fb6c279a7b4766a142e62674a, run as root.
  • hairspring 0.1.0 (engine commit 1bae6f7), chaos mode with max_timeslice_ns = 100000 and skid_margin = 25 (Java: see below).
  • Measured between 2026-10-21 and 2026-10-25.
  • Java used skid_margin = 35 and the JVM flags above.
  • Java was measured on 2026-10-26 and 2026-10-27.

"The results apply only to that configuration. We make no claim about other hosts, kernels or Hermit builds."

How to read those words

  • "Supported" has a narrow meaning: the language passed the acceptance test on that one configuration. It is not a promise that it will work on your machine, kernel or Hermit build. It may well work; it has not been measured.
  • "Not supported on this build" means the probe program did not complete under the pinned Hermit build. It is a statement about that build and configuration, not a judgement that the language can never work. The Python and .NET SDKs and fixtures exist and are documented.
  • No SDK is needed to run a program. The process backend runs any Linux command and treats a non-zero exit as a failure. Whether a given runtime behaves under Hermit is a separate question from whether an SDK exists for it.

The repository's determinism document also records earlier measurements on a different setup (kernel 6.8), including an unmodified Python consumer behind a simulated Dapr sidecar. The support matrix is the statement for the current pinned build; the two are about different configurations.

Platform limits

  • Linux x86-64 only for the process backend. It needs hardware performance counters: bare metal, or a VM that passes the CPU's counters through. macOS and Apple silicon cannot run it. The CLI and the sim and wasm backends run on macOS and Linux.
  • Only bare metal was tested. Ordinary cloud VMs and containers were not, and often do not expose the counters.
  • Root only was tested. The measurements ran Hermit as root. Unprivileged use was not tested.
  • Kernel. Linux 6.9 or newer is recommended; older kernels take a slower ptrace path. Schedules are not affected, timeouts may be.
  • hairspring doctor --process-backend tells you whether a machine meets the prerequisites (exit 0) or not (exit 4). "ok" means the prerequisites are met; it does not mean a program has been run under Hermit there.

Hermit is third-party

Hermit is a deterministic-execution tool from Meta (BSD-3-Clause, Copyright Meta Platforms, Inc. and affiliates). Hairspring builds against the maintained fork, rrnewton/hermit. The repository is clear that:

  • Hermit is a third-party project Hairspring does not control, and a young one;
  • the installer builds a pinned fork at a pinned commit with a small backport patch: read what it builds before running it on anything important;
  • a rebuild is not expected to produce the same binary hash, and a recording made with one Hermit build is tied to that build;
  • runs are slower than native: the README says roughly an order of magnitude, and the quick start says by an amount that depends on the program and the kernel;
  • Hairspring does not claim Hermit is production-hardened.

Known failure mode: skid overshoot

Hermit preempts threads using a hardware counter of retired branches (see How-Deterministic-Simulation-Works). The interrupt that stops a thread can arrive late; this is called skid. If it lands too late, the schedule is no longer the one the seed describes.

With skid_margin set, Hairspring asks Hermit to panic when this happens, so the failure is visible rather than silent. What Hairspring then does:

  • The trial is reported as an infrastructure failure, with the message "hermit panicked on a PMU skid overshoot: the preemption landed late, so this schedule is not reproducible; raise skid_margin". It is not a verdict about your program.
  • In a sweep, it is an infra row: listed apart, never in the denominator, never a pass. Three host failures in a row end the sweep as an error.
  • In a confirmation, the run is retried once at the same flags, then once at double the margin. A run at a higher margin is a different schedule and cannot count toward the reruns.

How often it happens depends on the machine, the program and the margin. As one data point, the repository's Go walkthrough reports four Hermit failures with this trigger in 2,000 runs of one arm (and none in the other two arms), and states that "nothing here says they are harmless, rare in general or unrelated to the workload".

The host is an input

  • Hermit's guest sees host files and the environment. The backend pins the environment and freezes Python's bytecode caches, but other mutable host state can still change results.
  • A package upgrade that changes the dynamic loader can stop an old bundle replaying. Replay detects this and refuses, rather than reporting a misleading pass.
  • Moving to another machine means recording again.
  • A bundle records hashes of the program's command line and working directory, not the paths. Replay always runs the command and directory from the current configuration. A program whose behaviour depends on the text of its own paths can follow a different schedule when the same files run from a differently named directory. This has been seen with a JVM, where the install path, the class path and the working directory are part of what the program reads. Replay now says when either differs from the recording, and a miss then exits 3 (inconclusive); it cannot show the old value, only that it differs. A bundle recorded before these hashes existed says they were not recorded. Replay from the same absolute paths as the recording. A hash is not a disguise: someone holding a bundle can hash guessed paths, such as /home/<name>/project, and compare, so a user name in a directory name can still be found. Check before sharing a bundle.
  • Java is supported only on its own configuration: skid_margin = 35 (the other languages: 25), interpreter-only JVM flags (-Xint -XX:+UseSerialGC -XX:ActiveProcessorCount=1 -XX:-UsePerfData -Xshare:auto), a reduced fixture (50 iterations per worker instead of 1,000), and slow trials (about 210 s as run (up to three at once; about 105 s alone)). It was not tested elsewhere.
  • Instrumentation (an SDK, or Hermit logging) changes the schedule compared with an uninstrumented program.

Inconclusive is not pass

Several outcomes look like "nothing went wrong" but are not treated as passes:

  • a trial that times out is inconclusive, unless you set hang_is_violation = true;
  • a trial whose guest events are unusable is inconclusive and never rated;
  • a replay that does not fail on a changed environment exits 3: "inconclusive, not evidence the bug is fixed";
  • with nothing rated, a report says 0/0 rated trials and that there is no rate. It never says 0%.

What is not claimed

From the README:

  • bit-exact replay of arbitrary programs on macOS (there is no supported way to do this);
  • a general-purpose deterministic hypervisor;
  • that Hermit is production-hardened.

From the documentation more broadly:

  • No rate is portable. Every count belongs to one machine, kernel, compiler, build and run.
  • Hits are not bug rates. Reports say "hits", never "found", "discovered" or "bug rate".
  • The published experiments say what they did not show. The glibc lost-wakeup walkthrough stopped, by its own pre-set rule, before its counted runs, and says it has no statement about whether Hermit reaches that bug at all. The SQLite WAL-reset page reports that the bug was hit only with a patch that widens the race window, and not on the unmodified build. Both are documented as they happened.

Confirming from the browser

It is off unless the person running the server turns it on (--max-confirmations). When it is on, any run token can start a confirmation, and the server records the checks as its own trusted runner, so a run token holder can make a confirmed failure become not reproducible, or the other way round. It also uses the server's machine time: up to the configured number of confirmations at once, each for up to two minutes, and a run token can keep every slot busy for that long, again and again, until per-token limits exist. It does not run on remote workers, and only a process target can be confirmed.

A confirmation reruns the target named by the failure's bundle. The server now checks that this is the failure's own target, both when a worker reports a failure and before it confirms a stored one, so checks cannot be recorded against a target they did not run.

A confirmation whose every check is inconclusive (for example because Hermit could not be started) still counts as finished. A failure's state is read from the newest finished confirmation, so such a confirmation replaces the state from the one before it and can move a confirmed failure back to candidate. The earlier checks stay stored and listed. Whether that is the right rule is open; it is how it works today.

Stopping the server (Ctrl-C or SIGTERM) during a confirmation cuts it short. A check whose run the stop killed (systemd's default KillMode=control-group signals the Hermit process with the server) is stored skipped, as an interrupted hairspring confirm stores it, so that confirmation is unfinished and the finding stays where the last finished confirmation put it. This holds when the server hears the stop within half a second of the Hermit process ending; a stop that arrives later is not noticed and the killed run counts as a miss. The strict verify has no such handling. The repository's systemd unit uses KillMode=mixed, which signals only the server and so avoids the window. We checked this with tests that signal the server and a fake Hermit, not with systemd.

We tested this over HTTP and with the real binary against a fake Hermit on a Mac. We did not run it against real Hermit.

Other limits worth knowing

From the README and architecture document:

  • Remote workers and the database speak plain TCP/HTTP: use a private network or a TLS proxy.
  • Large blobs live in the database; there is no S3 backend yet.
  • The simulated Dapr sidecar implements a subset only: no bindings, secrets, actors, workflows or gRPC.
  • Whether two runs used the same setup cannot be answered yet for most runs. Run inputs are recorded only for built-in simulated targets, and not for Hermit or WASM runs, for workers from older releases, or for runs stored by earlier versions. "Not recorded" means we do not know, not that the setups differ. Nothing compares runs by them yet.
  • An older release refuses a database written by a newer schema version. Back up before upgrading, and stop older servers and workers first.
  • The property filter on the failures list is bounded. It searches only the newest 1000 runs, or the one run you name with a run id, however old that run is. A failure of an older run is not listed, even if it matches, and the answer says so with cut_short: true. The work per request depends on the violations those runs hold, not on how many failures are stored. On one generated store (100000 runs, 300000 failures) on one machine, a prefix that matched nothing took 38 to 49 ms on SQLite and 198 to 225 ms on PostgreSQL; your figures will differ. In theory a guest that marks all 16384 of its properties violated in every run could push one request to tens of seconds on SQLite or a few minutes on PostgreSQL (extrapolated from those rates, never seen). The window is the newest runs at the time of each request, so while new runs arrive, a client paging through the list can see later pages shift.
  • A sweep stores at most 16 failure rows: one for its lowest failing seed and one for each later seed at which a property failed for the first time. A property whose first failing seed comes later has no row to confirm (see Sweeps-and-Confirmation).

Why this page exists

Testing tools are easy to oversell, and a tool that claims more than it measured teaches you to distrust its results. Hairspring's documents take the opposite approach: they say what was measured, where, and how, and they say "not tested" when something was not tested. This page tries to keep to the same standard.

Known advisory in the pinned Hermit build

The pinned Hermit build includes ruzstd versions affected by RUSTSEC-2026-0344 (a decompression bomb: a crafted zstd input can make the decoder allocate several gigabytes and the process abort; denial of service only, rated CVSS 7.5). Hermit decodes zstd only when it reads compressed debug sections of the program under test, which happens with Hermit's --happens-before and stack-trace options. Hairspring does not pass these by default, though hermit_args can. The effect is limited to that run failing for lack of memory. A fix (a lockfile-only update of ruzstd to 0.9.1) is carried on the fork's fix/ruzstd-advisory branch and will ship with the next pinned Hermit update. Changing the lockfile changes the built binary, so it is not applied to the current pin. cargo audit on the Hermit lockfile also reports an unused older ruzstd copy (0.8.3, reached only through an addr2line feature Hermit does not call) and warnings for unmaintained or yanked crates. These are noted; we have not found them reachable through Hairspring, but we have not audited every use.

Clone this wiki locally