Skip to content

Sweeps and Confirmation

Tom Barber edited this page Nov 6, 2026 · 19 revisions

Sweeps and confirmation

Finding a failure is the start, not the end. Two questions usually follow:

  1. How often does this happen? That is what sweep mode answers.
  2. Can I trust this failure? That is what the confirmation pipeline answers.

This page explains both, how to read their output, and what they cannot tell you. The vocabulary is defined in Concepts-and-Vocabulary.

Part 1: sweep mode

Why a normal run cannot give you a rate

A normal run is a search. It stops at its first confirmed failure. That is ideal for finding a bug, but it means the number of trials it ran was decided by when it happened to fail. If it failed on trial 3, that does not mean the failure happens one time in three. The report says so: a run that stopped at its first hit prints its counts and "not a rate".

What a sweep does

A sweep runs the whole trial budget, carries on past every failure, and stores every trial. Because the number of trials was fixed in advance, it can state a rate for the trials it ran.

hairspring test checkout --sweep --trials 1000 --seed 1   # trials at seeds 1 to 1000
hairspring test checkout --sweep --seeds 5000..5500       # seeds 5000 to 5499 (END is not run)

Things to know, all from the README and architecture document:

  • --trials is at most 10,000, and a sweep also ends at --budget-secs (its wall-clock budget).
  • On a process or wasm target, a trial is one seed. On a sim target, a trial is one exploration round of up to 100,000 steps, and the rounds share --budget-steps. A default simulation budget (200,000 steps) runs only two rounds, so raise --budget-steps to run more.
  • The exit code is 1 when any trial was a violation, and 130 when Ctrl-C was pressed while the command ran (a run it stopped is left cancelled with the trials already judged kept and the Hermit processes killed; a run that had already finished stands as stored).
  • Over HTTP, send "mode": "sweep" to POST /api/runs; over MCP, pass mode and max_trials to start_run.

Reading a sweep report

A sweep report has three kinds of line.

The counts. One line says how every trial ended:

sweep: <n> trials: <a> pass, <b> violation, <c> inconclusive, <d> unusable, <e> infra

Then a line naming the lowest failing seeds (at most ten; if there were more, it says "first 10 of N violations").

The rate. The rate is violations out of rated trials, where rated means pass or violation. It is always written as a count with an interval, never as a bare percentage:

hits: <k>/<n> rated trials (<interval>), ...
  • With at least one hit, the interval is a Wilson 95% interval. In plain words: given what was observed, it is a range that the true rate for this setup very plausibly lies in. Wide intervals mean you need more trials.
  • With zero hits, the report gives a single number instead: the exact one-sided 95% upper bound. "0 of 1,000" does not mean "never"; it means the rate is probably below that bound. From 30 rated trials it adds the "rule of three" (about 3/N) as a note.
  • Upper bounds are rounded up, lower bounds down, and anything below 0.01% is written <0.01%. The web explorer follows the same rule as the report.

What was left out. Timeouts (inconclusive), trials with unusable output (unusable) and trials that failed in the host (infra) are listed on a not rated trials: line. They are never in the denominator and never counted as passes. A sensitivity line then shows what the rate would be if every unrated trial had been a hit, with its own interval, so you can see how far the left-out trials could move the figure. If some of the timed-out trials showed a failure before they timed out, the line says how many, for example 8 inconclusive (2 showed a failure, not counted as findings). Those failures are never hits. The number is a lower bound: it counts only timed-out trials whose events were read and whose failure is recorded against a property's final kind, so a missing or small number does not show that the other trials failed nothing. A sweep that stopped short of its budget prints the counts only, and a run stored without a property summary (including runs from before the count existed) says nothing about it.

Notes. If a sweep did not simply run its budget, it says so:

  • ended at its step or wall-clock limit: it names the limit and suggests raising it;
  • stopped without reaching a limit the server can verify (a cancel, an old worker, a memory limit): "counts only, no rate is stated";
  • ended in an error, for example cancelled or three host failures in a row: the figures are marked partial, because "optional stopping can bias this figure".

How each violation in a sweep is checked

On the process backend, the sweep reruns the seed of each of the first ten violations to see if it fails the same way. The trial's row then ends with same-output reruns k/n (how many reruns gave the same behaviour hash and failing properties), or not confirmed: ... if the limit of ten was reached, the wall-clock budget ran out, or the sweep was cancelled. The limit of ten stops a flaky target spending its whole budget on reruns.

Two details matter when you read the figures:

  • A violation counts toward the rate whether or not its reruns matched. The rerun note is information about that trial, not a filter.
  • The lowest failing seed gets a full report with a trace and a replayable bundle, and so does each later seed at which a different property failed for the first time (at most 16 in all). Two properties that first fail at the same seed share one report; a property that fails again at a later seed does not get another. One exception: if the lowest failing seed is a failure that names no property (a non-zero exit, or a HAIRSPRING_VIOLATION: line), a property that fails at a later seed still gets its own report. hairspring test writes a bundle file for each. The other failing seeds are named and stored as trial rows.

What a sweep rate means, and does not

A sweep rate is a statement about these trials, on this target, on this machine, with these settings. It is not a "bug rate" of your software in production: the report's wording rule is that "the word is 'hits'", and it never says "found", "discovered" or "bug rate". The seeds are a sample of the scheduler's choices, not of real-world conditions.

The published Go walkthrough in the repository is a good model of honest wording: it states a count of runs with the property false, out of rated runs, with a Wilson interval, for one machine and one Hermit build, and says plainly that "this is a count of runs on one machine with our test driver, not a rate of defects".

Part 2: the confirmation pipeline

Why confirm at all

During a search, Hairspring already reruns a failing seed three times and compares a behaviour hash. The determinism document calls this "weaker than what it takes to trust a finding". A failure you are going to act on, or tell other people about, deserves explicit, recorded checks. That is hairspring confirm.

hairspring confirm <run-id | failure-id | bundle>

For a run or failure id, every check is stored in the database. For a bundle file it is only printed. Confirmation is supported by the process backend only. A simulation (sim) target cannot be confirmed, in a sweep or a search: the command stops with error: no backend available for target kind confirm is not supported by the sim backend `` (the stray backticks are in the message) and nothing is stored.

The checks

Check What it does Passes when
same_seed_rerun runs the seed again --reruns times (default 3, at most 10), each with a fresh private directory, the pinned environment and stdin from /dev/null every rerun fails the same way: the same kind of failure, the same property, the same behaviour hash and the same schedule fingerprint as the recording
bundle_replay replays the bundle, with the environment check the failure repeats; a miss counts as a failure of the check only on an environment that matches the recording
fixed_control (--control <target>) runs the same seed and flags on a second target that runs your fixed build the fixed build is clean at that seed
strict_verify (--strict-verify) runs Hermit's own strict determinism verification (two runs); either exit status of your program is accepted Hermit's result file says the runs matched bit for bit, and your program failed in the verify the way the recording did (a violation of the same kind, same property where one is recorded). A match in which it did not fail is a miss, treated like a rerun that does not fail. A recorded deadlock, or a bug that is a hang, cannot be verified: the check is inconclusive. A strict verify that fails or is inconclusive never moves a finding below where the other checks put it

A strict verify stages a .hermit-verify-summary-XXXXXX file in the target's working directory and removes it on normal exit. If Hermit is killed (for example by a timeout, or on Ctrl-C), Hairspring removes the staging files that appeared in the working directory during that verify, and names any it cannot remove in the check's detail. A verify running in the same directory at the same time can lose its file this way and become inconclusive. When the working directory sits inside a git work tree with an ignored/ line in its top-level .gitignore, the file is staged in that tree's ignored/ directory instead, and a file left there by a killed verify stays. Run confirm from a working directory that is not shared and does not hold evidence, and see Troubleshooting for clean-up.

Each check ends as pass, fail, inconclusive or skipped. A rerun that cannot be judged (a timeout, a Hermit fault, an events fault) is counted apart and is never a pass or a miss. A check that did not run because you cancelled or the wall budget (--wall-secs, at most 600) ran out is skipped, with the reason.

A note on the control: a fixed build is a different program, so it has a different schedule. The control says "the fixed build does not fail at this seed", never "the same interleaving passes".

The finding's state

From the stored checks, the failure gets one state. It is always derived from a single confirmation (the newest that finished), never stitched together from checks taken at different times.

State When hairspring confirm exits
candidate no check has repeated the failure (also while the only results are inconclusive or skipped) 3
rerun_repeated the reruns repeat it; no replay of the bundle has yet 3
confirmed the reruns and the bundle replay repeat it 1
verified and Hermit's strict verify agrees 1
not_reproducible a rerun or the replay did not repeat it, on an unchanged environment 0
control_failed the reruns repeat it, but the fixed build still fails at the seed 3

Exit 2 is an error, as everywhere in the CLI. If the environment differs from the bundle's recording, misses are inconclusive: environment differs, never failures, and the command exits 3.

Confirmed versus unconfirmed

  • Confirmed (or verified) means: on this machine, under the recorded environment, the same seed failed the same way on every rerun and on a replay of the bundle. That is strong evidence the failure is real and deterministic for this setup. Only these two states reach Hairspring's generated claim sentence, which reads "Reproduced (confirmed)" or "Reproduced (verified)" and names the environment.
  • Unconfirmed (candidate, rerun_repeated) does not mean false. It means the evidence is not complete yet: the replay has not repeated it, or the checks were inconclusive. Treat it as a lead, not a result.
  • not_reproducible on an unchanged environment is itself a finding: something in the run is not under Hermit's control (see the hidden inputs in How-Deterministic-Simulation-Works).
  • control_failed suggests your fix does not fix this failure, or the failure has another cause.

Confirming some failures does not confirm the rest. In the Go walkthrough, the first 100 hit runs were each confirmed, and the page says plainly that the other hits "are counted and listed, and were not confirmed".

Practical rules from the documentation

  • No logging. A target whose hermit_args turn on Hermit logging (--log, -v, --verbose) or whose environment sets HERMIT_LOG cannot be confirmed, because logging changed a guest's path.
  • Skid overshoot. If Hermit panics because a preemption landed late, that run is infrastructure, not a verdict. Confirmation retries it once at the same flags, then once at double the margin. A run at a higher margin is a different schedule, so it cannot count toward the reruns (the check becomes inconclusive with a caveat).
  • Do not put credentials in a target's arguments. The stored command is a record of the invocation, shown to every reader of the check, and it is not redacted. It is also not a line you can paste into a shell to repeat the check.
  • Ctrl-C on hairspring confirm stops the confirmation, kills the processes still in the guest's process group (as a timeout does; a child that calls setsid escapes) and exits 130. The checks that finished are stored, and the check that was running and the rest are recorded as skipped, so an interrupted confirmation does not lift or withdraw an earlier finding.
  • SQLite allows one writer at a time. A write waits up to 5 seconds for another process to finish; if the file is still locked it fails with storage error: database is locked and should be retried.

Confirming as the sweep ends

hairspring test <target> --sweep --trials 1000 --confirm

After each target's sweep, --confirm runs the confirmation pipeline on every failure row the sweep stored, with the defaults of hairspring confirm (three same-seed reruns and a replay of the bundle), stores the checks, and prints each result after the sweep's report. It does not run a control or a strict verify: use hairspring confirm <run-id> with --control or --strict-verify for those.

  • It needs --sweep and --format text.
  • The exit status stays the sweep's (1 when a property failed). The findings are in the printed text.
  • A sim target in the sweep is swept as usual, then skipped with the note --confirm: skipped, a sim target cannot be confirmed. The later targets still run. If confirming a process target's failures fails (its bundle is missing, say), the error is printed, the later targets still run, and the command exits 2 once they have.
  • Ctrl-C works as it does for hairspring confirm: the checks that finished are stored, the rest are recorded as skipped, and the command exits 130. The sweep had already finished and stands as stored. The targets after the interrupted one are not run, and the command says so.
  • It is a command-line option only. A sweep started on a server still needs hairspring confirm <run-id>, or, for one failure, the Run confirmation button on its page on a server started with --max-confirmations.
  • A confirmation started from the page runs on the server's own engine with the same defaults (three same-seed reruns and a replay, 120 seconds), never on a remote worker, and its checks are stored as from a trusted runner. A confirmation whose every check is inconclusive (for example because Hermit could not be started) still counts as finished. A failure's state is read from the newest finished confirmation, so such a confirmation replaces the state from the one before it and can move a confirmed failure back to candidate. The earlier checks stay stored and listed.

Not run here: against real Hermit, which needs Linux x86-64. The behaviour is from the project's tests, which use a fake Hermit script, and from hairspring test --help.

Honest limits

Stated in the repository's own documents:

  • A sweep stores at most 16 failure rows. It stores one for the lowest failing seed and one for each later seed at which a property failed for the first time, so each can be put through hairspring confirm <run-id>, which confirms every failure of the run. A property whose first failing seed comes after the sixteenth row has no row, and the run can open no claim. A failure that names no property (a non-zero exit, or a HAIRSPRING_VIOLATION: line) only has the row of the lowest failing seed. A sweep does not run the confirmation pipeline by itself: each row carries only the sweep's own rerun note, and hairspring confirm <run-id> does the full confirmation of every row. To have the command line do that as the sweep ends, add --confirm (see above).
  • Simulation targets cannot be confirmed. The simulation backend answers "confirm is not supported by the sim backend", so a failing seed from a sim sweep can be read and replayed but not confirmed.
  • A passing sweep's step count is the worker's own figure. The server does not work it out. It is used only to decide whether a sweep that holds fewer rows than its budget stopped at its step limit, so a worker that reports steps it did not spend can make a short sweep look as if it reached the limit. For a failed sweep the figure is ignored and only the server's clock counts.
  • Confirmations run where the engine is: the CLI's own process or the server's embedded engine. A remote worker does not run confirmations yet. hairspring test --sweep --confirm is a command-line option only: the server does not schedule confirmations for finished sweep runs.
  • Trust. Figures from workers an administrator has not marked as trusted are shown labelled "unverified worker data", and a stored check from an untrusted runner never opens a claim.
  • A confirmation is about one machine. It shows that the failure is repeatable under the recorded environment. It says nothing about other hosts, kernels or Hermit builds, or about why the interleaving happened.

Further reading

Clone this wiki locally