Skip to content

Your First Run

Tom Barber edited this page Nov 6, 2026 · 16 revisions

Your first run

This is a guided first run with the simplest bundled example, a built-in simulation target. It needs only the hairspring command (Installation), takes a few minutes, and works on a Mac or on Linux. Nothing here needs Hermit.

You will find a bug, read the minimal trace Hairspring gives you, replay it, see the fixed version pass, and then measure how often the bug shows up.

Every command and output below was run with the v0.1-RC1 macOS binary or a build of the same source, except where a line says otherwise.

1. Make a scratch directory

Hairspring writes its results to a .hairspring directory in the current directory, so work somewhere you can throw away:

mkdir hairspring-first-run && cd hairspring-first-run
hairspring targets
lock-lease               sim      Lease-based lock service; a paused client writes after its lease expired (split brain)
pubsub-atomic            sim      Same consumer with atomic check+apply+record (the fix; should pass)
pubsub-buggy             sim      At-least-once consumer; check, apply and record are separate steps (duplicate effect)

These three are compiled into the binary, so you need no config file. Each is a small model of a system, written in Rust against the Sim trait, that Hairspring can run entirely in memory. (The sim kind is explained in How-Deterministic-Simulation-Works.)

pubsub-buggy models a message queue that delivers at least once. Two consumers can receive the same message, and each handles a message in separate steps: check whether it was already processed, apply the effect, record that it was processed. pubsub-atomic is the same consumer with those three steps done as one. The property under test is that each message's effect happens exactly once.

2. Find the bug

hairspring test pubsub-buggy --seed 5
hairspring: exploring pubsub-buggy (seed 5)
FAILED  pubsub-buggy (sim)
  invariant violated; shrunk from 23 to 8 steps
  steps (8):
      1. Deliver(1, 0)
      2. Tick
      3. Tick
      4. Deliver(0, 0)
      5. Step(1)
      6. Step(0)
      7. Step(1)
      8. Step(0)
  bundle:
    hairspring-bundle v2
    seed=5
    build=0.1.0
    target=pubsub-buggy
    backend=sim
    path=aa28ac6355e750e5,81d23fd7003c2305,81d23fd7003c2305,5b2a969b42d238a4,80237c8667fadf86,cf21924e7b0ff7c7,80237c8667fadf86,cf21924e7b0ff7c7
  bundle written: .hairspring/bundles/pubsub-buggy-3893b41f.hairspring  (replay with `hairspring replay .hairspring/bundles/pubsub-buggy-3893b41f.hairspring`)

The eight characters at the end of the bundle's file name may differ from ours.

The command exits with status 1, which means "a property was violated". The other statuses are 0 (everything passed), 2 (an error) and 3 (inconclusive). Ctrl-C ends the command with 130 and leaves the run cancelled.

3. Read the output

Take it from the top.

  • --seed 5 is the exploration seed. Hairspring always prints the seed. If you leave --seed off it picks a random one and tells you which; run without a seed and you will see a different, equally valid answer each time.
  • invariant violated; shrunk from 23 to 8 steps. The search found a failing run 23 steps long and then shrunk it by removing steps that did not matter, keeping only what is needed to fail in the same way. Eight is what is left.
  • steps (8) is that minimal trace. In this model the actions mean:
    • Deliver(c, m) gives message m to consumer c. The broker hides a delivered message for 2 ticks.
    • Tick advances the clock by one.
    • Step(c) moves consumer c on by one stage: received, checked, applied, recorded.
  • bundle is a small text record of the run: the seed, the build, the target, the backend, and a fingerprint of each step taken. It is what makes the failure replayable.

So the story of this trace is short enough to read. Consumer 1 is given message 0. Two ticks pass, which is longer than the broker's hiding time, so message 0 is deliverable again, and consumer 0 is given it too. Both consumers then step through "checked", and both find the message not yet recorded as processed. Consumer 1 applies the effect; consumer 0 applies it again. The effect has now happened twice: the property is violated. Each of the eight steps is needed. Remove any one and the duplicate does not happen.

Different seeds find different minimal traces. Running hairspring test pubsub-buggy with a random seed gave us a nine-step trace that involves a crashed consumer instead. Both are real ways to break the same property.

4. Replay it

hairspring replay .hairspring/bundles/pubsub-buggy-3893b41f.hairspring

Use the file name your own run printed.

replaying 8 steps on target `pubsub-buggy` (backend sim)
    1. Deliver(1, 0)
    2. Tick
    3. Tick
    4. Deliver(0, 0)
    5. Step(1)
    6. Step(0)
    7. Step(1)
    8. Step(0)
reproduced: the property is still violated

The exit status is 1 while the bug exists. The statuses for replay are:

Exit Meaning
0 not reproduced, on the environment it was recorded in
1 reproduced: the property is still violated
2 error (including a replay refused because the dynamic loader changed)
3 not reproduced on a changed environment: inconclusive, not evidence the bug is fixed

On a sim target the replay is exact because the state is plain data. The environment rules matter for process targets; see Running-Your-Own-Software.

You can send the bundle file to a colleague. They can replay it with the same build.

5. See the fix pass

hairspring test pubsub-atomic --seed 5
hairspring: exploring pubsub-atomic (seed 5)
PASSED  pubsub-atomic  (200000 steps, 2 trials, 144 ms)
  hits: 0/2 rated trials, exact one-sided 95% upper bound 77.64% (binomial, 0 of 2)

Exit status 0. (The timing will differ on your machine.)

Look at the second line before you trust the first. Hairspring ran 2 trials and none failed. It does not say "no bug". It says that, from 0 failures in 2 trials, the failure rate could still be as high as 77.64% at 95% confidence. Two trials is very little evidence. A pass tells you only that the runs it made did not fail. Limits-and-Honesty covers this in detail.

6. Look at what was stored

hairspring results
hairspring show <run id>

results lists stored runs, newest first, with a run id like run_5ad9416a6736b439, its state, target and when it ran. show prints one run again. Both read .hairspring/hairspring.db, a SQLite file in the current directory. Add --json for machine-readable output (one JSON object per line).

7. Check that the target is deterministic

hairspring doctor pubsub-buggy
pubsub-buggy: deterministic over 8 seeds x 200 steps

doctor runs the same seeds twice and compares the traces. It matters because everything else relies on the same seed giving the same run. For your own simulation targets it reports the first divergence and a hint (look for HashMap iteration order, time, threads or randomness outside the simulation). The check is probabilistic for leaks of that kind, so try more seeds with --seeds.

8. Measure how often it fails: a sweep

A normal run stops at its first failure. That finds a bug, but it cannot say how often it happens. A sweep runs the whole trial budget past every failure and stores every trial:

hairspring test pubsub-buggy --sweep --trials 20 --seed 1

(Output trimmed to the summary lines.)

hairspring: sweeping pubsub-buggy (seeds 1..21)
FAILED  pubsub-buggy (sim)
  ...
  sweep: 20 trials: 0 pass, 20 violation, 0 inconclusive, 0 unusable, 0 infra
  failing seeds: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 (first 10 of 20 violations)
  hits: 20/20 rated trials (100.00%), Wilson 95% interval [83.88%, 100.00%]

The sweep reports violations out of rated trials with a Wilson 95% interval, names the lowest failing seeds, and counts timeouts and unusable trials separately: they are never in the denominator and never counted as passes. Do not read "100%" as a property of the world. On a simulation target a trial is one exploration round, and this toy bug is easy to hit. For real programs the numbers will look very different. Sweeps-and-Confirmation explains the report.

(hairspring confirm is for process targets. On a sim target it answers error: no backend available for target kind ``confirm is not supported by the sim backend`; the stray backticks are in the message itself.)

9. Open the web UI (optional)

hairspring serve

The server listens on http://127.0.0.1:7878 and prints where its REST API (/api), MCP endpoint (/mcp) and web UI (/) are. With no tokens configured it accepts only loopback connections. The release binary includes the UI; a source build includes it only if you ran make ui first. Stop the server with Ctrl-C. We checked that the server starts, that / serves the UI page, and that GET /api/targets and POST /api/runs work.

Every picture below comes from a scratch server on a Mac that held a few runs of the built-in simulation targets (pubsub-buggy, pubsub-atomic and lock-lease), run from the current source. Simulation targets declare no named properties and cannot be confirmed, so the pages show those states as they are. A populated property table, the coverage charts and a failure with recorded checks need the process backend with Hermit on Linux x86-64. We did not run that here, so those pages were not captured. The same walk, with the same pictures, is in the repository's quickstart. If the quickstart and this page differ, the quickstart is right.

The released binary may carry an older UI build than the current source, with older wording or without some of these pages. If a page here is missing from your build, run make ui and rebuild.

The Overview page

The Overview page in the light theme: failed runs and the newest failures under "Needs attention", then recent runs with status badges and a count of workers, beside a side navigation with Overview, Runs, Failures, Targets and Workers.

Overview, light theme, with simulation data. The name "Example Ltd" beside "Hairspring" was set with --organisation.

The UI opens on Overview. Needs attention lists failed or errored runs among the latest 20 and the newest failures. Recent runs lists the latest five, each with a status badge. Workers gives a count with the time of the most recent contact (that time says when a worker last got in touch, not whether it is working). When the server has no runs, it shows the first steps from the quickstart. If one of these lists cannot be loaded, the page says which one, and it never presents an unloaded list as empty. We checked the Overview page against a live server on a Mac.

The page has a side navigation column (Overview, Runs, Failures, Targets, Workers), which becomes a row of links under the top bar on a narrow window. The top bar has a Colour theme switch: System (follow your browser or operating system, the default), Light or Dark. The choice is saved in your browser, per browser. If the browser blocks storage, the choice lasts only for that page. We checked the navigation and the switch in the web tests and on a live server on a Mac.

The Overview page in the dark theme.

The same page in the dark theme.

The top bar in the dark theme: the Hairspring name and organisation, the search box, and the System, Light and Dark buttons with Dark pressed.

The top bar close up, with Dark chosen in the Colour theme switch.

To show your organisation's name in the top bar, start the server with hairspring serve --organisation "Acme Ltd" (or set HAIRSPRING_ORGANISATION). It is plain text of at most 80 characters, with no logo. The browser reads it from GET /api/health, which adds an organisation field only for a caller with a valid token (or any caller when the server has no tokens). A health request that carries a token counts against the caller's rate limit, and once that is used up the name is left out while the answer is still 200. We checked this in the server and web tests and on a live server on a Mac.

Start a run or a sweep

The Runs page with the Start run form set to a sweep of pubsub-atomic with 5 trials, the budget the server will enforce, and the filter bar above the list of runs.

The Runs page with the form filled in for a sweep. Simulation data.

The Runs page has a form that starts a run through the same API as POST /api/runs. You pick a target and a run type: "Find a failure" (a search, which stops at its first failure) or "Measure how often" (a sweep, which runs every trial of its budget and can take a long time). A sweep takes an optional number of trials from 1 to 10,000. The form shows the budget the server will enforce before you start. Once the server accepts the run, the browser opens its page. Confirming a failure is not started from the browser.

Starting a run needs a token with the run role. The Runs page asks the server for your role when it loads, and again after you save or clear a token. With the read role, Start run and Start sweep are disabled and a line beside them says your role cannot start runs and to ask an administrator for the run role.

The Runs page with a read token: the Start run button is greyed out and a line beside it says the read role cannot start runs or sweeps.

The same page with a read token, seen on a live server.

If the server cannot say what your role is (an older server, an error, or no answer within 5 seconds), the buttons stay available and the server decides: it answers 403 and the form says the token cannot start runs. We started a sweep from the form on a live server and saw the disabled button for a read token. The 403 case was checked with curl.

The Runs and Failures pages refresh the newest page every few seconds. Load more adds the next older page below, one request at a time; if it fails, the rows you already have stay and the reason is shown, and pressing it again tries again. When the last page arrives, a note says all rows are shown. Each page has a filter bar. Runs can be filtered by state, run type (search or sweep) and an exact target name of at most 64 bytes. Failures can be filtered by run id, by a property id prefix (failures of runs in which a property whose id starts with that text is stored as failed) and by state (Candidate, Rerun repeated, Confirmed, Verified, Not reproducible or Control failed); choosing a state adds a State column. The filters are kept in the address, for example /runs?state=failed&mode=sweep, so a filtered list can be bookmarked or shared, and Clear filters removes them. A list can be read down to position 100000; past that the page says it is as far back as the list can be read, and a filter narrows it. Rows read with Load more show their state as it was when they were read.

The Runs page with the state filter set to Failed, listing three failed runs.

A filtered list (/runs?state=failed). Simulation data.

We used the state filters on a live server. Load more, the 100000 limit and the other filters were checked in the server and web tests, not against a live run here (the scratch server held only a handful of runs).

Read a run

A run page for pubsub-buggy: Failed, a panel listing what the run shows and does not show, four tabs, an empty property table and a coverage section.

A run page, Properties tab. Simulation data: the target declares no properties, so the table is empty.

A run's page has a panel that says what the run shows and what it does not show. It states only what the server reports for that run: how many trials ran, any failure and its seed, and, for a sweep that finished, the counts and the rate the server gives for the trials that ran. A run that found nothing says only that those trials did not fail. The web UI words its results plainly: a run that found nothing shows "No failure seen" with its trial count, or "Stopped early" when it ended before its limit; it is never just "passed". A property reads "Failed", "Held" or "Never reached", and a replay reads "Failed again on replay" or "Did not fail on replay", which is one replay and not a confirmation. We saw "Failed", "Stopped early" and "Failed again on replay" on a live run; "No failure seen", "Held", "Never reached" and "Did not fail on replay" were checked against test data only.

A run's page has four tabs under the run's facts and the Cancel button: Properties (the property table and the coverage chart), Seeds and trials (a link to the seed explorer), Failures (this run's failures, with a count on the tab) and Activity (the run's events as they arrive). The chosen tab is kept in the address as ?tab=seeds, ?tab=failures or ?tab=activity; the plain run address opens Properties. The selected tab is bold with a thicker underline, and on a phone the row scrolls sideways. Left and Right, Home and End move between tabs. Opening a run page also asks the server for the coverage chart, and the page keeps its event stream open whichever tab is showing. We checked the tabs in the web tests and on a live server on a Mac.

The run page in the dark theme at 390 pixels wide: the facts, the outcome panel and the start of the tab row, under a top bar with the theme switch and the search box.

The run page at 390 pixels wide, dark theme.

The Coverage over trials section is three small charts that share one axis, the trial number. They count the distinct property ids reached, the distinct property ids seen true and the distinct schedules seen so far. A count never falls, and each chart has its own scale from zero; the label at the top of each y-axis is the round value the top edge stands for, and the line is drawn to that scale. A flat stretch means nothing new was seen there; it does not mean nothing is left to find. "Show as a table" lists the same points as figures. The chart draws at most 2,000 points; a longer series is thinned evenly and the page says "Downsampled" with how many points it shows. The figures under each heading come from every point, so the trial named there may not be one of those drawn. On a live simulation run we saw only the sentence a run with all-zero counts gets ("Every count stayed at zero over 20 trials ... This records that none were counted, not that none exist."). We checked the charts themselves in the web tests, not against a live run here.

Failures, replay and the confirmation ladder

The Failures page filtered to the Candidate state, listing three failures with their target, state, message, run, original and shortest step counts.

The Failures page, filtered to Candidate. Simulation data.

The Failures page lists the newest failures first, with the target, message, run and the step counts (original and shortest). It shows the newest 50 and has a Load more button for older ones. Without a state filter it does not show whether a failure has been confirmed; choose a state to add a State column, or open the failure to see its Confirmation section.

The property id prefix filter looks at the newest 1000 runs only. When older runs are stored, the page says "Results are limited to the newest 1,000 runs; add a run id to search an older run." The API reports the same thing in scope.cut_short (true when older runs were left out), with the window size in scope.recent_runs. To search an older run, give its run id as well (run= in the API). We checked this wording against the web code and the architecture document, not on a live server.

A failure's page shows the shortest failing steps and the bundle text. Replay runs the stored steps again on the server; Copy link, Copy summary and Copy bundle copy text (the page says "Copied", or asks you to copy by hand if the browser refuses); Download bundle saves the file for hairspring replay.

The failure page after pressing Replay: a badge reads "Failed again on replay" with a line saying one replay is not a confirmation.

After Replay. One replay on this server is not a confirmation, and it does not say how often the failure happens or that it is fixed.

The failure page for pubsub-buggy: the Confirmation section shows Candidate as the current step, with Rerun repeated, Confirmed and Verified marked Not reached, and the text "No checks recorded. This failure has not been confirmed."

The Confirmation section of a simulation failure: Candidate, with no checks recorded.

The Confirmation section shows where the failure stands on the ladder from candidate to verified, and a table of the checks recorded for it. By default it cannot start a confirmation (use hairspring confirm, which needs Hermit on Linux x86-64). If whoever runs the server started it with --max-confirmations, and your token has the run role, a Run confirmation button appears. It asks first: the confirmation runs on the server for up to 120 seconds, and the checks it stores are trusted and can move the failure's state either way. While it runs the page says so and checks every few seconds, then shows the stored checks. We tested this against a fake Hermit; we did not run it against real Hermit. No state on the page means the failure is fixed. In the checks table, "Pass" for a rerun, a replay or a strict verify means the failure was seen again as recorded. "Control failed" and "Not reproducible" are ends of the ladder, not rungs. A simulation failure has no checks, so what you see above is the honest starting state. We checked the Replay result, the Candidate state and the Confirmation section against a live server on a Mac, and the Confirmation section with stored checks, and its checks table, in the web tests only.

The failure page with the word Confirmation pressed: a box defines it as recorded checks that a failure repeats, with a Glossary link.

A word with a dotted underline opens its definition when pressed or activated from the keyboard.

Seeds and trials

The seed explorer for a sweep of pubsub-buggy: a strip of 20 judged trials all marked failed, a count by class and a table filter.

The seed explorer, opened from a run's Seeds and trials tab. In this simulation sweep all 20 trials failed.

The seed explorer has one mark per judged trial and a table of trials with filters (verdict, a property that failed or was reached, a property id prefix and a schedule fingerprint). Each row opens a trial page.

A trial page: "Seed 1 failed", a note that no detail was kept for this trial, and the schedule figures from its row.

A trial page from the same sweep. The simulation trial kept no detail, and the page says so.

Targets and workers

The Targets page listing lock-lease, pubsub-atomic and pubsub-buggy with their kind, budget and description.

Targets, as the server lists them.

The Workers page with a read token: a queue line, then a table with embedded-0 (Idle, Trusted) and remote-1 (Idle, Untrusted), their capabilities, what they hold and when they were last seen.

Workers, seen with a read token. embedded-0 is the worker inside hairspring serve; remote-1 was started with hairspring worker.

The Workers page shows what the server sends and nothing else. State is Busy (the worker holds a job lease that has not ended), Idle (it holds none and was heard from within one lease length, two minutes by default) or Not seen recently. Trust is Trusted or Untrusted, as set by an administrator; the page does not change it. Capabilities are what the worker said about itself and are not checked. Above the table, the queue line says how many jobs wait for a worker and how many are held. A read token sees no Id column, because a worker id is a credential for the worker endpoints. We checked this page against a live server with one embedded and one remote worker on a Mac.

Search and the glossary

The search box with "run_" typed in and a list of matching runs under it.

Typing run_ into the search box lists the newest runs.

The top bar has a search box. Type at least 2 characters of a run id, failure id, target name, property id, seed or schedule fingerprint; matches appear in groups, and Down, Up, Enter and Escape work the list (a second Escape clears the text). Only letters, digits and _ - . : / are accepted, up to 64 characters, and the search is case-sensitive. It looks in the 1000 newest runs (10 for property ids, and not for property ids at all when the text is a whole run or failure id, all digits or a fingerprint), plus a direct lookup of a whole run or failure id of any age, so "No matches." means none among those, not that the id was never stored. Property ids come from the program under test and are shown as escaped text. The same search is available as GET /api/search?q=... with the read role. If the box is missing, the server is older than the release that added the endpoint. We used the search box on a live server (run ids and target names). Its other groups, its limits and the rate-limit behaviour were checked in the server and web tests and in a browser against mocked answers.

The Glossary page with the first entries: Bundle, Confirmation, Control, Deterministic, Finding.

The Glossary page at /glossary. It defines the words the web UI uses, in a shorter list than the Glossary wiki page.

10. Clean up

cd .. && rm -rf hairspring-first-run

That removes the bundles and the database. Nothing was installed outside the directory.

Where next

Clone this wiki locally