Repository navigation
Example SQLite WAL reset
An attempt to reproduce a documented SQLite data race, under Hairspring's process backend. It is an experiment log, not a product feature, and its status line is plain: assisted reproduction achieved; unassisted finding not achieved. Read it as a case study in what a careful, partial result looks like.
Files: experiments/sqlite-wal (the code, patches, scripts and raw logs) and the short plan and status in docs/showcase-sqlite-wal-reset.md.
Fetch and build the two SQLite versions and the workload, run the workload natively, and understand exactly what the experiment did and did not show, so that you can tell a real result from a patched one.
From SQLite's own WAL documentation: a data race that can corrupt a database or lose committed writes. It affects SQLite 3.7.0 (2010) through 3.51.2 (2026-01-09) and is fixed in 3.51.3; backports exist for 3.44.6 and 3.50.7. It needs WAL mode, two or more connections to the same database, and tight timing. In outline: a checkpoint completes; a second checkpoint starts shortly after; while it starts, another connection commits a transaction that resets the WAL; because of a data race the second checkpoint does not notice the reset and leaves a wrong field in the WAL-index header; later transactions grow the WAL past the first checkpoint's threshold; a third checkpoint then skips part of a transaction, so committed data never reaches the database file.
The workload (a small C program, src/walwork9.c is the final version) has a writer connection doing tiny single-row transactions and a thread with a checkpointer and a reader connection. The property: after the workload, every committed transaction's rows are present and PRAGMA integrity_check returns ok. A failure prints HAIRSPRING_VIOLATION:.
All of this is from experiments/sqlite-wal/README.md. We did not repeat the Hermit runs.
-
Assisted, under Hermit. With a
nanosleepinserted into SQLite 3.51.2 at the WAL checkpoint's header check, the targeted workload lost exactly one committed transaction (committed=300 counter=299 rows=299) in 6 of 6 runs at one scheduler seed (4 at the default skid margin, 2 at skid margin 25). On 3.51.3 the same flags gave 0 of 29. Across a 28-seed arm, 1 of 28 seeds hit. The sleep is why it works: it turns the check into a scheduling point, which is inferred, not measured. - Assisted, natively. The same jittered patch hit in 186 of 1,641 runs on 3.51.2 and 0 of 2,363 on 3.51.3.
- Unassisted. On the unmodified 3.51.2 build there were 0 violations in 11,611 dangerous-window executions across 56 distinct schedules. The 95% upper bound (3/N) is uninformative.
-
--verifypasses at skid margins 100, 50 and 25 on the unmodified build.
Not claimed: finding the bug without the patch; random exploration finding it; a generic workload; any rate comparison with other tools. The workload is single-process; SQLite's cross-process WAL protocol was not exercised.
The key word is assisted. The patch puts a delay at the exact place the race needs, so a hit says the property and the machinery work and the scheduler can use a scheduling point you give it. It does not say Hermit would find the bug in unmodified SQLite. For that, the record is a null result with a bound that cannot tell "never" from "rarely".
| For | You need |
|---|---|
| The native steps below |
curl, sha256sum, gcc, python3, network access to sqlite.org |
| The Hermit runs | Linux x86-64 with performance counters, the pinned Hermit (Installation), and scripts you must edit (they assume the author's /opt/... directory layout) |
Copy the directory, so the download and builds do not land in your clone:
cp -R experiments/sqlite-wal ~/sqlite-wal-try && cd ~/sqlite-wal-try
sh setup.shsetup.sh downloads the SQLite 3.51.2 and 3.51.3 amalgamations from sqlite.org (they are not vendored), checks each against a pinned sha256, builds the baseline workload against each, and runs a native smoke test. It took about 20 seconds. We saw:
s3510200.zip: OK
s3510300.zip: OK
native smoke 3510200:
ok sqlite 3.51.2 committed=600 cycles=199
native smoke 3510300:
ok sqlite 3.51.3 committed=600 cycles=199
Two binaries are left in the directory, walwork9-3510200 and walwork9-3510300. You can run one by hand; the arguments are in the workload's usage line:
usage: walwork9 DB [txns writers autocheckpoint hold mode burst group sync_off yields]
The smoke command setup.sh uses is ./walwork9-3510200 smoke.db 600 1 0 5 4 0 3 1 1 (mode 4, group 3). On unmodified SQLite, natively, it passed on both versions. Mode 4 is the strict alternation: the writer commits group transactions then parks; the checkpointer backfills fully, releases the writer, and immediately runs the racing checkpoint.
The patch scripts in patches/ modify the amalgamation: patch_yield.py (scheduling points), patch_delay.py (a fixed delay in walCheckpoint) and patch_jdelay.py (a deterministic jittered delay, taken only on a fully backfilled WAL, sized by the HS_DELAY_BASE_US and HS_DELAY_SPREAD_US environment variables).
On macOS the patched build did not compile for us. The script's inserted declaration extern int usleep(unsigned int); conflicts with the macOS header (error: cannot apply asm label to function after its first use). The patches were written for Linux. On Linux, the build and run look like this (we ran it in an Ubuntu 24.04 container on Apple silicon, not in the experiment's own environment):
for v in 3510200 3510300; do
d=sqlite-amalgamation-$v; mkdir -p pj-$v; cp $d/*.h pj-$v/
python3 patches/patch_jdelay.py $d/sqlite3.c pj-$v/sqlite3.c
gcc -O1 -pthread -Ipj-$v -o walwork9-jdelay-$v src/walwork9.c pj-$v/sqlite3.c -ldl -lm
done
HS_DELAY_BASE_US=60 HS_DELAY_SPREAD_US=120 ./walwork9-jdelay-3510200 w.db 300 1 0 5 4 0 3 1 1The README's pointer is: "Hermit, one run of the assisted case: see scripts/pilot6_worker.sh (KIND=jc100d or jc100)". That script builds a command of this shape (paths are the author's; edit them; walwork8 is the earlier workload version the README says is equivalent):
hermit run --epoch=2026-01-01T00:00:00Z --chaos --max-timeslice=1000000 --skid-margin=25 --panic-on-rbc-overshoot \
--env HS_DELAY_BASE_US=100 --env HS_DELAY_SPREAD_US=300 --tmp=<private dir> --summary-json=<path outside /tmp> \
--sched-seed=3 -- ./walwork8-jdelay-3510200 /tmp/w.db 300 1 0 5 4 0 3 1
Points from the README's method notes that matter if you build on this:
- Seeds change the schedule only under
--sched-heuristic=random,stickyrandomor--chaos. A passing run always prints the same line, so output equality says nothing about schedule diversity; compare--summary-jsoncounts instead. -
--verifyneeds an idempotent guest: the workload removes its database files on every exit path, and runs use--tmpso each run gets a fresh/tmp. Pass--summary-jsona path outside/tmp. - Timeouts (
rc=124) are inconclusive, never passes. - A skid margin of 5000 made runs exceed 900 seconds; the effective default is 100, and 25 was fastest with no overshoot.
To run it through Hairspring rather than hermit directly, the shape is the one in Running-Your-Own-Software: a process target with chaos = true, max_timeslice_ns = 1000000 and skid_margin = 25, and a sweep (--sweep) over a seed range. Whether that reaches the bug on the unmodified build is exactly the open question this experiment did not answer.
-
src/walwork9.c: the workload, the modes, and the property check. -
patches/: exactly what "assisted" means, in three small scripts. -
logs/: raw result files, one line per run, and the table in the README that says which log shows what. Note that the README also names some.logfiles (native_mode4*.log,spike*.log); in the public repository only the.resultsfiles are present, because*.logis git-ignored. The native figures quoted above therefore come from the README text, and you cannot check them against a log file in the repository. Several workload versions are marked null by construction or superseded; the README says which not to reuse. -
README.md, "Method notes": the caveats, written by the people who hit them.
rm -rf ~/sqlite-wal-try. The downloaded zips and extracted amalgamations are inside it. Nothing else was written.
experiments/sqlite-wal/README.md (status, environment, logs, method notes) and docs/showcase-sqlite-wal-reset.md (the bug, why it is a good fit, the plan and its risks). The motivating public write-up is at https://antithesis.com/blog/2026/wal-reset-bug/. For how to read claims like these, see Limits-and-Honesty. Next: Examples-Overview.
Understand
- What-is-Hairspring
- Concepts-and-Vocabulary
- How-Deterministic-Simulation-Works
- Sweeps-and-Confirmation
- Limits-and-Honesty
Use
- Installation
- Your-First-Run
- Running-Your-Own-Software
- Guest-SDKs-by-Language
- Examples-Overview
- Hairspring-in-CI
- Operate-a-Server
- Upgrading-and-Versions
- Troubleshooting
Reference