Jepsen

Jepsen-style testing

granary has a Jepsen-style concurrent workload and fault-injection test suite (issue #177), covering the workloads/nemeses a single-node MVCC storage engine most needs to stress: concurrent transactional anomalies, un-fsynced write loss, process crash/restart, and clock skew.

Source: jepsen/ in the repo. CI: jepsen-nightly.yml (GitHub Actions, ubuntu-latest) runs the full suite nightly and on demand; there's also a Forgejo Actions mirror on the primary dev repo's self-hosted runners.

Architecture

Rather than running Jepsen's full distributed control plane (overkill for a single-node engine), this uses:

1. OCaml harness (jepsen/ocaml/) — runs N concurrent Lwt worker fibers against a single in-memory or file-backed granary database, recording every operation (:invoke/:ok/:fail) into a Jepsen-format EDN history file.
2. Clojure checker (jepsen/clojure/) — reads the EDN history offline and dispatches it to Elle's list-append/check (isolation anomalies) and other Jepsen workload checkers (set, bank, counter).
3. Nemeses — crash+restart, process pause, lazyfs (un-fsynced write loss via FUSE), and clock skew (via faketime).

Backends exercised: in-memory, plain file, WAL, and encrypted WAL — the full single-node storage-mode matrix.

Try it yourself

Prerequisites: Podman or Docker, and either the dev container (Containerfile) or the full Jepsen container (Containerfile.jepsen, which bundles the JVM/Clojure toolchain + lazyfs on top).

# Build the full Jepsen image (OCaml harness + Clojure checker + lazyfs)
docker build -t granary-jepsen -f Containerfile.jepsen .

# Run the whole gated suite (all workloads × all backends × all nemeses)
docker run --rm --device /dev/fuse --cap-add SYS_ADMIN \
  -w /workspace --entrypoint bash granary-jepsen jepsen/run/ci_full.sh

Or drive individual pieces directly — generate a history, then check it:

# list-append workload, in-memory backend, 4 workers
opam exec -- dune exec jepsen/ocaml/harness.exe -- \
  --backend mem --workers 4 --ops 20 --keys 5 --history /tmp/history.edn

cd jepsen/clojure
clojure -M -m jepsen.check /tmp/history.edn -w list-append

Other workloads: --workload bank|set|counter. Other backends: --backend mem|file|wal. Add --nemesis crash-restart --crash-after N to inject a mid-run crash+restart.

Proving the checker actually catches violations

# Generate deliberately-broken histories (dirty read, lost update, ...)
opam exec -- dune exec jepsen/ocaml/negative_control.exe --

# Elle should exit 1 (violation detected) on each
clojure -M -m jepsen.check /tmp/granary_negative_dirty_read.edn -w list-append
clojure -M -m jepsen.check /tmp/granary_negative_lost_update.edn -w list-append

See jepsen/README.md and jepsen/run/README.md in the repo for the full command reference. See Home.