Rewind VM

Deterministic Linux VMs you can scrub, rewind and fork.
Catch a flaky build or test once, then replay it exactly, step through it, and branch it.

Website · Nix tutorial · Container tutorial · Design · Releases

Rewind VM runs a Nix build, a test suite or any Linux command inside a KVM virtual machine whose every run is a function of its inputs. The same inputs give the same run, at the same steps, every time. A failure you saw once is a failure you keep: replay it, scrub through it step by step, read any file as it was at any step, and fork it with a different thread interleaving from any point.

$ rewind check github:fzakaria/rewindvm#mylib
schedule   0: exited:0       6169 steps  a9d703ba8977  run 6380bb57fcb451da
schedule   1: exited:0       7202 steps  a9d703ba8977  run e5003a7041880029
schedule   2: exited:0       7233 steps  a9d703ba8977  run 70416cb698a5d521
...
schedule   6: exited:2       5468 steps    run 1506c6347419c2d2
...

schedule 6 ends differently; narrowing the steps it perturbs
perturbing only steps 3629..5140 still ends differently
step 5139 decides it: a reschedule there makes the run fail

passing: run 1989f6d02c080616, schedule 6 over steps 3629..5139
failing: run 5c910df9774b38f2, schedule 6 over steps 3629..5140
the two are the same run until step 5139

where ./tests/test_pool_shutdown first behaves differently:
  ...
  both           5150   166/174   write(1, "job 15 done: 42559\n")
  failing        5153   166/174   SIGSEGV code=1 addr=0x108
  failing        5155   166/166   SIGSEGV code=0 addr=0x0
  passing        5151   166/174   write(1, "worker picked job 18\n")
  passing        5162   166/173   write(1, "job 17 done: 43360\n")

$ rewind events 5c910df9 | grep SIGSEGV
      5153   166/174   SIGSEGV code=1 addr=0x108
...

$ rewind replay 5c910df9
identical: 1713 events over 5192 steps

On x86_64 Linux with KVM:

$ curl -fsSL https://rewindvm.dev/install | sh

With Nix, run it straight from the flake, or install it into your profile:

$ nix run github:fzakaria/rewindvm -- pmu status
$ nix run github:fzakaria/rewindvm#app
$ nix profile install github:fzakaria/rewindvm github:fzakaria/rewindvm#app

On NixOS, add the flake as an input and turn on its module:

inputs.rewind.url = "github:fzakaria/rewindvm";

# in your configuration, with inputs.rewind.nixosModules.default imported;
# it also adds rewindvm.cachix.org to Nix's substituters
programs.rewind.enable = true;
programs.rewind.app.enable = true;
# AMD only: make the branch counter exact at every boot
programs.rewind.amdBranchCounterWorkaround = true;

Or take the tarballs from the latest release yourself:

# the rewind command, with the VM's kernel
$ curl -L https://github.com/fzakaria/rewindvm/releases/latest/download/rewind-x86_64-linux.tar.gz | tar xz
# the desktop app
$ curl -L https://github.com/fzakaria/rewindvm/releases/latest/download/rewind-app-x86_64-linux.tar.gz | tar xz
# the kernel's debug symbols for rewind gdb, next to the command (150 MB)
$ curl -L https://github.com/fzakaria/rewindvm/releases/latest/download/rewind-debug-x86_64-linux.tar.gz | tar xz

/dev/kvm must be readable and writable by you. On AMD Ryzen and EPYC, run sudo rewind pmu enable once after each boot so the VM's clock can follow the work done inside it; Time inside the VM explains why.

# build a derivation in the VM; the output is checked against the host's
$ rewind nix nixpkgs#hello

# run the derivation under many thread schedules, and name the step that decides a failure
$ rewind check github:fzakaria/rewindvm#mylib

# the same, with the line of code each thread involved was on
$ rewind check --where github:fzakaria/rewindvm#mylib

# how many of 16 schedules from a step of a recorded run end differently,
# each a fork of the run
$ rewind check --run <run> --schedule-from <step> --schedules 16 --all --no-narrow

# judge every schedule by what a correct run is: here, a config rewritten in place
$ rewind check --property 'atomic-replace=*.conf' github:fzakaria/rewindvm#config-reload

# vary the program's randomness as well as its schedule
$ rewind check --seeds 8 github:fzakaria/rewindvm#hash-order

# any command in a root filesystem: a directory, an erofs image, or a docker export
$ rewind run --root mylib.tar --cwd /src -- make check

# whether this machine can record runs, and what to fix where it cannot
$ rewind doctor

# list runs and look inside one
$ rewind ls
$ rewind show <run>
$ rewind open <run> <step> --compare <other run>
$ rewind log <run> --steps
$ rewind ps <run> <step>
$ rewind cat <run> <step> /build/env-vars

# a shell inside the VM at a step, in a throwaway fork; --with brings more
# Nix packages into it
$ rewind shell <run> <step> --pid <pid>
$ rewind shell <run> <step> --pid <pid> --with nixpkgs#strace

# gdb on a fork at a step, with the symbols and sources of the kernel and of
# the process running there; arguments after -- go to gdb
$ rewind gdb <run> <step>
$ rewind gdb <run> <step> -- -batch -ex 'break pool.c:77' -ex continue -ex bt

# every thread of a process, even with the CPU idle, as at a deadlock
$ rewind gdb <run> <step> --pid <pid> -- -batch -ex 'thread apply all bt'

# the line of the program's own code a thread was on, with its callers: by
# default the thread of the step's event; --json for programs
$ rewind where <run> <step>
$ rewind where <run> <step> --tid <tid> --json

# which thread held the CPU at each step of a window, replayed step by step
$ rewind threads <run> --from <step> --to <step>

# branch a run at a step under another schedule or seed, or replay it exactly
$ rewind fork <run> <step> --schedule 2
$ rewind fork <run> <step> --seed 7
$ rewind replay <run>

# remove a run's forks that ran exactly as an older one did
$ rewind prune <run> --identical --dry-run
$ rewind prune <run> --identical

# remove a run and every fork of it, and forks of those
$ rewind remove <run> --dry-run
$ rewind remove <run>

# remove many runs in one call, here every fork, keeping the runs forked from
$ rewind ls | awk '/fork of/ {print $1}' | xargs rewind remove

# remove the cached images and stored pages no run uses any more
$ rewind gc --dry-run
$ rewind gc

# share a run as one file
$ rewind export <run> --replayable
$ rewind import <file>.rwd
$ rewind import https://github.com/fzakaria/rewindvm/releases/download/case-studies/nix-gc-closure-sigpipe-replayable.rwd

A program can say what a correct run is from the inside with the Rewind SDK, a file or two for C, Rust, Go, Python, Java or shell: rewind check judges every run by its assertions, and the app marks where one broke.

The app scrubs a recorded run. Drag the playhead along the timeline and the build log, process tree and files follow it, or jump to the failure or to where the run parts from a passing one.

• Click a file to read it as it was at the playhead, syntax colored, or as a colored hex dump.
• Show source (s) opens the line of the program's own code the thread was on, with the stack frames that led there; click a frame to see its line. Debug info and sources come from the program itself, or from nixseparatedebuginfod2 for Nix packages.
• Search (Ctrl+F or /) finds text in the build log, the console, file paths and events, and takes the playhead to each match.
• Bookmarks (b) mark a step with a note, kept with the run and in its .rwd exports.
• Two runs that differ only in their schedules show the step where the schedules part, as a dashed blue mark, and the card says what only one of them got there, such as a reschedule.
• The Compare tab sets the two runs' events side by side from just before they part, in full, with what differs marked; x swaps to the compared run at the matching step.
• Show threads (t) draws a lane per thread over a window of steps, a bar where the thread held the CPU, this run above the compared run.
• Previous and Next can stop at one thread, process, file or kind of event.
• Open shell and Attach gdb work inside the VM at the playhead.
• Fork from here branches the run under another schedule. Its menu turns it into Check from here, which tries 8 to 64 schedules from the playhead and counts how they ended, a cell per schedule.
• The Runs panel draws a build's runs as a tree of schedules and forks.

$ nix run github:fzakaria/rewindvm#app -- ~/.local/share/rewind/runs/<run>

It opens .rwd exports too, from a file or an https URL, and comes with an example run and a short tour. See pricing for licenses.

• A SIGPIPE in Nix's gc-closure test: Rewind's first run of Nix's functional tests failed in gc-closure.sh, a flake nobody had reported. A two-line printf piped into head -n1 under pipefail is two writes, and when head exits between them the writer dies of SIGPIPE. Rewind narrows the failure to the two steps between those writes; on the host the test never failed in 200 runs.
• A hang in Nix's store schema migration: a known, fixed bug, NixOS/nix#15693, reproduced on the version it was reported against. rewind shell and gdb inside the VM pin it to SQLITE_BUSY_SNAPSHOT retried inside an open transaction, which explains why the first fix did not stop it and the second did.
• Lost task output in devenv: a known, fixed bug, cachix/devenv#2281, where a task's last lines went missing. A test that passed 2000 times on the host failed under 218 of 257 schedules, and rewind gdb shows the last line still in the reader's buffer when tokio::select! took the child's exit. The fix passes every schedule.

The derivations they run are in examples/case-studies/flake.nix.

The VM has one vCPU on stock KVM, so code in it runs on the real CPU. Its Linux kernel carries a small Rewind platform: interrupts arrive only when the VM hands control to Rewind, the clock moves only then, and the timestamp counter and hardware RNG are hidden. Each of those handoffs is a step, and the sequence of steps is the same on every run. Inputs, a Nix closure or a root filesystem, become a read-only erofs image mapped into the VM's memory. Keyframes of the VM's memory go into a content-addressed page store, so seeking to any step restores the nearest keyframe and runs forward.

A GNU hello build from nixpkgs runs at close to native speed in the VM, and its output is bit for bit the one Nix builds on the host. Design has the details, the limits, and what comes next.

$ nix develop                  # the toolchain, with REWIND_KERNEL and REWIND_INITRD set
$ cargo test --workspace
$ cargo run --release -p rewind -- nix nixpkgs#hello
$ nix flake check              # prose, the NixOS module, and a determinism check that needs /dev/kvm
$ nix fmt                      # Nix, Rust, Python, HTML and Markdown
$ nix run .#serve              # the website on a local port

• guest/linux/: the kernel patch and config fragment for the VM.
• crates/rewind-vmm/: the virtual machine monitor: KVM, boot, steps, time, keyframes.
• crates/rewind-init/: PID 1 inside the VM, a workspace of its own.
• crates/rewind-trace/: reading and querying run traces.
• crates/rewind-store/: the content-addressed page store.
• crates/rewind-core/: input images, Nix derivations, runs, seeking.
• crates/rewind/: the rewind command.
• crates/rewind-app/: the desktop app, a workspace of its own.
• examples/mylib/: the flaky thread pool the tutorials use.
• nix/: one file per derivation; flake.nix only wires them together.
• site/: rewindvm.dev.
• docs/: tutorials, the time reference and the design.
• tools/: the prose check and the local site server.

Pushing a v* tag that matches the version in Cargo.toml publishes a release with both tarballs.

The engine and the command are open source. The desktop app's source is here to read, but copying, modifying or redistributing it needs permission.