v0.9.0

Rewind VM 0.9.0 is about builds that do not reproduce. The guest kernel now records each write to a file and each file a program opens, without adding a step, and the new rewind blame uses those records to name the program that wrote the first byte that differs between two runs, and what it read to get there. rewind diff and the desktop app compare one file across two runs, rewind check --dirents lists the guest's directories in other orders, as another ext4 disk would, and the site's viewer is now the desktop app itself, built for the browser. The reproducible builds tutorial walks through all of it.

Before you upgrade

The guest kernel is new and speaks interface 4. A run's id hashes the guest, so the same command makes a different run than it did with 0.8.1, and a seed that failed one way may now fail another. The monitor still runs 0.8.1's guest, so runs recorded with 0.8.1 replay as they are; they have no read and write records, so blame falls back to bisecting them with forks, and says so before it starts. The manifest format stays 3, except for runs made with the new --stall-max, which name format 4 so that older builds refuse them.

Why a build differs

• rewind diff --file PATH reads one file in each run where its job exited and prints a plain summary, such as neatvnc.pc differs: 16 of 407 bytes, first at byte 216., then the first lines that differ, or the bytes, as cmp shows them. rewind diff --outputs lists the files either job wrote and left behind that differ. Both take --json.
• rewind blame explains why a file differs. It names the step and the process that wrote the first differing byte in each run, and from there follows the files that writer read back to the first one that also differs, to that file's writer, and on. It prints the answer first, a sentence for each link of the chain, and a likely cause at the end, such as a command line that differs between the runs. On sagittarius-scheme it goes from the library through ld, as and cc1 to time_stub.c and the genstub run that wrote it.
• rewind check --dirents N builds the first seed's unperturbed schedule again under directory keys 1 to N and judges each run against the guest's own order. A key whose run ends differently is reported as a differing seed is, and both runs get keyframes. The key can also be set by hand with --kernel-args rewind.dirents=N, and is part of the run's id, so a run made with one replays.
• Under a key, getdents on tmpfs, ramfs, overlayfs and erofs lists . and .. first and the rest in the order of a SipHash of each name keyed by N. setuptools 80.9.0 in nixpkgs builds a wheel that says Generator: setuptools (1.0) under key 2.

Read and write records

• The guest appends a record of every write to a regular file, with where in the file the bytes went, and of every open of one, to a 1 MiB buffer the monitor drains at steps the run already makes. The records add no exit and change nothing about the run, so they are always on. They took a neatvnc build from 12.70 s to 12.91 s.
• rewind replay compares the records after the events, and reports where they first differ when only they do.
• Records the buffer had no room for are counted in a record of their own, so a trace says when it is missing some.

Schedule points

• The SDK's point() makes a VM exit and records nothing, which gives a schedule a place to switch threads in code that makes no system call, such as an atomic load and the store computed from it. C, Rust, Go and Python have it; Java's streams and the shell cannot write nothing, so theirs do not.
• A new example, counter, and a schedule points tutorial show a lost hit that no schedule finds without a point and that check finds and narrows to the point with one.
• --stall-max DURATION on run, nix, check and fork raises the longest pause a schedule gives the running thread, from 1.28 ms up to 1 s, for races that need a thread held off the CPU for longer. Go's http2 TestTransportDisableKeepAlives_Concurrency hung in 0 of 260 runs under the default, and in 7 and 19 of 260 at 50 ms and 100 ms.

Long replays

• A look inside a run (cat, diff, where, gdb, shell, threads, blame) replays from the nearest keyframe, and now keeps a keyframe after every 10 s of replay on the way. Runs that check made have no keyframes, so every read of one used to replay from boot; on a 296,484-step sagittarius run, later reads went from 10 to 75 s down to 3 to 11 s.
• The same commands show how far a replay has got as a status line on a terminal, or as JSON lines with --progress json or REWIND_PROGRESS=json.

CPUs and hosts

• --cpu v3-compat is the x86-64-v3 model without ERMS, for hosts such as the AMD EPYC 7763 that lack it. Without --cpu, such a host gets v3-compat and a line on stderr saying so. Runs recorded on it replay on any host with the v3 baseline.
• A replay, fork, shell or gdb session on a host that cannot show the run's CPU model names the model and the bits this CPU lacks, and says to replay on a host that has them, rather than giving advice only a new run can take. rewind doctor says which model a run without --cpu gets on this host, and why.
• rewind show always spells out --cpu, since a command without it picks a model by the host.
• On a CPU with 36 physical address bits, such as Ivy Bridge, --cpu host records without the extras slot, so only rewind shell --with is lost. Every run used to fail before boot with EINVAL. (#2)
• The branch counter self-test runs on the host's CPU, so a host without the whole v3 baseline can use counter time.
• rewind raises its soft limit on open files to the hard limit at startup, so check -j 256 no longer fails with EMFILE on a host whose soft limit is 1024.

Desktop app

• A new tab shows one file as each run had it, side by side or one under the other, text matched line by line with the characters that differ marked, and anything else as a hex dump with the differing bytes marked. Hovering a differing byte or line says which step's write set it and by which process, and a click goes to that write. The text can be selected and copied.
• With two runs open, a click in the Files panel compares that file, and the Compare tab lists the files that differ at the end of each run.
• A replay with more than 3 s left shows where it started, where it is, about how long is left, a bar and a Cancel button.
• A click on a row of the process tree shows that process's card: its whole command line, its parent, the steps it started, exec'd and exited at, its threads and its output up to the playhead. Attach gdb and Open shell start in it.
• The process tree starts each row with the step its process started at, and only that number seeks. The row of the thread at the playhead is lit.
• The toolbar shortens its labels in a narrow window instead of cutting off the fork button.
• A run whose failing process was killed by a signal, such as SIGPIPE, opens on the signal's delivery, not on the exit a step later.
• The divergence card, the tour, the tooltips and the Runs panel are written in plain words: the card says how each run ended and what Rewind did to one of them, "pauses the running thread for 1280 µs", where it said "only this run has a 1280 µs stall".

The browser viewer

• rewindvm.dev/viewer is the desktop app built for the browser, read only. viewer/?url=<rwd>&compare=<rwd>&step=<n> opens an export, compared with another, at a step, and the address follows the playhead. It opens files picked or dropped on the page and links served with CORS headers.
• Anything that runs the recording again, a shell, gdb, a file at a step, a fork or a check, says it is in the desktop app.
• Each case study opens its recording in the viewer, and the gc-closure case study opens with its demo video, which the front page also links.

Examples and docs

• New examples: tiles, whose pkg-config file takes its order from Python's hash seed, plugins, whose index takes its order from the directory listing, and counter.
• New tutorials on reproducible builds and schedule points. Every tutorial is recorded again with this release.
• For Go programs the docs advise GOGC=off, with GOMAXPROCS=1 as the fallback. A Go study found the collector hangs on a vCPU spinning in bgsweep, whatever --cores is, and that GOMAXPROCS=1 leaves check no goroutine order to vary.
• --timeout's help says schedule 0 has no limit without it, so a suite that hangs on schedule 0 hangs the whole check.
• check, fork and replay print what Rewind did in plain words: "a request to switch threads", "a 1280 µs pause of the running thread", "finding the step that decides it".
• The bugs page lists ten more bugs Rewind found, in process-compose, Pueue, watchexec, concurrently and Supervisor.
• The app is free for open source work: email tacos@lunchtimesurf.com with a link to the project.

Fixes

• Rewind starts when $HOME is on NFS. Lock files were opened write-only, and the NFS client's emulation of flock needs a descriptor open for reading for a shared lock, so every command failed with EBADF. (#4, with a reproducer from @knedlsepp in #3)
• A run that fails before its machine starts, as when the host lacks the recorded CPU model, no longer leaves a directory behind that rewind ls and the app listed as interrupted.
• blame names the program that made the write: a bash that redirected and then exec'd python3 is now python3.
• The app's gdb pane starts gdb with pagination off. The first command typed used to answer gdb's "Type for more".
• A command line with newlines in it stays on one row of the process tree.
• The file viewer's hex dump colors every byte of a line. It used to color the first byte alone.

New Contributors

• @knedlsepp made their first contribution in #3

Full Changelog: v0.8.1...v0.9.0