faultline v0.2.0

faultline v0.2.0 runs goroutine code. A test can start ordinary Go code as a process: code that starts goroutines, serves net/http or gRPC over simulated TCP, writes files on a simulated disk, and crashes and restarts, all in virtual time inside go test. Every random choice still comes from one seed. Built through the faultline-toolexec wrapper, a goroutine-mode seed replays exactly. Without it, replay is best-effort: when a seed doesn't replay, faultline fails it and shows the first record where the two runs differ. Tests written for v0.1 run unchanged and still replay exactly.

Install

faultline needs Go 1.26 or newer. The gRPC and etcd/raft modules are only for tests that use them:

go get github.com/hmdsefi/faultline@v0.2.0
go get github.com/hmdsefi/faultline/shims/grpcsim
go get github.com/hmdsefi/faultline/harness/etcdraft
go install github.com/hmdsefi/faultline/cmd/faultline@v0.2.0

Add the nested modules without a version, as above. With a version, go get github.com/hmdsefi/faultline/shims/grpcsim@v0.2.0 can fail with module github.com/hmdsefi/faultline@v0.2.0 found, but does not contain package. You see it when go.mod already requires faultline and the module cache does not hold the nested module yet, as on a new machine or a clean CI runner. The FAQ entry on this error explains why, and how to add a specific version.

For exact replay of goroutine code, add the wrapper to your module as a tool, install it, and prepare it once per Go toolchain:

go get -tool github.com/hmdsefi/faultline/runner/toolexec/cmd/faultline-toolexec
go install github.com/hmdsefi/faultline/runner/toolexec/cmd/faultline-toolexec
faultline-toolexec prepare
go test -toolexec=faultline-toolexec ./...

The root module still depends only on the standard library and gograph. shims/netsim and shims/fsx are packages of the root module. gRPC support (shims/grpcsim) and the wrapper (runner/toolexec) are modules of their own, so google.golang.org/grpc stays out of your build unless you add it.

The Claude Code plugin's version follows the release. To update an installed plugin to 0.2.0:

claude plugin marketplace update faultline
claude plugin update faultline@faultline

Goroutine mode

Set Options.Mode to faultline.ModeGoroutine and add nodes with World.AddProcess (tagged server) or World.AddClientProcess (tagged client). A process is a func(ctx context.Context, p *faultline.Process) error that runs on its own goroutine and can start as many goroutines as it needs. It reaches the simulated world through p.Net, a *netsim.Host, and p.FS, the node's simulated volume as an fsx.FS. p.Rand(), p.Clock() and p.Logf give it seeded randomness, the node's clock and the trace, and p.Do runs a function on the simulator, for example to record history.

opts := faultline.Options{Mode: faultline.ModeGoroutine, Seeds: 100, Duration: 30 * time.Second}
faultline.Run(t, opts, func(w *faultline.World) {
	w.AddProcess("kv", func(ctx context.Context, p *faultline.Process) error {
		s, err := kv.Open(p.FS) // the service's own code, on the node's simulated disk
		if err != nil {
			return err
		}
		ln, err := p.Net.Listen("tcp", ":8080")
		if err != nil {
			return err
		}
		srv := &http.Server{Handler: s}
		go func() { <-ctx.Done(); srv.Close() }()
		if err := srv.Serve(ln); !errors.Is(err, http.ErrServerClosed) {
			return err
		}
		return nil
	})
	// ... clients, fault planners and checks, as in event mode
})

Each run happens inside a testing/synctest bubble, so time.Sleep, timers and context deadlines in the code under test follow virtual time. One driver goroutine steps the simulator, one event at a time, and only when every other goroutine in the bubble is durably blocked.

• Crash and restart. Fault schedules work as in event mode. When a process's node crashes, its context is cancelled with the cause faultline.ErrCrashed, its connections are reset, its open files go stale (ESTALE), and every later network or file call from that incarnation fails. Go can't stop goroutines, so the ones still running become ghosts: they keep running but can't change the world. A restart runs the process function again with a new Process. A paused node's network, file and Do calls wait until it resumes, and clock faults apply to p.Clock().
• End of a run. Processes get Options.Drain of virtual time (60 s by default) to exit after their contexts are cancelled. faultline logs the goroutines still blocked after that, with their stacks, and writes them to goroutines.txt. With Options.FailOnLeak, they fail the seed with kind leak.
• Stall watchdog. When the simulator makes no progress for Options.StallTimeout of wall-clock time (60 s by default), faultline writes a report to stderr and to stall.txt in the artifact directory, then ends the test binary. The report lists the bubble's goroutines and the usual causes, such as a mutex held across a blocking network call, or real I/O that bypasses netsim and fsx.
• Replay line first. Before each goroutine-mode seed, faultline prints the seed's replay command to stderr, so a panic that ends the whole test binary still names the seed that was running.
• Options.Procs sets GOMAXPROCS during a run (1 by default), and Options.NetSim configures TCP and UDP. Event mode ignores both, along with Drain, StallTimeout and FailOnLeak.

Processes and event-style nodes can share a world and exchange UDP datagrams.

Simulated TCP and UDP: shims/netsim

netsim implements net.Conn, net.Listener and net.PacketConn on faultline's simulated network: a TCP handshake, segments, retransmission, flow control, half-close, resets and deadlines, plus UDP datagrams. Node names resolve as host names, so http://kv:8080/ reaches the node named kv. netsim.HTTPTransport(h) and netsim.HTTPClient(h) give net/http a transport that dials through a host. Serve with (*http.Server).Serve on a listener from Host.Listen. faultline's tests run an http.Server and client over it, with 10 GET requests and a 1 MiB POST on one kept-alive connection.

gRPC: shims/grpcsim

grpcsim.NewClient(h, addr, opts...) connects a gRPC client through a host, with insecure transport credentials and a backoff whose jitter is 0. Options you pass override these. grpcsim.Listen(h, address) returns a listener for (*grpc.Server).Serve. faultline's tests run a gRPC health service that crashes at 1 s and restarts at 2 s, and check that the client reconnects and gets SERVING again.

Files: shims/fsx

fsx.FS and fsx.File are interfaces modelled on package os. fsx.OS(root) is the real implementation, confined to one directory with os.Root. fsx.Sim(volume) runs on a simulated volume, and Process.FS is the node's volume. The simulated disk keeps the crash models of v0.1: a crash can lose unsynced writes, keep part of them or tear the last one, and a new file's directory entry is durable only after SyncDir. fsx.IOFS(fsys) gives a read-only io/fs view, and fsx.Conformance() returns the checks faultline runs against both implementations, so you can run them against another FS.

Exact replay: the toolexec backend

The Go runtime decides some things a seed can't: which of several woken goroutines runs first, map order, select choices and global math/rand. The faultline-toolexec wrapper, in the module runner/toolexec, rewrites the packages of your module at build time so that these come from the seed. Every operation that can block or wake a goroutine (channel operations, select, sync locks, WaitGroup, Once, Cond, goroutine starts, time.AfterFunc callbacks) becomes a scheduling point, and the driver releases one waiting goroutine at a time with a seeded choice. Map ranges with ordered keys, maps.Keys and friends, math/rand and reflect.Select draw from a second seeded stream.

• When a seed is exact. A goroutine-mode seed is exact when the test binary is built with -toolexec=faultline-toolexec and Options.Procs is 1. The report says determinism: exact (toolexec backend), and report.json and FAULTLINE_RESULTS lines carry "determinism_level": "exact" and "exact_backend": "toolexec". The replay command adds -toolexec=faultline-toolexec. With Procs above 1, Run logs that the seeds are best-effort.
• FAULTLINE_EXACT=1. Set it in CI, and a goroutine-mode test that would not run exactly (built without the wrapper, or with Procs above 1) fails with a setup error before its first seed.
• What it rewrites. Every package of your module, and of any other module outside the module cache. The standard library and the module cache stay unchanged unless you list them in FAULTLINE_TOOLEXEC_STD (for example net/http) or FAULTLINE_TOOLEXEC_DEPS (module paths). Line numbers are kept, so stacks and compiler errors point at your code.
• Coverage gaps. A construct the wrapper leaves unchanged although it can change goroutine order, such as a range over a map with pointer keys or sync.Map.Range, is recorded with its file and line. A failing seed prints how many the test binary has, and its artifact lists them in toolexec-gaps.txt.
• What it doesn't control. Code outside the rewrite set, sync/atomic, data races, and goroutines woken by timers that fire at the same virtual instant. The callbacks of time.AfterFunc and context.AfterFunc start in registration order, but a woken goroutine that starts a goroutine, registers such a callback, draws from math/rand, ranges over a map or first calls a Once before its next scheduling point can still diverge. Divergence detection stays on: when two runs of such a seed in one go test run differ, the seed fails as a determinism failure.
• Cost. On an Apple M4 Max with Go 1.27.3, median of three runs: faultline's own 51 goroutine-mode tests took 1.21 times as long through the wrapper, 100 seeds of a replicated key-value store with HTTP clients 1.61 times, and 100 seeds of an HTTP program 1.01 times (3.12 times with net/http rewritten).

What best-effort replay means

Without the wrapper, a goroutine-mode run is labeled level 3, best-effort replay. An event-mode run is exact, as in v0.1: the same seed gives the same trace hash.

• What faultline controls. Time, the network, the disk and faults. Requests that processes make at the same virtual instant are ordered by node and by a key the call chooses; only requests with the same node and key keep their arrival order. Runs use one P unless you set Options.Procs.
• What it doesn't. Goroutines woken by the same event run in the Go scheduler's order. Global math/rand, map iteration and a select with several ready cases in your code use Go's own randomness. State kept between runs in one test binary (package variables, sync.Once, caches) carries over.
• How divergence is caught. The trace hash covers every record of the simulator, including each connection, segment and datagram of a process and each change it makes to a file. A failing seed runs a second time to write its artifacts, and with FAULTLINE_CHECK_DETERMINISM=1 every passing seed runs twice. When the trace hashes differ, the seed fails with kind determinism, and the message shows the first differing record, when it can the event before or during which the first run diverged, and the common causes. A difference inside a process that changes nothing it sends, writes or reports to faultline leaves the hash unchanged, so it is not reported.

Known limitations

• Code under test reaches the network and disk only through p.Net and p.FS, or anything built on them. A service needs its listener, dialer and filesystem passed in. A direct net.Dial or os.Open does real I/O outside the simulation.
• Under the wrapper with the default rewrite set, net/http's own goroutines and timers are outside the scheduler. In faultline's tests, an HTTP program with three client goroutines sharing one client diverged in about 1 of 2,500 seed comparisons; with FAULTLINE_TOOLEXEC_STD=net/http, in none of 9,960.
• Under the wrapper, goroutines woken by timers of the same virtual instant run in the Go runtime's random order, as described above.
• The wrapper must be installed and passed by name: go test -toolexec="go tool faultline-toolexec" fails, because the go command runs the assembler for standard-library packages outside your module.
• go test -cover builds are not exact: the wrapper doesn't rewrite the instrumented sources and records a coverage gap instead.
• netsim has no TLS (you can layer crypto/tls over a *netsim.Conn), no IPv6 addresses, no DNS protocol and no Unix sockets. A process and an event-style node can exchange UDP only.
• fsx.Sim has no symbolic or hard links and ignores permissions; modification times are the simulator's epoch, and there are no file locks.
• A goroutine-mode test must not call t.Parallel, and only one goroutine-mode Run runs at a time in a test binary.
• os.Exit or log.Fatal in code under test ends the test binary.

Test goroutine code and Replay goroutine code exactly walk through both modes, and the Claude Code plugin 0.2.0 covers goroutine-mode tests and exact replay.

Upgrading from v0.1.x

• No code changes. ModeEvent is the zero value of Options.Mode, so existing tests stay in event mode, and their trace hashes are the same as in v0.1.2. The golden trace hashes CI checks are unchanged.
• report.json and FAULTLINE_RESULTS lines now carry "determinism_level": "exact" for event-mode runs too, and "exact_backend": "toolexec" for exact goroutine-mode seeds. The format version is still 1; a v0.1 report without the field means exact.
• FAULTLINE_EXACT was a reserved variable, and any value was a setup error. It now takes 0 or 1; any other value is a setup error.
• World.AddProcess and World.AddClientProcess in an event-mode test fail it with a setup error.
• To move a service into goroutine mode, pass it the listener, dialer and filesystem it uses. In production that's net.Listen, a net.Dialer and fsx.OS(dir); in the test, p.Net and p.FS.
• The harness module v0.2.0, shims/grpcsim v0.2.0 and runner/toolexec v0.2.0 require faultline v0.2.0.

What comes next

The roadmap lists the phases after this one, starting with failure minimization, sweeps and history checkers.

The full list of changes is in the Phase 2 milestone.