Proposal: simulation testing shared by frameworks built on bedrock

[Zaba505]

Note

This is the half of z5labs/humus#1277, "Proposal: simulation testing", that doesn't depend on humus, moved here. That proposal runs a whole system of Apps in one testing/synctest bubble, in the style of TigerBeetle's VOPR. Its harness runs bedrock.App values on bedrocktest Processes, and its network is a Host, so neither needs humus. humus keeps the models of its own packages, its hooks and its CI, and builds on this proposal in its revision 1. Nothing in this proposal changes humus.

It builds on #614, the host module, and changes the jitter of #606's retry. It conflicts with some earlier decisions, among them bedrock's own testing claim. Conflicts with prior decisions names each one, for debate here.

Summary

• bedrock gets simulation testing in the style of TigerBeetle's VOPR, in two parts:
  • hostsim, a package in the host module: a simulated network of hosts. Each host hands out a Host's Listen and Dial, and a filesystem confined to a directory of its own. Its connections behave as TCP's do, every chunk waits on its link until a driver delivers it, and its faults are only those TCP can produce.
  • sim, a new nested module: the harness. It runs a whole system of bedrock.App values, each a node on a hostsim host, in one testing/synctest bubble. It owns the seed, the plan (the swarm, the workload, and faults and lifecycle events at fake times), the delivery of every chunk and the checks.
• It is mechanism without policy, as fit, circuit and retry are. bedrock ships no models. What a service promises, what its dependencies can do and how long its recovery may take belong to the framework and the App. humus is the first framework to build on it (z5labs/humus#1277), and the command-line framework planned in Proposal: a Host shared by frameworks built on bedrock #614 can be the second.
• Determinism comes from the seam, on the stock toolchain. The harness waits for quiescence, sorts the pending events, picks one by seed and gives each a distinct seeded delay. Goroutines inside a node interleave freely, because Go can't seed that without a forked runtime, source translation or a hypervisor. So every run measures whether that freedom leaked into its history, and labels each failing seed "replays exactly" or "replays k of n".
• bedrock's testing claim changes. "It does not claim deterministic simulation testing" becomes a three-part claim that every run measures: the plan is exact, the history is exact when the self-check says so, and interleaving inside a node isn't reproduced.
• retry's jitter can be seeded. Exponential seeds a generator of its own from crypto/rand when it's built, so testing/cryptotest seeds it in a simulation. Its signature doesn't change (open question 3).
• It reports through go test: a seed per subtest, printed with a replay line before the run, eight failure classes, a trace in T.ArtifactDir(), and Go's fuzz engine to store and shrink inputs.
• It lands in two phases: the harness, with hostsim delivering at once; then the network's seeded schedule, its faults and crashes.

func TestEcho_simulation(t *testing.T) {
	sim.Run(t, func(r *sim.R) {
		echo := r.Node("echo", func(h *hostsim.Host) bedrock.App {
			return echoApp{Listen: h.Listen}
		}, sim.Replicas(1, 3))
		m := newEchoModel()
		sim.Workload(r, m, send)
		r.Check(m)
		r.Lifecycle(echo, sim.Crashes(), sim.Restarts())
	})
}

Background

TigerBeetle's simulation testing

The VOPR runs a whole TigerBeetle cluster in one process, on one thread, as a function of a seed. z5labs/humus#1277's Background gives the full account. The essentials:

• A run is a function of its seed and its commit. CI uses the commit SHA as the seed, so every run differs and every one can be replayed.
• Every source of nondeterminism sits behind a seam the harness owns: time, disk, network, the IO loop, process lifecycle and randomness. Time is ticks, and the simulator never sleeps.
• Faults are realistic, and stay inside a written envelope, so a failure means a bug and not an impossible scenario.
• Safety is checked on every tick. Liveness is checked after healing a core, within a bound.
• Each seed draws its own swarm of topology, fault rates, latencies and workload weights.
• The cut is above TCP. TigerBeetle's real TCP code never runs in the VOPR, which works because its protocol tolerates any message fault.

bedrock's engineering rules already come partly from TigerStyle (doc.go). Deterministic simulation testing is the one practice they say bedrock doesn't claim.

What Go gives, and what it doesn't

testing/synctest gives a bubble with a fake clock. The clock advances only when every goroutine in the bubble is durably blocked. The bubble also detects deadlocks and goroutines left behind. Everything made inside the bubble runs on that fake clock, retry's waits and circuit's clock too. The cost follows events, not simulated time: 24 simulated hours of heartbeats took 25 ms of wall time.

A bubble doesn't order goroutines. They run on the normal scheduler across every P. Timers due at the same instant are randomized on purpose (golang/go#73876), and the Go team declined a reproducible bubble scheduler (golang/go#67434, golang/go#73850). In experiments on Go 1.27:

• Events at distinct fake instants gave 1 history hash in 500 runs, under every setting.
• Same-instant timers, select and map order gave a different history every run.
• Channel hand-offs gave 1 hash in 500 runs at GOMAXPROCS=1, 310 in 500 at 32, and 200 in 200 under -race.

Every route to exact interleaving replay in Go leaves the stock toolchain, or imposes a programming model on the code being tested:

• a forked runtime: Polar Signals, and proposal: runtime: add option to specify seed for globalRand golang/go#67690;
• source translation: gosim, whose hooks are per Go release and don't support 1.27;
• a cooperative scheduler that the code is written against: Temporal, Service Weaver;
• a deterministic hypervisor: Antithesis, which is commercial.

An App is ordinary Go on the stock toolchain. So this proposal builds on synctest, and makes determinism a property the harness checks rather than one the runtime enforces.

What bedrock has today

• An App runs whole in a test. It reaches its process only through bedrock.Process, and bedrocktest's Process starts no goroutines, so a whole App runs inside a bubble.
• The host module hands an App the network and the filesystem. hosttest's network starts no goroutines and runs inside a bubble, and its conformance table holds it to real loopback sockets.
• retry and circuit take no injected clock, so they run on synctest's.
• The testing claim stops short. Testing claims says: "It does not claim deterministic simulation testing. The Go scheduler, the network and the filesystem are not simulated, so a run cannot be replayed from a seed."

What's missing is everything that makes a test a simulation:

• a seed;
• a swarm;
• a workload drawn from the seed;
• faults and lifecycle events on a seeded schedule;
• a network whose deliveries the harness orders;
• a history to check determinism against;
• a liveness phase;
• a way to replay and report a failing seed.

Why here, and not in humus

z5labs/humus#1277 put the harness and the network in humus. Neither depends on humus:

• The nodes are bedrock values. #1277 runs each node as a bedrock.App on a bedrocktest Process. Its start, SIGTERM, crash, kill and restart are defined on App, Process and Host. Its seed tree, swarm, driver loop, self-check, run modes, failure classes, trace and replay line need only the standard library.
• The network is a Host. #1277's network "shares humustest's internals", and those internals moved here in Proposal: a Host shared by frameworks built on bedrock #614: hosttest is humustest's network, ported. An internal package can't be shared across modules, so a network in humus would copy hosttest's address rules and its conformance table. The copies have already drifted. On 2026-10-20 humus replaced net.Pipe in humustest, so that a TLS connection closes as it does on a real socket. That was two days after hosttest was ported, and hosttest still uses net.Pipe.
• A second framework needs it. The command-line framework planned in Proposal: a Host shared by frameworks built on bedrock #614 and Proposal: certificate helpers shared by frameworks built on bedrock #621 runs Apps that dial more than they serve. Simulating them through humus would link in the OpenTelemetry SDK, every OTLP exporter and grpc-go, as Proposal: a Host shared by frameworks built on bedrock #614 found.
• The precedent is fit, circuit, retry and host (Feature request: request-scoped failure injection (Netflix FIT) on OpenTelemetry baggage #578, A circuit breaker should be provided by bedrock #595, A retry package owned by bedrock #606, Proposal: a Host shared by frameworks built on bedrock #614). bedrock ships the model, and the framework ships the wiring. Here bedrock ships the harness and the network, and each framework ships the models of its own packages.

Proposed API

The sketches below show the shape and the role of each part. Their exact signatures come from the prototype (open question 8).

Modules and packages

Package	Module	Role
host/hostsim	host	The simulated network. Hosts joined by directed links, connections with TCP's semantics, deliveries held for a driver, routed names, the faults TCP can produce, and scripted controls. Each host's filesystem is hosttest's, in a directory of its own.
sim	github.com/z5labs/bedrock/sim, new	The harness. The run and its seed, the Source and its feeds, the seed tree, the swarm, nodes and lifecycle events, the driver loop, the workload driver, the checker interface, the liveness phase, the trace and history hash, the failure classes, the run's identity and the replay line.
retry	retry	Exponential seeds its own generator from crypto/rand (open question 3).

• hostsim's library code imports only the standard library and the host module's own packages, as hosttest's does. It shares hosttest's address rules through an internal package of the host module.
• sim's library code imports only the standard library, the bedrock module and the host module. It's the first nested module that requires the bedrock module (conflict C5). It uses bedrock and bedrocktest from the one, and host, hosttest and hostsim from the other.
• Names: labels and attributes are sim.seed, sim.node and sim.class, and a sweep's seeds come from BEDROCK_SIM_SEEDS (open question 4).

A simulation test

An App's author writes four things:

• an operation vocabulary of typed Go values, each acting on the system through its public interfaces;
• a model of what the system promises;
• which of their framework's models apply, if their framework ships any;
• optionally, an inspection function over their store's seam.

var send = sim.Operation[*echoModel]{
	Name: "send",
	Run: func(ctx context.Context, op *sim.Op, m *echoModel) error {
		msg := op.ID()
		reply, err := exchange(ctx, op.Dial, "echo:7", msg)
		m.Sent(msg, reply, err) // an error here is an indeterminate outcome, not a failure
		return nil
	},
}

The workload runs on a host of its own, and op.Dial is that host's Dial, so its connections cross the network like any other. There is no client helper, as #614 decided for hosttest.

sim.Run gives each seed one bubble and one subtest. sim.Fuzz runs the same body on Go's fuzz engine, so its corpus stores the failing inputs and its minimizer shrinks them:

func FuzzEcho_simulation(f *testing.F) {
	sim.Fuzz(f, simulateEcho)
}

A scenario test is the same harness with a fixed seed and scripted events:

r.At(2*time.Second, sim.Restart(echo))
r.At(2*time.Second, sim.SilentConn("workload", "echo:7"))

hostsim

package hostsim // github.com/z5labs/bedrock/host/hostsim

// Network is a simulated network of hosts. It starts no goroutines, and it
// delivers nothing by itself: every connect, every chunk written to a
// connection and every close waits until its driver delivers it. Build it
// inside the testing/synctest bubble that uses it.
type Network struct{ /* unexported fields */ }

func New(tb testing.TB, opts ...Option) *Network

// Host returns the host named name, and adds it on first use.
func (n *Network) Host(name string) *Host

// Route makes a name reach a set of hosts, one host per connection as a
// Kubernetes ClusterIP does. The set can change over time.
func (n *Network) Route(name string, hosts ...string)

// Pending returns what waits to happen, in a canonical order that doesn't
// depend on scheduling. Deliver makes one of them happen.
func (n *Network) Pending() []Event
func (n *Network) Deliver(e Event)

// Apply starts a fault or a scripted control, and Heal ends it.
func (n *Network) Apply(c Control)
func (n *Network) Heal(c Control)

// Host is one host on a Network. A framework builds the Host it hands an
// App from its Listen, Dial and FS, as it does from hosttest's.
type Host struct{ /* unexported fields */ }

func (h *Host) Listen(network, address string) (net.Listener, error)
func (h *Host) Dial(ctx context.Context, network, address string) (net.Conn, error)
func (h *Host) Listening(address string) <-chan struct{}

// FS is hosttest's filesystem, in a directory of the host's own, and it
// refuses every call once the host is fenced.
func (h *Host) FS() *FS

// Fence closes every listener and connection the host holds, and refuses
// its later Listen and Dial calls and its file operations.
func (h *Host) Fence()

sim

package sim // github.com/z5labs/bedrock/sim

// Run runs body once for each seed, each in a subtest and a
// testing/synctest bubble of its own.
func Run(t *testing.T, body func(r *R), opts ...Option)

// Fuzz runs body on Go's fuzz engine, with the fuzz input as the run's
// decisions.
func Fuzz(f *testing.F, body func(r *R), opts ...Option)

// R is one run: one seed and one bubble.
type R struct{ /* unexported fields */ }

func (r *R) Network() *hostsim.Network

// Node adds a name to the network, with one or more replicas behind it.
// Each replica is a host and a bedrocktest Process, and app builds its App
// at every start.
func (r *R) Node(name string, app func(h *hostsim.Host) bedrock.App, opts ...NodeOption) *Node

func (r *R) Check(c Checker)
func (r *R) Lifecycle(n *Node, gens ...Generator)
func (r *R) At(d time.Duration, e Event)

// Eventually is a liveness property: after the core heals, progress must
// return true within bound, in fake time.
func (r *R) Eventually(name string, bound time.Duration, progress func() bool)

// Errorf reports a violation from any goroutine in the bubble.
func (r *R) Errorf(format string, args ...any)

// Rand returns the stream of the seed tree at label.
func (r *R) Rand(label string) *Rand

// Workload drives ops against m, each started as a harness event.
func Workload[M any](r *R, m M, ops ...Operation[M])

// Checker judges a run. Step runs on the driver goroutine after every
// event, and End after the liveness phase.
type Checker interface {
	Step(r *R) error
	End(r *R) error
}

// Source is where a run's decisions come from, with the method set of
// math/rand/v2's Source.
type Source interface{ Uint64() uint64 }

func PCG(seed uint64) Source // a seeded PCG, for sweeps and fleets
func Bytes(b []byte) Source  // fuzz bytes, TigerBeetle's "finite PRNG"

Node options include Replicas(min, max), Args, Env and Files, which build each replica's bedrocktest Process and filesystem. The events and their generators are in Nodes and lifecycle.

One run

1. The seed comes in, from BEDROCK_SIM_SEEDS or as a fuzz input. Before the run, the harness writes one unbuffered line to the test binary's standard error, with the seed and the replay command. It also sets a sim.seed pprof label and emits T.Attr. Each survives a different failure: a panic, a -timeout kill, or -json output.
2. Crypto is seeded with cryptotest.SetGlobalRandom on the outer t (open question 2). One TLS 1.3 transcript came out of 30 processes.
3. The bubble starts. The network, and everything the test makes for the nodes, is made inside it.
4. The swarm is drawn, each dimension from its own stream. Any dimension can be overridden.
5. Nodes start. Each replica runs its App on a bedrocktest Process with a root context of its own, on its own hostsim host, under a sim.node pprof label.
6. The driver loop runs:
   1. synctest.Wait();
   2. collect the pending events: the network's deliveries, operations due, and faults and lifecycle events due;
   3. sort them canonically and pick one by seed;
   4. advance fake time by a distinct seeded delay;
   5. apply the event;
   6. append it to the history and its hash.
7. Safety checks run throughout. A violation goes from any goroutine through r.Errorf to t.Errorf, with its class.
8. The liveness phase heals a core and waits for progress within each bound.
9. At the end:
   • synctest's leak check runs;
   • each node's labelled goroutines must reach zero;
   • the class is emitted;
   • on a failure, the trace is written to T.ArtifactDir();
   • the history hash is printed.
10. A wall-clock watchdog outside the bubble classes a real-time hang as inconclusive, never as a pass or a fail.

Determinism at the seam

Source of nondeterminism	Seam	Status
Time, timers, contexts and deadlines, in dependencies too	synctest's fake clock	Covered for everything made in the bubble
The network	hostsim, through the Host a framework hands its App	Covered for code that listens and dials only through its Host
Files	hostsim's FS, which is hosttest's: real files in each host's directory	Real. File-change events are keyed on content, because ModTime is real time
Arguments, environment, streams and signals	bedrock.Process	Covered: nodes run on bedrocktest Processes
Event order between nodes	the harness's seam scheduler	Seeded
Timers at the same instant	a distinct seeded delay for each harness event	Removed at the seam. Ties inside dependencies stay random
Goroutine order inside a node, select, map order	none on stock Go	Free. The self-check measures it, and -race explores it
Crypto randomness: TLS, keys, rand.Text	cryptotest.SetGlobalRandom	Seeded, process-wide, so seeded runs are serial
retry's jitter	a generator seeded from crypto/rand (open question 3)	Seeded with crypto
Other math/rand/v2 top-level draws	none	Unseedable. Canonical ordering absorbs them unless they change what crosses the seam

Four more rules make the seam hold:

• Decisions are keyed on identity, never on call count. Faults and latencies are keyed on an operation's ID, a node pair or a listener's name, and installed at fake times. In #1277's research, a fault PRNG drawn in call order fired a different number of times on each run.
• The seed tree hashes labels. A child seed is H(root, label path), so adding a component or a dimension doesn't reshuffle the other streams. Draws stay on the driver goroutine. The harness turns a Source's output into values with its own functions, because math/rand/v2 promises no stable output for Rand's methods across releases.
• Workload starts are harness events. Over real crypto/tls and HTTP/2 with the seam scheduler, a run gave 1 trace per seed in 300 runs at GOMAXPROCS 32, 4 and 1, bytes included. When the workload's own goroutines started 10 requests at once, it gave 284 to 292 distinct traces in 300.
• The self-check runs on every run. Each run records its history and hashes it. Repeats of a seed compare hashes, and a mismatch names the first event that diverged. TigerBeetle has no such check. In Go it is essential, and one repeat isn't enough (see Testing), so a failing seed runs several times.

The reproducibility claim: two run modes

	Explore mode	Replay mode
Flags	-race, several Ps	-cpu=1, no -race, GOGC=off, a fresh process, -count=1, the recorded toolchain through GOTOOLCHAIN
Finds	Scheduling bugs inside a node	What the seam schedule exposes
Reproduces	Statistically: the seed runs k times	Exactly, when the run has no timer ties and no select with several ready cases

A tie-free run replayed exactly across processes, under -cover and -gcflags=all=-N -l, and on go1.26.5, 1.27.0 and 1.27.2. dlv test … -- -test.run 'FuzzX/<id>' stopped inside a bubble. A breakpoint stops real time, not fake time.

The harness claims three things, no more:

1. The plan is exact. Given the seed and the run's identity, it reproduces the swarm, the workload, and every fault and lifecycle event with its fake time, whatever the interleaving.
2. The history is exact when the self-check says so. The history is the harness-owned events: deliveries, operation starts and results, faults and checks. It reproduces whenever each node's reaction to an event doesn't depend on goroutine order.
3. Interleaving inside a node is not reproduced. Bugs there are found in explore mode and reproduced statistically. A scripted control then pins each one as a deterministic scenario test.

This claim replaces bedrock's Testing claims (conflict C1).

The run's identity is printed with every seed:

• the seed;
• the commit: GITHUB_SHA, plus the head and base SHAs on a pull request, or git rev-parse and whether the tree is dirty, because test binaries carry no vcs.* build info;
• the toolchain;
• GOOS, GOARCH and GOAMD64;
• the build flags, -cpu, GOGC, GODEBUG and -run.

Nodes and lifecycle

Event	In process
Start(n)	App.Run on a fresh bedrocktest Process, with its own root context, on the replica's host
Signal(n, sig)	Delivers sig through bedrocktest's Signal. If the App holds no registration for it, a real process would take the default action, so the harness crashes the replica, where bedrocktest would fail the test
Crash(n)	Fence, then wind down. Fence the replica's host: every listener and connection closes, and later Listen, Dial and file calls are refused. Stop reading the Process's streams. Then cancel its root context, and discard its exit status
KillAfter(n, d)	SIGTERM, then a crash at +d, as terminationGracePeriodSeconds does
Restart(n)	A crash or a SIGTERM, then a start on the same host
Rollout(n, app, ready)	For each replica: take it out of its route, SIGTERM it, start it (possibly with another App, which models version skew), wait until ready returns nil, and add it back
Partition(n, d)	The network clogs every connection of the replica's host

Crashes(), Restarts(), Rollouts(ready) and Partitions() are generators that draw these events from the swarm for r.Lifecycle. A rollout's readiness is always the caller's: bedrock doesn't know what ready means for an App (open question 7).

Go can't kill a goroutine, so an in-process crash leaks: in #1277's research, a Kafka Processor made its side effect 0.5 s after its consumer "crashed". So an edge counts as crashed only if the harness revokes it. Each replica has a crash context that ends at its crash, for the test's own in-memory stand-ins to consult, so a zombie's calls fail. A zombie that can still reach an unfenced edge is reported as a partial partition, not a crash.

A restart isn't a fresh process. Package-level state survives it, and the docs say so.

bedrocktest doesn't change. The harness builds each Process with NewProcess and runs the App on a context of its own, and it uses Notified to see whether a signal is registered.

The network

The cut is at net.Conn, through the Host. The real code runs from the connection up: TLS, HTTP/2, gRPC and every client library. humus's recent bugs live there: an HTTP/2 write that fails because the client went away (z5labs/humus#1144), a rollout that opens a gRPC client's breaker (#1176), and a server that cancels streams before its GOAWAY (#1254). A RoundTripper, a Handler or a library's API stay useful for unit tests of logic above the transport, but they aren't transport simulation.

The network is new, in the host module. hosttest's documented semantics stay as they are (conflict C4, open question 5). The network is made of hosts joined by directed links. The driver draws each link's latency and jitter bound per seed, and the network keeps each link's buffer. It starts no goroutines. An 11-minute partition took 17 to 37 ms of wall time.

Its semantics are TCP's, each checked by host's conformance table against loopback:

• the first write after a FIN succeeds, and the next fails with EPIPE;
• closing with unread data sends an RST, and the peer reads what arrived, then gets ECONNRESET;
• a connect completes in the backlog before Accept;
• errors are *net.OpError wrapping *os.SyscallError.

hosttest's Dial returns only once Accept takes the connection, so it never leaves a connection half-open the way TCP does. That is why #1254 failed 8 times in about 33 container runs, and never on humus's in-memory network.

Faults are only those TCP can produce: stalls (clogs and partitions, which queue bytes and lose none), resets, RST on close, silent peers, silent connections, refused or black-holed dials, paused listeners, slow readers, short reads, and a partial write followed by an error. It never injects loss with the stream going on, duplication, reordering, truncation followed by more bytes, or corruption. Vortex removed its truncation fault after it caused a false failure.

Protocol faults come from real peers through their real APIs. GOAWAY comes from http.Server.Shutdown, RST_STREAM from a client that cancels, and an error status from a handler. Nothing comes from a frame-level fake peer.

Zero latency stays in the swarm. With 1 ms of modelled latency, #1144 never happened, because fake time froze while the busy handler ran.

Scripted controls are typed values. Each has a "hold until" form, and each is recorded in the trace:

• route a name: one host per connection, a chosen host, and changes over time;
• hold a direction, or hold after N bytes, which models "request processed, response lost";
• partition;
• reset by predicate;
• open a silent connection;
• refuse or black-hole dials;
• pause a listener;
• throttle a reader.

humus hand-builds three of these in its tests today: held reads, ClusterIP routing and a handshaking connection.

Workloads, models and checks

One decision Source has three feeds:

• a seeded PCG, for sweeps and fleets;
• fuzz bytes, TigerBeetle's "finite PRNG", for go test -fuzz with its corpus and shrinking;
• a three-line adapter to rapid's bit stream, if the App's author prefers rapid.

bedrock ships no models. The harness runs checkers and liveness properties, and the framework and the App supply them:

• A framework's models state what its packages promise, such as a queue's delivery guarantee or a server's drain order, with liveness bounds computed from the same settings its nodes run with.
• The App's model states what the App promises. It gets each operation's outcome, and treats an error as an indeterminate outcome, Jepsen's :info, which a later read must resolve.

Oracles don't depend on order wherever the system allows concurrency. They compare multisets with counts, per-key order, final states and per-operation outcomes. Even at GOMAXPROCS=1, six spans ended in 249 distinct orders in 300 runs.

Visibility has three levels:

• the framework's internals, through hooks of its own;
• the App's state, through its seams only;
• dependencies, through what their fakes expose.

A panic fails a run. bedrock never recovers a panic, so a panic in a node crashes the test binary, as it would crash the process. A framework that recovers panics, as humus does in its handlers, decides whether a recovered one fails a run.

Liveness

• The core is the replicas that are up and stay up, each dependency the property needs, and a healed network between them. A dependency outside the core stays down for good, and the property becomes a fail-fast one.
• Bounds are inputs. A loop with capped backoff and unlimited attempts separates "not yet" from "never" after a bound computed from its settings, such as retry's Exponential maximum delay. The framework or the App computes each bound from the settings its nodes run with, and passes it to Eventually.
• Readiness is the startup half, within a bound. Workload progress is the recovery half.

The swarm

• The harness varies what an operator or the environment varies: counts, timings, fault rates, and settings documented as tuning. The framework says which of its settings those are. It doesn't vary what defines the App's meaning.
• Ranges include zero: one replica, zero latency, no faults.
• Biased churn regions find more. In #1277's research, a churn swarm found a Kafka offset rewind in 1.05% of seeds, against 0.009% for the default swarm, about 115 times as often.
• Each dimension has its own stream, so knocking one out doesn't reshuffle the rest, and each can be a separate fuzz argument for the mutator to move.

Failures, traces and minimization

Class	How it shows	Kills the binary?
correctness	t.Errorf from any goroutine in the bubble, then T.Attr("sim.class", …)	no
panic	a runtime panic	yes
deadlock	synctest's "all goroutines in bubble are blocked"	yes
leak	synctest's "blocked goroutines remain"	yes
liveness	the harness's bound check, in fake time	no
inconclusive	a real-time hang, caught by the watchdog or -timeout	yes (timeout)
replay diverged	the self-check's hash mismatch	no
excused	a dependency or a stated precondition broke its guarantee	no

Three classes kill the process, so a fleet runs seeds in supervised processes: the fuzz engine's workers, or a runner that reads test2json.

A trace has one line per state change: the fake elapsed time, the node, the component's name, the event, and key=value pairs. A framework names its own components. The trace is written to T.ArtifactDir() only on a failure, and its last lines are printed.

Minimization.

• A fuzz target whose input is a choice sequence gets real shrinking from Go's engine. A 53-byte failing run became 5 bytes. Reruns gave 4, 15 and 5 bytes, so the result is small, not minimal.
• A uint64 seed is recorded, never shrunk. Coarse minimization covers it: keep the fastest failing seed per signature, truncate the step budget, and knock out swarm dimensions.
• Shrinking is unreliable under free interleaving, because a shrunk input may not reproduce.

A fixed seed's regression form is a corpus entry that lands with the fix. A hand-written scenario test replaces the entry once it stops pinning the failure.

What an App's code must do

1. Reach the network and files only through the Host its framework hands it. Its own clients take the Host's Dial. Peers are addressed by name. Code doesn't rely on *net.TCPConn.
2. Make everything inside the bubble: timers, contexts, channels, WaitGroups and certificates. No package-level timers or contexts, and no pinned tls.Config.Time or Rand.
3. Hold no mutex across a channel wait or a sleep, because a mutex wait isn't durable blocking.
4. Honour the context it's given, or the leak checks fire.
5. Put stores behind their own seams, with in-memory models whose fault decorators return the real client's error shapes, and container tests that check the models.
6. Make identities unique and observable, for example an ID in each message, and make processing idempotent where the contract says so.
7. Expect no package-level state to reset at a restart. For crash fidelity, every side effect crosses a revocable edge: the Host, or the replica's crash context.
8. Read files rotation-safely: by content, tolerating a torn certificate and key pair, keeping the last good value.
9. Run seeded tests serially. Crypto is seeded process-wide, so seeded tests use no t.Parallel.
10. Treat math/rand/v2 draws as free nondeterminism. The self-check reports when they leak into the history.

These rules are documented, not enforced by an analyzer. The watchdog and the self-check report violations at run time.

retry's jitter

Exponential draws from math/rand/v2's top-level functions today (schedule.go), which nothing can seed. #606 settled that "tests pass their own schedule". In a simulation the framework builds the schedule inside its own clients, so the test has none to pass (conflict C3).

As recommended in open question 3, Exponential reads a seed from crypto/rand when it's built, and draws from a generator of its own, under a lock:

• The schedule stays safe for concurrent use, and one schedule is still shared by every call.
• Its signature and its distribution don't change.
• In a simulation, crypto is seeded, so a schedule built in the bubble draws the same delays for the same seed. The draws of concurrent calls still follow scheduling, which the self-check measures.

What a framework adds

• A node from its own App type, whose Host it builds from a hostsim host's parts. humus runs an Init.
• The models of its packages, their faults, and liveness bounds computed from its settings.
• Hooks into its internals, checks of its own, and its telemetry as an event feed.
• Its dependencies' fakes as nodes, such as franz-go's kfake listening on a hostsim host.
• Its CI: fleets of fresh seeds, and how a failing seed is reported.

humus does all of this in z5labs/humus#1277.

Design decisions

• Mechanism without policy. bedrock ships the harness and the network, and no models, envelopes or default checks, as fit ships no codecs, circuit no policy and retry no classifier. Every range, rate, budget and bound is a value in test code that can be overridden.
• The harness is a nested module of its own. The root module stays exactly the process model. The harness requires bedrock because its nodes are Apps, and host because its network is a Host. Neither host nor the root module requires it.
• The network is in the host module. It's a third implementation of the Host contract, next to OS() and hosttest. It shares hosttest's address rules and gives each host hosttest's filesystem, and host's conformance table runs on all three networks. In any other module, it would copy both.
• A new network, not a changed hosttest. hosttest delivers at once, and hostsim holds every delivery for its driver. Tests that rely on hosttest's semantics keep their meaning.
• synctest and the stock toolchain. Exact interleaving replay would make every App's author adopt a forked runtime, a source translator or a hypervisor, or write against a cooperative scheduler. A narrower claim, measured on every run, costs them nothing.
• Two run modes. Replay mode alone misses bugs that need several Ps by construction: humus's #959 failed only at -cpu 2 and 4. Explore mode alone can't replay anything.
• The cut is at net.Conn. TigerBeetle cuts higher because its protocol tolerates any message fault. HTTP/2, gRPC and TLS don't, and their shutdown paths are where bugs hide.
• Faults and latencies are keyed on identity. Call order is scheduling, and scheduling isn't seeded.
• Pause is a partition, named as one. All nodes share one bubble clock, and Go can't suspend some goroutines and not others.
• Clock skew stays out of the whole-system run. One bubble has one clock. Certificate windows and token leeway are unit tested with an injected time, as TigerBeetle tests skew in its clock's unit tests.
• The scripted controls are exported. An App's author needs them to pin their own failures.
• The App rules are documented, not enforced by an analyzer. bedrock enforces its own rules with tools in CI, but these are rules for the code under test, and an analyzer of them would have false positives. The watchdog and the self-check catch violations at run time.
• A seed is valid only with its commit and toolchain, as in TigerBeetle. The determinism canary runs on every Go bump.

Conflicts with prior decisions

Each conflict names the decision, where the design bites, and the case for it.

• C1. bedrock's testing claim. Testing claims says bedrock "does not claim deterministic simulation testing", and the TigerStyle list repeats it. Case: its reasons stay true for the scheduler, which isn't simulated, and for the filesystem, which stays real. The network becomes simulated. The claim becomes the three-part claim, which every run measures, and the TigerStyle entry becomes "adapted" (open question 1).
• C2. The boundary lint. forbidigo bans os.Getenv and os.Stderr in every module, tests included, outside the OS edge (.golangci.yaml). The harness reads a sweep's seeds and the run's identity, such as GITHUB_SHA, from the test binary's environment. It writes its seed line unbuffered to the test binary's standard error, which is the only output that survives a panic. Case: one file of sim, its edge, joins the exceptions, as os.go and bedrocktest's conformance test have. It touches the test binary, never a node's process, and every node still runs on a bedrocktest Process (open question 6).
• C3. A retry package owned by bedrock #606's "tests pass their own schedule" (its proposed API, section 4). A simulation runs a whole App, whose framework builds the schedule inside its clients, so the test has no schedule to pass. Case: open question 3 seeds the schedule through testing/cryptotest, which bedrock's Beyond the boundary already names as where a test's randomness comes from.
• C4. Proposal: a Host shared by frameworks built on bedrock #614's hosttest. Proposal: a Host shared by frameworks built on bedrock #614 made hosttest the Host of a test, with the rules Proposal: humustest — whole-service testing on top of bedrocktest humus#633 settled, among them that Dial refuses rather than waits. Its connections are net.Pipes, and its Dial returns once Accept takes the connection. A second test network with other semantics is two networks to keep. Case: hostsim serves a different purpose, since every delivery waits for its driver. hosttest stays as it is, and the conformance table lists each place hosttest differs from loopback, rather than hiding it (open question 5).
• C5. Nested modules don't require the bedrock module. fit, circuit, retry and host each never import it, so "a framework that wants only the Host gets nothing else" (Proposal: a Host shared by frameworks built on bedrock #614). The harness must, because its nodes are Apps. Case: whoever uses the harness runs Apps anyway. host and hostsim stay free of it.
• C6. bedrock's tests run in parallel, and bedrock has no global state. bedrocktest's docs say a test on a fake Process "can run in parallel". cryptotest.SetGlobalRandom is process-wide and refuses t.Parallel. Case: the global lives in test binaries only, and bedrock's doc already names cryptotest. Tests that don't run the harness stay parallel (open question 2).

Testing

The evidence

No prototype of the harness was built. The research behind z5labs/humus#1277 ran throwaway experiments on Go 1.27, which aren't kept. These are the results this proposal rests on:

• Seam scheduling works on toy nodes and over real TLS and HTTP/2.
  • A seeded scheduler at the seam gave one trace per seed in 300 runs at GOMAXPROCS=32 and under -race.
  • Over real crypto/tls and HTTP/2, it gave 1 trace per seed in 300 without -race, and 2-3 per seed with it.
  • It wasn't measured with real services on a Host's network.
• The self-check catches leaks. A deliberately racy node leaked into the history 11 times in 300 runs (285 in 300 under -race). Running each seed twice caught it in 33 of 100 processes.
• The close model changes bug rates. humus's #1144 happened 18 times in 100 with humustest's close, and once in 100 with TCP-like close, under -race on one CPU. With 1 ms of latency it never happened.
• A seeded workload found a real bug. A seeded workload of produce, start, stop and sleep steps drove humus's Kafka consumer on kfake, with the fix for humus's #757 reverted. It found #757's offset rewind, from 23 to 5. Seed 329 failed 20 times in 20.
• Lifecycle in one bubble is cheap. Fifty restarts took 0.05 s.
• Replay holds across tools. A tie-free run replayed exactly across processes, under -cover, under -gcflags=all=-N -l, on three toolchains and under Delve.
• Shrinking works but isn't stable. A choice-sequence fuzz input shrank from 53 bytes to 5. Reruns gave 4, 15 and 5.
• A swarm fuzz of a trivial bubble ran 660,000-720,000 executions a second.

How it is tested

• Coverage. Statement coverage of hostsim and sim is 100.0%, as for every package of every module, and the coverage gate covers the new module.
• Conformance. host's network table runs on hostsim and hosttest and, behind the conformance build tag, on loopback. It gains RST on close with unread data, the write after a FIN, a connect in the backlog, and *os.SyscallError. Each place hosttest differs from loopback is a listed exception of the table, and a fenced host is checked against a killed loopback process.
• Canaries:
  • a planted-bug package that the harness's own tests drive as a subprocess, asserting the seed line, the class, the traceback label, a shrunk input under a bound, and that the replay reproduces;
  • a determinism canary: a target known to tie must be reported "replay diverged".
• Examples. sim's package doc has a tested example on bedrock alone, the echo system above.
• Correct code must pass every check. The first exit-code oracle in #1277's research reported 26 false failures on correct code.

Phases

One proposal. Its stories are filed in order, each blocked by the one it needs.

Phase 1: the harness, and the network delivering at once.

• Builds:
  • the new module, with seed plumbing, the Source and its feeds, the seed tree and the swarm;
  • the run's identity, the failure classes, the watchdog, the trace, the history hash and self-check, and both run modes;
  • seeded crypto, and retry's jitter;
  • hostsim's hosts and routed names, with TCP's semantics and the conformance table's new rows. The driver delivers every pending chunk at once, in canonical order;
  • nodes with start, signal, restart after SIGTERM, and rollout;
  • the workload driver, with workload starts as harness events, checkers, and the liveness phase;
  • the planted-bug and determinism canaries, and the example.
• First users: humus's queue/kafka and lifecycle models, in #1277's phase 1.

Phase 2: the network's schedule, faults and crashes.

• Builds:
  • seam scheduling of deliveries, with seeded link latencies and a distinct delay for each event;
  • faults, and the scripted controls;
  • crashes as a fence and a wind-down, the crash context, kill-after-grace, and partitions.
• First users: humus's gRPC and HTTP models, and its wallet and shoppingcart walkthroughs, in #1277's phase 2.

Phase 1 puts each node on a hostsim host from the start, so the signature that hands an App its host doesn't change in phase 2.

Migration from today

• bedrock. The change is additive, except for retry's draws:
  • A sixth module. It needs CI build, vet, test and lint steps, the coverage gate, a depguard rule, and the CLAUDE.md commands and package layout. tag-modules.sh already finds every nested go.mod, so releases need nothing.
  • host gains hostsim, and its conformance table grows.
  • retry's Exponential draws from a generator seeded from crypto/rand. Its signature and distribution don't change.
  • doc.go's Testing claims and TigerStyle list take the three-part claim, and the README gains a section.
  • forbidigo excepts sim's edge.
  • example/httpserver doesn't change, because the root module doesn't require nested modules.
• humus. humus adopts this in revision 1 of Proposal: simulation testing humus#1277. Nothing in this proposal changes humus. humus doesn't have to adopt the host module first: humus.FS is an interface, which hostsim's FS satisfies, so humus builds a humus.Host from a hostsim host's parts.
• The command-line framework can simulate its commands as Apps, with the servers they call as nodes.

Open questions

1. Is the three-part reproducibility claim acceptable as bedrock's testing claim? As proposed: the plan is exact, the history is exact when the self-check says so, and interleaving inside a node is statistical, with explore and replay modes. The alternatives:

   • require exact interleaving replay, through a forked runtime, source translation or a hypervisor;
   • keep "not claimed", and ship the harness without a claim.

   Recommendation: as proposed. The first alternative takes every App off the stock toolchain. The second hides what the self-check measures on every run.

2. Seed crypto with cryptotest.SetGlobalRandom? It is process-wide, forbids t.Parallel, and is unsupported with the FIPS v1.0.0 module.

   • (a) Seed it in every simulation run, so runs are serial.
   • (b) Seed it only in replay mode.
   • (c) Never seed it, and keep TLS bytes out of the history.

   Recommendation: (a). Seeded keys and certificates make traces comparable. They're still made fresh in each run, and nothing outside the run trusts them.

3. How is retry's jitter seeded?

   • (a) Exponential seeds a generator of its own from crypto/rand when it's built. Its signature doesn't change, and cryptotest seeds it in a simulation.
   • (b) Exponential takes a rand.Source. The harness could then give each node a stream keyed by its name, but every caller passes a source, and a framework has to carry it from the harness to its clients.
   • (c) Keep A retry package owned by bedrock #606: draws from the top-level functions. Each framework gives its simulations a seeded schedule of its own, through a hook.

   Recommendation: (a). It changes no signature, and cryptotest becomes the one switch a simulation needs, which is what bedrock's doc already says tests use.

4. Where do the network and the harness live, and what are they called?

   • (a) hostsim in the host module, and sim as a new nested module.
   • (b) Both in the new module, copying hosttest's address rules and its conformance table.
   • (c) The network as options on hosttest, and the harness in the new module.

   The names are open too: sim could be simulation, and BEDROCK_SIM_SEEDS, sim.seed, sim.node and sim.class could change with it. Recommendation: (a), with the names as proposed.

5. A new network, or a TCP-faithful hosttest? As proposed, hostsim is new, and hosttest stays as it is (C4). The alternative makes hosttest TCP-faithful and schedulable, which leaves one network to keep but changes semantics that Proposal: a Host shared by frameworks built on bedrock #614 settled and that tests rely on. Recommendation: as proposed.

6. How does the harness reach the test binary (C2)?

   • (a) An edge file reads the environment and writes the seed line to standard error, excepted from forbidigo.
   • (b) Seeds come only from code and fuzz inputs, and the seed line goes through t.Log, which a panic or a -timeout kill loses.

   Recommendation: (a).

7. Replicas and rollouts in bedrock? A command-line framework has no replicas.

   • (a) In sim, with the caller's readiness check. Routing is hostsim's anyway, and a rollout is a restart of each replica behind a route.
   • (b) Each framework's.

   Recommendation: (a).

8. Prototype phase 1 before the stories are filed? Seam determinism with real services on a Host's network is unmeasured, and the sketches' signatures are unsettled. A prototype of the harness, with the echo example and humus's queue/kafka as its first users, would settle both. Recommendation: yes, posted as a revision here and on Proposal: simulation testing humus#1277 before stories.

Out of scope

• The models, envelopes, hooks, checks and telemetry of any framework.
• Fakes of any dependency's server, and frame-level HTTP/2 or gRPC fake peers.
• Exact interleaving replay: a forked runtime, source translation or a hypervisor (open question 1).
• Filesystem simulation: crash semantics, ENOSPC and EIO, as in Proposal: a Host shared by frameworks built on bedrock #614.
• Clock skew in the whole-system run. It is unit tested.
• A true process pause, which needs real processes.
• ListenPacket, Unix sockets and name resolution, as in Proposal: a Host shared by frameworks built on bedrock #614.
• An analyzer for the App rules.
• Fleets, CI jobs and failure reporting, which belong to each repository.
• Changes to fit and circuit.

References

• humus: Proposal: simulation testing, and humustest
• bedrock: fit, circuit, retry, host and certificate helpers
• TigerBeetle at 0450c32b: TIGER_STYLE.md, the VOPR's docs and src/vortex.zig; and its posts Simulation Testing for Liveness, Swarm Testing Data Structures and A Descent Into the Vörtex
• Jepsen, TigerBeetle 0.16.11
• Go: testing/synctest, testing/cryptotest, and the issues golang/go#67434, golang/go#73850, golang/go#73876 and golang/go#67690
• Prior art in Go: gosim, Polar Signals, WarpStream and Antithesis's deterministic hypervisor