Proposal: simulation testing

[Zaba505]

Important

Revision 1 moves the harness and the simulated network to bedrock, in z5labs/bedrock#628. humus ships humussim, kafkasim, grpcsim and kubesim on bedrock's sim, with its models, hooks, verify mode, telemetry and CI. Questions 1, 2, 3 and 7 move to bedrock, conflict C6 is dropped and C11 moves there, and phases 1 and 2 build on bedrock's.

Note

A new feature for humus. It isn't part of the re-evaluation in Proposal: humus on the bedrock process model, so it has no wave. It builds on accepted proposals: the lifecycle, OpenTelemetry without globals, humustest, queue processing and the Kafka runtime, failure injection and Kubernetes controllers. It works next to Proposal: finding flaky tests.

As a feature request it conflicts with some earlier decisions, among them that humus writes no fakes of other projects' servers. Conflicts with prior decisions names each one, for debate here.

Summary

• humus gets simulation testing in the style of TigerBeetle's VOPR, as a toolkit. End users simulate their own humus services with it, and humus's packages and walkthroughs are its first users.
• A simulation runs a whole system in one testing/synctest bubble: the end user's services, their peers and the dependency fakes, each a node on a simulated network that it reaches through its humus.Host. A harness owns the seed, the plan (the swarm, the workload, and faults and lifecycle events at fake times), the delivery of every network chunk and the checks.
• The real code runs from net.Conn up: TLS, HTTP/2, gRPC, franz-go and client-go. That cut is lower than TigerBeetle's, and keeps in scope the transport and shutdown code where humus's recent bugs were (openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144, grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176, example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254).
• Determinism comes from the seam, on the stock toolchain. The harness waits for quiescence, sorts the pending events, picks one by seed and gives each a distinct seeded delay. Goroutines inside a node interleave freely, because Go can't seed that without a forked runtime, source translation or a hypervisor. So every run measures whether that freedom leaked into its history, and labels each failing seed "replays exactly" or "replays k of n".
• Faults are only those the real medium can produce. Dependency faults come from upstream fakes (kfake, client-go's fakes). Request faults come from the workload, through fault's own edge model. fault, bedrock/fit and bedrock don't change.
• Checks are layered: humus's per-package models, the end user's model, internal hooks, a humus.verify build tag, telemetry and leak checks. Liveness is checked after a healthy core is healed, within bounds computed from the nodes' Settings.
• It reports through go test: a seed per subtest, printed with a replay line before the run, eight failure classes, a trace in T.ArtifactDir(), and Go's fuzz engine to store and shrink inputs. CI keeps a deterministic gate, and fresh seeds run off it.
• It lands in four phases: the harness on Kafka; the network and crashes; dependencies and envelopes; CI and the fleet.

func TestOrders_simulation(t *testing.T) {
	sim.Run(t, func(r *sim.R) {
		broker := kafkasim.Broker(r, "kafka:9092")
		orders := r.Service("orders", app.Init,
			sim.Replicas(1, 3),
			sim.Env(map[string]string{"KAFKA_BROKERS": "kafka:9092"}),
		)
		r.Workload(placeOrder, cancelOrder, produceEvent)
		r.Check(kafkasim.AtLeastOnce(broker, "orders"))
		r.Check(ordersModel)
		r.Lifecycle(orders, sim.Crashes(), sim.Rollouts())
	})
}

Background

TigerBeetle's simulation testing

The VOPR runs a whole TigerBeetle cluster in one process, on one thread, as a function of a seed. Its essentials, from TigerBeetle's source at 0450c32b and its own posts:

• A run is a function of its seed and its commit. CI uses the commit SHA as the seed, so every run differs and every one can be replayed.
• Every source of nondeterminism sits behind a seam the harness owns: time, disk, network, the IO loop, process lifecycle and randomness. The real Replica runs with simulated Storage, MessageBus and Time. Time is ticks, and the simulator never sleeps.
• Faults are realistic, and stay inside a written envelope. A "fault atlas" caps where disk faults may land, from the quorum maths, so a failure means a bug and not an impossible scenario.
• Operating errors are handled, and programmer errors are asserted, which lets fault injection coexist with crashing on a failed assertion.
• The oracles are layered and see inside: dense production assertions, cross-replica checkers, and an exact model of the expected replies.
• Safety is checked on every tick. Liveness is checked after healing a core, within a bound. Liveness mode found a bug that safety mode hid.
• Everything is bounded, and each seed draws its own swarm of topology, fault rates, latencies and workload weights.
• There is no shrinker. About 60 scenario tests and 16 fuzzers share the harness, and a fleet of about 1,000 cores runs fresh seeds continuously.
• Vortex covers what the simulator stubs out: real processes, real TCP through a fault proxy, real kills. It can't replay a failure.

TigerStyle supplies the oracle and the bounds: assert invariants and the negative space, put a limit on everything, handle operating errors and assert programmer errors.

Two findings shape this proposal:

• TigerBeetle simulates no external system. Its one integration, change data capture to AMQP, is unit tested and smoke-tested against real RabbitMQ containers.
• TigerBeetle's cut is above TCP. Its real TCP code never runs in the VOPR. That works because its protocol tolerates any message fault. humus's bugs live in the code that cut skips.

What Go gives, and what it doesn't

testing/synctest gives a bubble with a fake clock. The clock advances only when every goroutine in the bubble is durably blocked. The bubble also detects deadlocks and goroutines left behind. Everything made inside the bubble runs on that fake clock, dependencies too: franz-go, kfake, grpc-go keepalives, net/http timeouts, client-go's clock, the OTel batcher, and bedrock's retry and circuit. The cost follows events, not simulated time. 24 simulated hours of heartbeats took 25 ms of wall time, and an HTTP workload took about 1 s per simulated hour.

A bubble doesn't order goroutines. They run on the normal scheduler across every P. Timers due at the same instant are randomized on purpose (runtime/time.go:711-717, golang/go#73876), and the Go team declined a reproducible bubble scheduler (golang/go#67434, golang/go#73850). In experiments on Go 1.27:

• Events at distinct fake instants gave 1 history hash in 500 runs, under every setting.
• Same-instant timers, select and map order gave a different history every run.
• Channel hand-offs gave 1 hash in 500 runs at GOMAXPROCS=1, 310 in 500 at 32, and 200 in 200 under -race.

Every route to exact interleaving replay in Go leaves the stock toolchain, or imposes a programming model on the code being tested:

• a forked runtime: Polar Signals, and proposal: runtime: add option to specify seed for globalRand golang/go#67690;
• source translation: gosim, whose hooks are per Go release and don't support 1.27;
• a cooperative scheduler that the code is written against: Temporal, Service Weaver;
• a deterministic hypervisor: Antithesis, which is commercial.

humus's end users write ordinary Go handlers on the stock toolchain. So this proposal builds on synctest, and makes determinism a property the harness checks rather than one the runtime enforces.

What humus has today

Most of the parts are already here:

• 1,336 synctest.Test calls in 168 test files.
• humus.Host on every dial and listen. No humus package hard-codes net.Dial, net.Listen, http.DefaultClient or a resolver outside OSHost.
• kfake on the Host.
• fault on bedrock/fit for request-scoped faults.
• orderstatus, shoppingcart and wallet already run their shared cases whole in a bubble. The same cases run on the testcontainers and kind tiers through their Target.

What's missing is everything that makes a test a simulation:

• a seed;
• a swarm;
• a workload drawn from the seed;
• faults and lifecycle events on a seeded schedule;
• models that judge any run;
• a liveness phase;
• a history to check determinism against;
• a way to replay and report a failing seed.

The bugs that got past it

Each of these escaped humus's current tests, or was found only by a flaky CI run or a reproduction someone built by hand:

• #757: a fenced-out Kafka member's late commit moves the new owner's offset.
• #959: a SIGTERM reported as a Service returning.
• #1144: over HTTP/2, a write that fails because the client went away is logged as "response failed".
• #1176: a rollout opens a grpcclient's breaker.
• #1250: a Secret deleted mid-reconcile comes back with the old password.
• #1254: grpcserver cancels streams before the GOAWAY. It failed 8 times in about 33 testcontainers runs, and never on the in-memory network, because humustest's Dial never leaves a connection half-open the way real TCP does.

Testing says which of these the design finds, and which it replays.

Proposed API

The sketches below show the shape and the roles of each part. Their exact signatures come from the phase 1 prototype (open question 11).

Packages

Package	Module	Role
sim	root	The harness. It holds the run and its seed, the Source, the seed tree, the swarm, nodes and lifecycle events, the simulated network and its fault schedule, the workload driver, the checker and reporter interfaces, the liveness phase, the trace and history hash, the failure classes and the replay line. It also holds the models for the root module's packages: the lifecycle, httpserver, httphealth, openapi, httpclient, health and job.
grpc/grpcsim	grpc	Models for grpcserver and grpcclient: how streams end at Shutdown, the breaker's rules, and the rollout promise.
queue/kafka/kafkasim	queue/kafka	kfake as a node. It listens on a node's Host, installs kfake.Fault and ControlKey at fake times, and reads GroupInfo for the commit oracles. It derives the queue model and the Kafka liveness bound from kafka.Settings.
kube/kubesim	kube	client-go's fakes wired into a whole controller, through a client seam that today exists only in kube/export_test.go. It also holds the API-server fidelity reactors (open question 5) and the controller models.
internal/simhook	root	Nil function fields that only the sim packages set. Every humus module can import it, as grpc/grpcauth imports internal/authn today.
build tag humus.verify	all	Expensive invariant checks behind a const, compiled out of default builds.

Labels and attributes take the humus.sim. prefix: humus.sim.seed, humus.sim.node and humus.sim.class. humus.Host's interface doesn't change, and neither do humustest's documented semantics, fault, bedrock or bedrock/fit.

A simulation test

The end user writes four things:

• an operation vocabulary of typed Go values, each acting on their service through its public interfaces, as the walkthroughs' cases already do;
• a model of their own service's semantics;
• which of humus's package models apply;
• optionally, an inspection function over their store's seam.

var placeOrder = sim.Operation[*ordersModel]{
	Name: "place order",
	Run: func(ctx context.Context, op *sim.Op, m *ordersModel) error {
		id := op.ID("order")
		resp, err := op.HTTP("orders").Post(ctx, "/orders", order(id, op.Intn(5)+1))
		m.Placed(id, resp, err) // an error here is an indeterminate outcome, not a failure
		return nil
	},
}

sim.Run gives each seed one bubble and one subtest. sim.Fuzz runs the same body on Go's fuzz engine, so its corpus stores the failing inputs and its minimizer shrinks them:

func FuzzOrders_simulation(f *testing.F) {
	sim.Fuzz(f, simulateOrders)
}

A scenario test is the same harness with a fixed seed and scripted events:

r.At(2*time.Second, sim.Rollout(ledger))
r.At(2*time.Second, sim.SilentConn("wallet", "ledger:9443"))

One run

1. The seed comes in, from HUMUS_SIM_SEEDS or as a fuzz input. Before the run, the harness writes one unbuffered stderr line with the seed and the replay command. It also sets a humus.sim.seed pprof label and emits T.Attr. Each survives a different failure: a panic, a -timeout kill, or -json output.
2. Crypto is seeded with cryptotest.SetGlobalRandom on the outer t (open question 2). One TLS 1.3 transcript came out of 30 processes.
3. The bubble starts. The network, the CA, telemetry and the fakes are all made inside it.
4. The swarm is drawn, each dimension from its own stream. Any dimension can be overridden.
5. Nodes start. Each node is an end user's Init, run as an App value with humustest.Start's semantics. It gets its own root context, a node-scoped Host, and a humus.sim.node pprof label.
6. The driver loop runs:
   1. synctest.Wait();
   2. collect the pending events: chunks queued on links, operations due, faults and lifecycle events due;
   3. sort them canonically and pick one by seed;
   4. advance fake time by a distinct seeded delay;
   5. apply the event;
   6. append it to the history and its hash.
7. Safety checks run throughout. A violation goes from any goroutine through the reporter to t.Errorf, with its class.
8. The liveness phase heals a core and waits for progress within a bound.
9. At the end:
   • synctest's leak check runs;
   • each node's labelled goroutines must reach zero;
   • the class is emitted;
   • on a failure, the trace is written to T.ArtifactDir();
   • the history hash is printed.
10. A wall-clock watchdog outside the bubble classes a real-time hang as inconclusive, never as a pass or a fail.

Determinism at the seam

Source of nondeterminism	Seam	Status
Time, timers, contexts and deadlines, in dependencies too	synctest's fake clock	Covered for everything made in the bubble
Network and files	humus.Host	Covered. The end user's own clients take its dial and resolver, as pgx's DialFunc and LookupFunc do in the walkthroughs
Signals and the process	bedrock.Process	Covered. Simulations run App values, because the Mains call os.Exit
Event order between nodes	the harness's seam scheduler	Seeded
Timers at the same instant	a distinct seeded delay for each harness event	Removed at the seam. Ties inside dependencies stay random
Goroutine order inside a node, select, map order	none on stock Go	Free. The self-check measures it, and -race explores it
Crypto randomness: TLS, CA keys, rand.Text	cryptotest.SetGlobalRandom	Seeded, process-wide, so seeded runs are serial
OTel trace and span IDs	sdktrace.WithIDGenerator	Seeded in sim's telemetry
franz-go's RNGs	seeded from time.Now()	Fixed for free in a bubble
humus's retry jitter, from bedrock/retry.Exponential	an internal hook	Open question 3
grpc-go's and net/http's math/rand/v2 draws	none	Unseedable. Canonical ordering absorbs them unless they change what crosses the seam

Four more rules make the seam hold:

• Decisions are keyed on identity, never on call count. Faults and latencies are keyed on a request ID, a (partition, offset), a node pair or a listener name, and installed at fake times. A fault PRNG drawn in call order fired a different number of times on each run. A record's latency seeded from (seed, partition, offset) took the same time whichever node processed it.
• The seed tree hashes labels. A child seed is H(root, label path), so adding a component or a dimension doesn't reshuffle the other streams. Draws stay on the driver goroutine. The harness turns a Source's output into values with its own functions, because math/rand/v2 promises no stable output for Rand's methods across releases.
• Workload starts are harness events. Over real crypto/tls and HTTP/2 with the seam scheduler, a run gave 1 trace per seed in 300 runs at GOMAXPROCS 32, 4 and 1, bytes included. When the workload's own goroutines started 10 requests at once, it gave 284 to 292 distinct traces in 300.
• The self-check runs on every run. Each run records its history and hashes it. Repeats of a seed compare hashes, and a mismatch names the first event that diverged. TigerBeetle has no such check. In Go it is essential, and one repeat isn't enough (see Testing), so a failing seed runs several times.

The reproducibility claim: two run modes

	Explore mode	Replay mode
Flags	-race, several Ps	-cpu=1, no -race, GOGC=off, a fresh process, -count=1, the recorded toolchain through GOTOOLCHAIN
Finds	Scheduling bugs inside a node. #959 failed only at -cpu 2 and 4	What the seam schedule exposes
Reproduces	Statistically: the seed runs k times	Exactly, when the run has no timer ties and no select with several ready cases

A tie-free run replayed exactly across processes, under -cover and -gcflags=all=-N -l, and on go1.26.5, 1.27.0 and 1.27.2. dlv test … -- -test.run 'FuzzX/<id>' stopped inside a bubble. A breakpoint stops real time, not fake time.

The toolkit claims three things, no more:

1. The plan is exact. Given the seed and the run's identity, it reproduces the swarm, the workload, and every fault and lifecycle event with its fake time, whatever the interleaving.
2. The history is exact when the self-check says so. The history is the harness-owned events: deliveries, operation starts and results, faults and checks. It reproduces whenever each node's reaction to an event doesn't depend on goroutine order.
3. Interleaving inside a node is not reproduced. Bugs there, like humus: a SIGTERM can be reported as a Service returning, when the Service returns as soon as it is cancelled #959 and openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144, are found in explore mode and reproduced statistically. A scripted control then pins each one as a deterministic scenario test, as held reads did for openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144.

The run's identity is printed with every seed:

• the seed;
• the commit: GITHUB_SHA, plus the head and base SHAs on a pull request, or git rev-parse and whether the tree is dirty, because test binaries carry no vcs.* build info;
• the toolchain;
• GOOS, GOARCH and GOAMD64;
• the build flags, -cpu, GOGC, GODEBUG and -run.

Nodes and lifecycle

Event	In process
start(node)	App.Run on a fresh bedrocktest Process, with its own root context and node-scoped network
sigterm(node)	The graceful path
crash(node)	Fence, then wind down. Close every listener and connection the node holds, and refuse its later Listen and Dial calls. Fence its stdout, stderr, telemetry and file opens. Then cancel its root context and discard its exit status
sigkill-after(node, d)	SIGTERM, then a crash at +d, as terminationGracePeriodSeconds does
restart(node)	A crash or a SIGTERM, then a start at the same address
rollout(service, init)	For each replica: take it out of the endpoints, SIGTERM it, start it (possibly with another Init, which models version skew), wait for readiness, and add it back
partition(node, d)	The network clogs every connection of the node

Go can't kill a goroutine, so an in-process crash leaks: in an experiment a Kafka Processor made its side effect 0.5 s after its consumer "crashed". So an edge counts as crashed only if the toolkit revokes it. Each node gets a crash context that the end user's in-memory fakes consult, so a zombie's calls fail. A zombie that can still reach an unfenced edge is reported as a partial partition, not a crash.

A restart isn't a fresh process. Package-level state survives it, and the docs say so.

The network

The cut is at net.Conn, through humus.Host. It is the only cut that keeps #1144, #1176 and #1254 in scope, and it is where humus already cuts. A RoundTripper, a Handler, bufconn or a library's API stay useful for unit tests of logic above the transport, but they aren't transport simulation.

The network is new, in sim. It shares humustest's internals and leaves humustest's documented semantics alone (see conflict C11). It is made of nodes, each a Host, joined by directed links. Each link's latency, jitter bound and buffer are drawn per seed. The network starts no goroutines. An 11-minute partition took 17 to 37 ms of wall time.

Its semantics are TCP's, each checked by a conformance test against loopback, as humustest's conformance_test.go already does:

• the first write after a FIN succeeds, and the next fails with EPIPE;
• closing with unread data sends an RST, and the peer reads what arrived, then gets ECONNRESET;
• a connect completes in the backlog before Accept;
• errors are *net.OpError wrapping *os.SyscallError.

Faults are only those TCP can produce: stalls (clogs and partitions, which queue bytes and lose none), resets, RST on close, silent peers, silent connections, refused or black-holed dials, paused listeners, slow readers, short reads, and a partial write followed by an error. It never injects loss with the stream going on, duplication, reordering, truncation followed by more bytes, or corruption. Vortex removed its truncation fault after it caused a false failure.

Protocol faults come from real peers through their real APIs. GOAWAY comes from http.Server.Shutdown or grpcserver's Shutdown. RST_STREAM comes from a client that cancels, and a gRPC status from a handler. Nothing comes from a frame-level fake peer.

Zero latency stays in the swarm. With 1 ms of modelled latency, #1144 never happened, because fake time froze while the busy handler ran.

Scripted controls are typed values in sim. Each has a "hold until" form, and each is recorded in the trace:

• route a name: a ClusterIP name with one pod per connection, headless names, and endpoint changes over time;
• hold a direction, or hold after N bytes, which models "request processed, response lost";
• partition;
• reset by predicate;
• open a silent connection;
• refuse or black-hole dials;
• pause a listener;
• throttle a reader.

humus hand-builds three of these per test today: held reads, ClusterIP routing and a handshaking connection.

Dependencies

Dependency	Inside the simulation	Outside it	Fidelity guard
Kafka	kfake, which franz-go keeps, on a node's Host. It starts no goroutine of its own. Faults are kfake.Fault, ControlKey and topology changes at fake times. Simulations build with franz-go's -tags synctests, which removed every mutex hang: 0 in 600 seeds, against 8-10% without it	Containers, kind	The container tests' list of what kfake can't do
Kubernetes API	client-go's fakes. 10 simulated minutes of leader election took 0.75 s	k3s, kind	The bare fake never checks resourceVersion, and gave a false split brain (open question 5)
PostgreSQL	The end user's own store interface with an in-memory model, as orderstatus, shoppingcart and wallet have. Faults take pgx's real error shapes: *pgconn.PgError with a SQLSTATE, pgconn.Timeout and unexpected EOF	Containers, kind	The store's own container test
RabbitMQ	Nothing (open question 6). humus's RabbitMQ code keeps its stand-in unit tests	Containers, kind	—
HTTP and gRPC peers	The end user's other services, as Apps on the same network. Invented third-party APIs are the end user's own handlers	WireMock on kind	—
OTLP	humustest's in-memory telemetry, read by the oracles	The Collector on kind	—
JWT key sets	A handler on the Host, with key rotation as a workload event	WireMock	—
Files	Real files through humustest's FS. File-change events, such as a Secret swap at a fake time, are keyed on content, because ModTime is real time	—	—

Every stand-in inside the simulation keeps a differential test against the real dependency, which shows where they differ. That is the guard against TigerBeetle's false-positive lesson, which the research reproduced on humus's own stack.

Faults and the envelope

One seeded schedule of timed events carries:

• lifecycle events;
• network faults and controls;
• dependency faults;
• file-change events;
• request scenarios, as in "request k carries scenario S".

fault and bedrock/fit don't change.

• The workload is the edge. Its single-threaded generator draws, for each request, whether it carries a scenario and which rules. It writes them into the baggage with the end user's own Encoder, so the draws follow the generator, not the scheduler.
• Environment faults come from the network, the fakes and the lifecycle. They land below humus's clients, so retries, breakers and ambiguous outcomes run for real.
• fit can't produce a "sent, then the answer was lost". The network produces it, with "hold after N bytes" or a reset after the write, and kfake produces it for a broker.

The envelope is composed.

• humus ships per-package pieces as typed Go values:
  • the faults its adapter can inject;
  • checkers for the package's stated guarantee;
  • the package's progress property, with a bound computed from the same Settings the node runs with.
• The end user composes them, and adds their service invariants, such as ARolloutOfTheWallet's exactly-once effect. They also state their assumptions, such as "my Processor is idempotent".
• The schedule stays inside the composed envelope. For example, it doesn't crash a broker the envelope assumes is durable.

The package docs are precise enough to derive the safety envelopes. They give no time bounds for progress after a dependency heals, which the pieces add.

Beyond the envelope:

• Inside humus's own code, a fault beyond a package's envelope may cost availability: a stop, an exit 1, shed load or a stall. It never silently breaks the package's stated safety rule. Runs with beyond-envelope faults check safety only.
• A violation caused by a dependency breaking its own guarantee, or by the end user breaking a stated precondition, is classed excused.

Workloads and models

One decision Source has three feeds:

• a seeded PCG, for sweeps and the fleet;
• fuzz bytes, TigerBeetle's "finite PRNG", for go test -fuzz with its corpus and shrinking;
• a three-line adapter to rapid's bit stream, if the end user prefers rapid.

humus's package models:

• Queue processing.
  • Q1: no loss at commit.
  • Q2: committed offsets are monotonic, and no commit is sent under a member that doesn't hold the partition (queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757's rule).
  • Q3: no duplicates under AtMostOnce.
  • Q4: under AtMostOnce, the commit comes before processing.
  • Q5: every loss is explained.
  • Q6: order holds within one member's hold of a partition.
  • Q7: a Process error under AtLeastOnce ends the Service.
  • Q8: shutdown drains.
  • Q9: progress after healing.
• Request and response.
  • Exactly one response or transport error per request, and one server span per handled request.
  • An indeterminate outcome, Jepsen's :info, which a retry with the same idempotency key must resolve to one effect.
  • A planned fault has a known outcome.
  • Client attempts, the retry budget and the breaker's edges stay legal.
  • Each gRPC stream ends with exactly one status.
  • Drain order holds.
• Controllers. Convergence within a bound, finalizers, panics recorded and not retried, and one reconcile at a time per key. Leader overlap is a bounded-overlap property, never strict mutual exclusion.
• Jobs. Exit codes as a total function, and effects over sequences of runs.

Oracles don't depend on order wherever humus allows concurrency. They compare multisets with counts, per-key and per-partition order, final states and per-request outcomes. Even at GOMAXPROCS=1, six spans ended in 249 distinct orders in 300 runs.

Visibility has three levels:

• humus's internals, through hooks;
• the end user's state, through their seams only;
• dependencies, through what the upstream fake exposes, such as kfake's GroupInfo.

#757's rule needed only the third. RabbitMQ offers none, so its oracles stay on the client side.

Telemetry is the event feed. The oracles read per-request outcomes, processing events, breaker and retry decisions, recovered panics and fault evaluations from it. They read it with seeded IDs, simple processors and AlwaysSample. humus adds no production telemetry just for tests, because dashboards and alerts depend on what humus emits.

Recovered panics fail a run by default: "handler panicked", "RPC panicked", "operation panicked", "reconcile panicked" and "the KeySet panicked". The end user can override this per service.

Checks, hooks and verify mode

humus doesn't own the process, so a failed invariant is never a crash in production:

• A programmer error at construction keeps panicking, as now.
• A runtime violation that would break the contract if humus went on doesn't take the action, and fails the Service with a typed error. Examples are a commit that moves backward, or an ack of an unprocessed delivery. This is what group.fail already does in queue/kafka/service.go.
• Any other violation emits a humus.-named ERROR record and span event, and humus goes on.
• In simulation, the harness hook is called first, and every violation is a correctness failure that carries the seed.

Hooks are nil function fields in internal/simhook, with no measurable cost in production:

• Kafka: partition assigned, revoke completed, commit about to be sent, batch skipped;
• RabbitMQ: ack or nack, connection lost or reconnected;
• kube: reconcile start and end, finalizer removed, lease acquired or lost;
• servers: drain started, listener closed;
• a progress signal, "work in flight: n". synctest's idleness isn't quiescence when periodic tickers run: client-go's queue-metrics ticker suppressed a deadlock report, and a stuck receive waited 48 fake hours without one.

Verify mode. Expensive checks sit behind const verify, under //go:build humus.verify. They are compiled out by default, with 0 symbols left in a default build. Code under if verify has no side effects. humus's unit tests run with and without the tag, and the heaviest checks are sampled.

Liveness

• The core is the service's replicas that are up and stay up, each dependency the property needs, and a healed network between them. A dependency outside the core stays down for good. The property then becomes a fail-fast one: calls fail promptly through the breaker, and paths that don't need the dependency keep working.
• Bounds come from Settings. Every humus background loop is "capped backoff, unlimited attempts", so "not yet" and "never" separate after a bound computed from the drawn settings. Simulation configurations shrink the caps through existing settings: kube's per-object 1000 s, grpc-go's 120 s reconnect, RabbitMQ's 30 s.
• Readiness is the startup half, within a bound. Workload progress is the recovery half. Readiness latches on purpose, so "ready" doesn't mean "recovered".

The swarm

• The harness varies what an operator or the environment varies: counts, timings, fault rates, and Settings fields documented as tuning, within their Check* ranges.
• It doesn't vary what defines the service's meaning. Delivery, Classify, topology, HTTP2Only and RabbitMQ Concurrency belong to the end user.
• Ranges include zero: no slow records, one node, one partition, zero latency.
• Biased churn regions find more. A churn swarm found queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757's rewind in 1.05% of seeds, against 0.009% for the default swarm, about 115 times as often.
• Each dimension has its own stream, so knocking one out doesn't reshuffle the rest, and each can be a separate fuzz argument for the mutator to move.

Failures, traces and minimization

Class	How it shows	Kills the binary?
correctness	t.Errorf from any goroutine in the bubble, then T.Attr("humus.sim.class", …)	no
panic	a runtime panic	yes
deadlock	synctest's "all goroutines in bubble are blocked"	yes
leak	synctest's "blocked goroutines remain"	yes
liveness	the harness's bound check, in fake time	no
inconclusive	a real-time hang, caught by the watchdog or -timeout	yes (timeout)
replay diverged	the self-check's hash mismatch	no
excused	a dependency or a stated precondition broke its guarantee	no

Three classes kill the process, so a fleet runs seeds in supervised processes: the fuzz engine's workers, or a runner that reads test2json.

A trace has one line per state change: the fake elapsed time, the node, the component's humus. name, the event, and key=value pairs. It is written to T.ArtifactDir() only on a failure, and its last lines are printed.

Minimization.

• A fuzz target whose input is a choice sequence gets real shrinking from Go's engine. A 53-byte failing run became 5 bytes. Reruns gave 4, 15 and 5 bytes, so the result is small, not minimal.
• A uint64 seed is recorded, never shrunk. Coarse minimization covers it: keep the fastest failing seed per signature, truncate the step budget, and knock out swarm dimensions.
• Shrinking is unreliable under free interleaving, because a shrunk input may not reproduce.

A fixed seed's regression form is a corpus entry that lands with the fix. A hand-written scenario test replaces the entry once it stops pinning the failure.

What end users' code must do

1. Reach the network and files only through Setup.Host. Their own clients take its dial and resolver. Peers are addressed by name. Code doesn't rely on *net.TCPConn.
2. Make everything inside the bubble: timers, contexts, channels, WaitGroups and certificates. No package-level timers or contexts, and no pinned tls.Config.Time or Rand.
3. Hold no mutex across a channel wait or a sleep, because a mutex wait isn't durable blocking.
4. Honour the handler's context, or the leak checks fire.
5. Put stores behind their own seams, with in-memory models whose fault decorators return the real client's error shapes, and container tests that check the models.
6. Make identities unique and observable, for example an ID in each message, and make processing idempotent where the contract says so.
7. Expect no package-level state to reset at a restart. For crash fidelity, every side effect crosses a revocable edge: the Host, or the node's crash context.
8. Read files rotation-safely: by content, tolerating a torn certificate and key pair, keeping the last good value. A torn pair happened through humustest's FS in an experiment.
9. Run seeded tests serially. Crypto is seeded process-wide, so seeded tests use no t.Parallel.
10. Treat math/rand/v2 draws as free nondeterminism. The self-check reports when they leak into the history.

These rules are documented, not enforced by an analyzer. The watchdog and the self-check report violations at run time.

humus's own packages and the walkthroughs

First user	What it runs	What it pins
queue/kafka	A seeded workload of produce, start, stop and sleep steps over the real Service on kfake, with a default and a churn swarm, oracles Q1-Q9 and -tags synctests	#757
the lifecycle	SIGTERM, a second signal, grace expiry and Drain races in explore mode. The exit-cause oracle checks the exit cause and the Drain's ShutdownCause, not just the exit code	#959's class
grpc	Rollouts behind ClusterIP routing with N streams, and silent connections. The doc promises "the new stream lands on another pod" and "the breaker ignores Unavailable ends", and "Shutdown returns within drain plus grace". A transport fuzzer of Shutdown ordering	#1176, #1254, and #1254's side finding that a handshaking connection holds Shutdown for 2 minutes
httpserver, openapi/v3, httpclient	HTTP/2 clients that cancel and close, zero-latency links, Shutdown under silent connections (net/http's 5 s idle rule held Shutdown for 6.4 s), and breaker classification across many streams	#1144's class
kube	A whole controller on client-go's fakes, leader election under partition with a conflict model, and a 410 relist	—
auth/jwt, fault/jsonscenario, openapi parameters, retryAfter and pushback, health's And and Or, the queue header carriers	Targeted fuzzers sharing the Source: codec round trips, differential checks and models. humus has no func Fuzz target today	—
orderstatus	Its cases become a vocabulary, a generator and a model, with the end user's projector.Fold as a differential oracle for humus's machinery	—
wallet	Already a small VOPR cluster: two ledger replicas, a key set and the wallet, with round-robin routing and rollouts. Its harness moves onto sim's nodes and network	#1254
shoppingcart	Already whole in a bubble: cart, catalog and an identity provider on one Host	—
billing	Becomes whole: PostgreSQL behind its store, and the invented payments API as a handler	—
databases	Whole once kubesim's client seam lands	#1250
mailer	Not simulated (open question 6)	—

The existing synctest tests and shared cases become scenario tests in place. They gain seed control, a trace and a replay line, and aren't rewritten.

CI

• The gate is deterministic: corpus entries, scenario tests and a small fixed seed range, in the existing Test jobs. Plain go test already runs corpus entries.
• Replay-mode jobs run beside the -race jobs, because -race randomizes the scheduler.
• Fresh seeds run off the gate: in a non-required job on pushes to main, and in a small scheduled fleet. A fleet of eight shards for 6 hours is about 8 core-days a night, against TigerBeetle's roughly 1,000 core-days a day. The binding limit is the Free plan's 20 concurrent jobs, which a CI run already shares among 54 jobs.
• Seeds come from GITHUB_SHA, the event, the job and an index, and are always printed. The merge group's SHA is the SHA that lands on main, so a seed from the SHA alone would run twice on the same code.
• Failures are recorded in layers: the job summary (the replay line and the trace's tail, since a summary holds at most 1 MiB), an artifact, then a Bugs discussion. A report command shares Proposal: finding flaky tests #1259's runner and deduplicates by signature (the test, the class and the first frame). triage-bug then files a story for the backlog.
• triage-bug runs a failing seed 4 times in replay mode, to show determinism, and 4 times under -race, to show its rate.
• Canaries prove the pipeline works:
  • a planted-bug package that the harness's own tests drive as a subprocess, asserting the seed line, the class, the traceback label, a shrunk input under a bound, and that the replay reproduces;
  • a fleet canary that must fail within each fleet run's budget;
  • a determinism canary: a target known to tie must be reported "replay diverged".
• End users get a recipe in the docs, not a humus CI action.

Design decisions

• synctest and the stock toolchain. Exact interleaving replay would make every end user adopt a forked runtime, a source translator or a hypervisor, or write handlers against a cooperative scheduler. A narrower claim, measured on every run, costs end users nothing.
• Two run modes. Replay mode alone would miss humus: a SIGTERM can be reported as a Service returning, when the Service returns as soon as it is cancelled #959 by construction, and explore mode alone can't replay anything.
• The cut is at net.Conn. TigerBeetle cuts higher because its protocol tolerates any message fault. humus's protocols don't, and its bugs live above the cut.
• A new network in sim, not a changed humustest. The existing tests keep their meaning.
• Faults and latencies are keyed on identity. Call order is scheduling, and scheduling isn't seeded.
• fault and fit don't change. The failure injection proposal's rule against "failing a dependency for every request" is about production traffic, and a simulation's environment faults never pass through fit.
• Envelope pieces are code. Bounds stay right as Settings change, and end users don't re-derive AtLeastOnce from doc text.
• Pause is a partition, named as one. All nodes share one bubble clock, and Go can't suspend some goroutines and not others.
• Clock skew stays out of the whole-service run. One bubble has one clock. JWT leeway and certificate windows are unit tested with an injected time, as TigerBeetle tests skew in its clock's unit tests.
• The scripted controls are exported. End users need them to pin their own failures.
• Recovered panics fail a simulation by default, because humus's own recovery would otherwise hide them. In production humus reports, and the process's owner decides.
• Simulation rules are documented, not enforced by an analyzer, which would have false positives.
• Test binaries may set dependency globals. apimachinery's utilruntime.ErrorHandlers backoff, initialised at real time, blocked a caller for about 27 fake years, so kubesim neutralises it.
• A seed is valid only with its commit and toolchain, as in TigerBeetle. The determinism canary runs on every Go bump.

Conflicts with prior decisions

Each conflict names the decision, where the design bites, and the case for it.

• C1. humus writes no fakes of other projects' servers (#765, Proposal: humus on the bedrock process model #629 revision 10). RabbitMQ gets no in-simulation broker (open question 6). client-go's fakes lack API-server semantics, and a conflict reactor is humus-written fake behaviour, though humus's kube tests and the databases walkthrough already carry such reactors. Case: keep the rule, and ratify one narrow exception, fidelity corrections to an upstream fake (open question 5). The TCP model fakes the kernel, not a server, and is checked against loopback.
• C2. Proposal: queue processing and the Kafka runtime #683's "no test package" and "no API for small things." The toolkit adds sim, kafkasim, grpcsim and kubesim. Case: Proposal: queue processing and the Kafka runtime #683 declined a Kafka test package because "the tests' helpers are a few lines each". The Kafka envelope is oracles Q1-Q9 over kfake's commit history, which queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757 shows takes humus's own expertise. Hooks stay internal/ (open question 4).
• C3. No imposed defaults. Swarm ranges, fault rates, recovered panics failing a run, seed counts. Case: each is an overridable value in test code, fault classes are extensible, and no codec is added.
• C4. Typed Go values, not templates or tags. The humus.verify tag and franz-go's -tags synctests. Case: a build constraint isn't a declaration language. Schedules and controls are typed values.
• C5. No package-level state, and no client-go globals. cryptotest.SetGlobalRandom is process-wide; test binaries replace utilruntime.ErrorHandlers; OTel's error handler, set by every App.Run, sends one App's errors to the last-started App's stderr. Case: these live in test binaries only. The real-time-initialised backoff goes upstream, and the OTel handler is routed per App in phase 1.
• C6. bedrock/retry's jitter from the math/rand/v2 globals. Case: open question 3 seeds it without a bedrock change.
• C7. No hardcoded secrets; generate randomly per test. With seeded crypto, humustest's CA key is a function of the seed. Case: nothing is hardcoded, and the CA is per test and trusted by nothing. Read strictly, crypto is seeded only in replay mode (open question 2).
• C8. Every test runs under -race, in CI, triage-bug and the backlog. Case: replay-mode jobs are added and the -race jobs stay.
• C9. Test-only seams stay test-only (kube/export_test.go's SetClients). Case: end users can't run a whole controller in a bubble the way humus's tests do. kubesim closes that gap without a production option.
• C10. Proposal: finding flaky tests #1259's rules: local and manual only, a test failing every run is refused, duplicates found by name. A deterministic seed fails every run, and seed subtests have unique names. Case: share Proposal: finding flaky tests #1259's runner, canaries and Bugs step. Give simulation failures their own rule ("replays exactly" is real) and deduplicate by signature. The flake tool stays local (open question 8).
• C11. humustest's documented semantics (Proposal: humustest — whole-service testing on top of bedrocktest #633's "Dial refuses rather than waits"). Case: the TCP-faithful network is new, in sim (open question 7).
• C12. humus.FS returns *os.File, so no fault-injecting filesystem is possible. Case: keep it. Filesystem simulation is out of scope, and file-change events use real files.
• C13. The walkthrough rules (Proposal: humus on the bedrock process model #629 revisions 13-17). The cases would carry a vocabulary, a generator and a model. Case: the scripted cases stay as scenario tests, and the pages stay concise (open question 9).
• C14. Shared cases run in parallel. Seeded crypto forbids t.Parallel. Case: a practice, not a decision. A case takes 10-90 ms, and unseeded scenario tests stay parallel.
• C15. Test builds don't differ from production. Case: verify mode is a named exception, its code has no side effects, and unit tests run with and without the tag.

Testing

The evidence

No prototype of the toolkit was built. The research behind this proposal ran throwaway experiments on Go 1.27, with humus at 08b44fa, franz-go v1.22.2 and client-go's fakes. They aren't kept. These are the results the design rests on:

• queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757 was found by the design's pieces. A seeded workload of produce, start, stop and sleep steps drove the real queue/kafka Service on kfake, against a copy with queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757's fix reverted.
  • The oracle "committed offsets never move backward", checked against kfake.GroupInfo, found a rewind from 23 to 5.
  • Seed 329 failed 20 times in 20.
  • The fixed code passed the same seed every time, with one outcome multiset and two processing orders.
  • The default swarm found it in 1 seed in about 11,016, and the churn swarm in 28 in 2,662.
  • Of the churn failures, 10 of 28 reproduced every time, 27 at least once, and 1 never.
• Seam scheduling works on toy nodes and over real TLS and HTTP/2.
  • A seeded scheduler at the seam gave one trace per seed in 300 runs at GOMAXPROCS=32 and under -race.
  • Over real crypto/tls and HTTP/2, it gave 1 trace per seed in 300 without -race and 2-3 per seed with it.
  • It wasn't measured over humustest.Host with real humus services.
• The self-check catches leaks. A deliberately racy node leaked into the history 11 times in 300 runs (285 in 300 under -race). Running each seed twice caught it in 33 of 100 processes.
• The bare client-go fake misleads. Under a stalled renewal it gave split-brain leader election in 10 of 10 runs, about 9m40s each. A conflict reactor brought takeover down to the real 3.5-4.6 s.
• The close model changes bug rates. openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144 happened 18 times in 100 with humustest-like close and once in 100 with TCP-like close, under -race on one CPU. With 1 ms of latency it never happened.
• Lifecycle in one bubble is cheap. Fifty restarts took 0.05 s.
• Replay holds across tools. A tie-free run replayed exactly across processes, under -cover, under -gcflags=all=-N -l, on three toolchains and under Delve.
• Shrinking works but isn't stable. A choice-sequence fuzz input shrank from 53 bytes to 5. Reruns gave 4, 15 and 5.
• Verify mode is free in production. A humus.verify const left 0 symbols in a default build.
• A swarm fuzz of a trivial bubble ran 660,000-720,000 executions a second.

What it would have caught

"Found" means the design's swarm and oracles reach the trigger. "Replayed" means the seed reproduces the failure. Only #757 was run against the design's pieces. Every other row is inferred from the bug's triage record.

Bug	Found?	Replayed?	Phase
#757 a fenced member's late commit	Yes, shown	Yes	1
#959 SIGTERM reported as a Service returning	Rarely, in explore mode, with the exit-cause oracle	No: statistical. Replay mode never finds it, because it needs several Ps	1
#1144 HTTP/2 went-away logged as "response failed"	Yes, in explore mode at a few percent, with zero-latency links and a client that cancels a streaming request	No: statistical. Held reads pin it	2
#1176 a rollout opens the breaker	Yes, when the swarm draws a rollout and 5 or more streams	Yes: topology and lifecycle, not scheduling	2
#1250 a deleted Secret comes back	Yes, once controllers run whole on fakes with "delete a Secret" in the workload	Likely: ordered by fake time. Testing the fix needs the uid fidelity reactor	3
#1254 streams cancelled before GOAWAY	Yes, with the silent-connection fault. Sometimes without it	Yes with that fault (4 of 4), statistical without it (2 of 4)	2
#1255 the test's reader stopped early	Not a service bug. The design's rule that correct code must pass exposes such readers	kfake.Fault on ListOffsets forces the trigger	1

So:

• The design finds and replays the bugs that topology, lifecycle and faults decide: queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757, grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176, example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost #1250 and example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254.
• It finds, but can't replay, the bugs that one node's goroutine order decides: humus: a SIGTERM can be reported as a Service returning, when the Service returns as soon as it is cancelled #959 and openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144.
• Two bugs escaped humus's current in-memory tests through model fidelity: example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254 through connect semantics, and example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost #1250's fix through missing API-server semantics. That is why the conformance and differential checks are part of the design, not extras.

How the toolkit is tested

• The three canaries above: planted bug, fleet and determinism.
• The network's conformance tests against loopback, under -tags conformance.
• A differential test for every stand-in against its real dependency.
• A regression benchmark: copies of humus with the fixes for queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757, humus: a SIGTERM can be reported as a Service returning, when the Service returns as soon as it is cancelled #959, openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144, grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176 and example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254 reverted, swept in explore mode, reporting the rate per seed.
• Unit tests with and without humus.verify.
• The rule that correct code must pass every oracle. The first exit-code oracle in the research reported 26 false failures on correct code.

Phases

One proposal and one milestone. Its stories make the cross-package changes: hooks, verify checks and invariant reporting in queue/kafka, kube, the servers and the clients.

Phase 1: the harness, on Kafka.

• Builds:
  • seed plumbing, the Source and its feeds, the seed tree and the swarm;
  • the run's identity, the failure classes, the watchdog, the trace, the history hash and self-check, and both run modes;
  • seeded crypto and OTel IDs;
  • nodes on today's humustest Host, with start, SIGTERM, restart and rollout;
  • internal/simhook, the invariant policy and humus.verify;
  • liveness from Settings;
  • the planted-bug and determinism canaries;
  • the deterministic gate in the Test jobs.
• First users: kafkasim with oracles Q1-Q9, the lifecycle model, and orderstatus's vocabulary, generator and model.
• Pins: queue/kafka: a fenced-out member's late commit can move the new owner's committed offset #757, and humus: a SIGTERM can be reported as a Service returning, when the Service returns as soon as it is cancelled #959's class.

Phase 2: the network and crashes.

• Builds:
  • the simulated network with node-scoped Hosts, TCP semantics and conformance tests;
  • the time-keyed, name-keyed fault schedule, and the scripted and routing controls;
  • seam scheduling of deliveries and workload starts;
  • crashes as a fence and a wind-down, the crash context, and sigkill-after-grace;
  • a CA that covers the simulated span, since humustest's certificates last 24 h;
  • transport fuzzers of Shutdown ordering and of openapi's went-away classification.
• First users: grpcsim, the HTTP models, wallet and shoppingcart.
• Pins: openapi/v3: over HTTP/2, a write that fails because the client closed its connection is logged as "response failed" #1144, grpcclient: a rollout of a dependency served by grpcserver opens the breaker of a Client with five streams open to the pod #1176 and example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254, with example/wallet: a rollout of the ledger can end a watch with LEDGER_UNAVAILABLE, when the wallet opens the ledger's watch again on the pod that's shutting down #1254's grace overrun.

Phase 3: dependencies and envelopes.

• Builds:
  • kubesim, with the client seam, the fidelity reactors and the controller models;
  • PostgreSQL error-shape guidance for end users' stores;
  • file-change events;
  • the composed envelope pieces with excused classification;
  • the targeted fuzzers.
• First users: kube, databases and billing.
• Pins: example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost #1250.

Phase 4: CI, fleet and reporting.

• Builds:
  • the non-required fresh-seed job and the scheduled fleet;
  • the report command on Proposal: finding flaky tests #1259's runner;
  • triage-bug's replay-mode rule;
  • the fleet canary;
  • coverage of what the corpus reaches, through GOCOVERDIR and covdata;
  • the end users' recipe.

Phase 3 needs phase 2 only for crash events, so the two can run in parallel once phase 1's interfaces land.

Migration from today

Nothing breaks.

• humustest, fault, bedrock and fit keep their behaviour.
• The existing synctest tests and shared cases keep running as they do, and gain seeds, traces and replay lines as they become scenario tests.
• The new CI jobs sit beside the existing ones.
• An end user who never imports sim sees no change.

Open questions

1. Is the three-part reproducibility claim acceptable? As proposed: the plan is exact, the history is exact when the self-check says so, and interleaving inside a node is statistical, with explore and replay modes. The alternatives:

   • require exact interleaving replay, through a forked runtime, source translation or a hypervisor;
   • also evaluate Antithesis as a hosted, replayable tier over real dependencies.

   Recommendation: as proposed. The alternatives take end users off the stock toolchain, or are commercial and outside-in.

2. Seed crypto with cryptotest.SetGlobalRandom? It is process-wide, forbids t.Parallel, and is unsupported with the FIPS v1.0.0 module.

   • (a) Seed it in every simulation run, so runs are serial.
   • (b) Seed it only in replay mode.
   • (c) Never seed it, and keep TLS bytes out of the history.

   Recommendation: (a), or (b) if the no-secrets rule is read strictly (C7). Seeded keys and certificates make traces comparable.

3. humus's retry jitter in simulation?

   • (a) Leave it unseeded, so the self-check absorbs or reports it.
   • (b) An internal hook makes humus build a seeded schedule per client in simulation, with no public API and no bedrock change.
   • (c) Hash (seed, client, attempt), which defeats jitter's purpose.
   • (d) Add a source parameter to bedrock's retry.Exponential.

   Recommendation: (b). Phase 1 measures how much jitter moves the history.

4. Where does the toolkit live? As proposed: sim in the root module, plus kafkasim, grpcsim and kubesim in their modules. The root module can't import kfake or client-go. The alternative extends humustest and the *servertest packages instead. Recommendation: as proposed, which answers Proposal: queue processing and the Kafka runtime #683's reason (C2). The names are open too: sim could be simulation.

5. Ratify "fidelity corrections to an upstream fake"?

   • (a) Yes. Each correction is a minimal reactor that restores a documented API-server semantic, with a differential test against k3s, and it is proposed upstream to client-go too.
   • (b) No reactors. Kube simulations never inject pauses or slow renewals, and fixes like example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost #1250's are left to the end user's own tests.
   • (c) envtest, which brings controller-runtime.

   Recommendation: (a). Without it, kube fault simulation reports a split brain that can't happen (C1).

6. RabbitMQ in the deterministic tier?

   • (a) Keep Proposal: RabbitMQ #765: RabbitMQ stays in containers.
   • (b) A model of the queue contract (redelivery, duplicates, a crash mid-processing) that skips rabbitmq.Service.
   • (c) A humus-written AMQP simulator, reversing Proposal: RabbitMQ #765.
   • (d) A third-party Go broker, of unknown fidelity.

   Recommendation: (a). TigerBeetle leaves its own AMQP integration outside its simulator too. The mailer and end users' RabbitMQ services aren't simulated whole.

7. Should the simulated network replace humustest's semantics? As proposed, it is new, in sim, and humustest stays as it is (C11). The alternative makes humustest TCP-faithful, which leaves one network to keep but changes documented behaviour. Recommendation: as proposed.

8. Where does exploration run, and who posts failing seeds?

   • (a) Fresh seeds on every pull request, blocking the merge, as TigerBeetle's CI does.
   • (b) A deterministic gate. Fresh seeds run in a non-required job on main and in a scheduled fleet. CI posts a deduplicated Bugs discussion from the fleet only, which needs discussions: write.
   • (c) As (b), with a non-required fresh-seed job on pull requests too.

   Recommendation: (b). A fresh-seed gate would block pull requests on bugs they didn't cause (C10).

9. Restructure the walkthroughs' shared cases into a vocabulary, a generator and a model?

   • (a) Yes for orderstatus, wallet and shoppingcart, and for billing once its store and payments handler are in place. The scripted cases stay as scenario tests.
   • (b) Keep the cases as they are, and add separate simulation tests.

   Recommendation: (a). The Target seam already lets one vocabulary run in the bubble and on kind (C13).

10. One proposal in four phases, or four proposals? Four would give smaller reviews, but phase 1's interfaces would then be decided before the network and kube needs that shape them were visible. Recommendation: one proposal, whose phases order its stories.

11. Prototype phase 1 before the stories are filed? Three things the design rests on are unmeasured or untried:

    • seam determinism over humustest.Host with real humus services;
    • the composition of envelope pieces;
    • the restructured cases.

    A prototype of the harness on queue/kafka and orderstatus would settle them, and fix the sketches' signatures. Recommendation: yes, posted as a revision before stories.

Out of scope

• Exact interleaving replay: a forked runtime, source translation or a hypervisor (open question 1).
• A humus-written RabbitMQ broker, or a queue contract model (open question 6).
• Frame-level HTTP/2 or gRPC fake peers, and scripted wire peers serving workloads (C1).
• Filesystem simulation: crash semantics, ENOSPC and EIO (C12).
• Clock skew in the whole-service run. It is unit tested.
• A true process pause, which needs real processes.
• The Vortex-like tier: process kill and pause on kind, Toxiproxy or Chaos Mesh, seeded fault plans, fault timelines and operation histories. It brings its own decisions, such as privileged chaos daemons and kind overlay rules. Today's testcontainers and kind tiers continue unchanged. A separate proposal can take it up.
• Antithesis, which is commercial and hosted.
• Self-hosted runners and a 24/7 fleet. GitHub advises against self-hosted runners on public repositories.
• Changes to bedrock or bedrock/fit.
• Crash-on-violation in production.

References

• TigerBeetle at 0450c32b: TIGER_STYLE.md, ARCHITECTURE.md, the VOPR's docs, src/vopr.zig, src/testing/ and src/vortex.zig
• TigerBeetle's posts: Simulation Testing for Liveness, Random Fuzzy Thoughts, A Descent Into the Vörtex, Swarm Testing Data Structures, Fuzzer Blind Spots (Meet Jepsen!), A Tale of Four Fuzzers and protocol-aware DST
• Jepsen, TigerBeetle 0.16.11
• FoundationDB, Simulation and Testing
• Go: testing/synctest, testing/cryptotest, and the issues golang/go#67434, golang/go#73850, golang/go#73876 and golang/go#67690
• Prior art in Go: gosim, Polar Signals, WarpStream, Antithesis's Go SDK and deterministic hypervisor, and Resonate
• franz-go's kfake, and its synctests build tag in pkg/kgo/internal/xsync
• humus: the lifecycle, humustest, queue processing and the Kafka runtime, RabbitMQ, failure injection, Kubernetes controllers and finding flaky tests
• The bugs: #757, #959, #1144, #1176, #1250, #1254 and #1255

[Zaba505]

Revision 1: the harness and the network move to bedrock

z5labs/bedrock#628 proposes the half of this proposal that doesn't depend on humus: the harness, as a new bedrock module sim, and the simulated network, as hostsim in bedrock's host module. Neither needs humus. Every node is a bedrock.App on a bedrocktest Process, and the network is a Host. humustest's network was ported to bedrock's hosttest in bedrock#614, so a network here would copy it again. The command-line framework planned on bedrock needs the same harness, without importing humus.

This revision builds on that proposal. humus keeps the models of its packages, its hooks, verify mode, its telemetry and its CI. Eight things change:

1. humus's own package is humussim, built on bedrock's sim. It runs an Init as a node, builds the node's humus.Host from its hostsim host, feeds humus's telemetry to the oracles, and holds the models of the root module's packages. kafkasim, grpcsim and kubesim build on bedrock's sim too.
2. Open questions 1, 2, 3 and 7 move to bedrock, as its questions 1, 2, 3 and 5. This proposal takes bedrock's answers.
3. humus's retry jitter needs no hook, if bedrock answers its question 3 as recommended. retry.Exponential then seeds itself from crypto/rand, which each run already seeds. Conflict C6 is dropped.
4. Conflict C11 moves to bedrock, as its C4. humustest stays as it is.
5. Names come from bedrock: sim.seed, sim.node, sim.class and BEDROCK_SIM_SEEDS. The names humus invents for its components, telemetry and hooks keep the humus. prefix.
6. Phases 1 and 2 build on bedrock's phases 1 and 2, and each waits for the bedrock release that brings them. Phases 3 and 4 are humus's alone.
7. humus doesn't have to adopt bedrock's host module first. humus.FS is an interface, which a hostsim host's filesystem satisfies, so humussim builds a humus.Host from a hostsim host's parts. Adopting the host module stays a proposal of its own.
8. Changes to bedrock leave Out of scope. bedrock gains sim and hostsim, and retry's jitter becomes seedable. fault and bedrock/fit still don't change.

Each section below replaces or adds to the part of the same name in the post. The rest of the post stands.

The note at the top

A third paragraph is added:

It depends on z5labs/bedrock#628, which holds the harness and the simulated network. Phases 1 and 2 build on its phases.

Summary

The first bullet becomes:

humus gets simulation testing in the style of TigerBeetle's VOPR, as a toolkit on bedrock's harness and simulated network (z5labs/bedrock#628). End users simulate their own humus services with it, and humus's packages and walkthroughs are its first users.

In the bullet on faults, "fault, bedrock/fit and bedrock don't change" becomes "fault and bedrock/fit don't change". The last bullet gains "Phases 1 and 2 build on bedrock's."

The example becomes:

func TestOrders_simulation(t *testing.T) {
	sim.Run(t, func(r *sim.R) {
		broker := kafkasim.Broker(r, "kafka:9092")
		orders := humussim.Service(r, "orders", app.Init,
			sim.Replicas(1, 3),
			sim.Env(map[string]string{"KAFKA_BROKERS": "kafka:9092"}),
		)
		m := newOrdersModel()
		sim.Workload(r, m, placeOrder, cancelOrder, produceEvent)
		r.Check(kafkasim.AtLeastOnce(broker, "orders"))
		r.Check(m)
		r.Lifecycle(orders, sim.Crashes(), humussim.Rollouts())
	})
}

Packages

The table becomes:

Package	Module	Role
sim	github.com/z5labs/bedrock/sim	bedrock's harness (z5labs/bedrock#628). The run and its seed, the Source, the seed tree, the swarm, nodes and lifecycle events, the driver loop, the workload driver, the checker interface, the liveness phase, the trace and history hash, the failure classes, the run's identity and the replay line.
host/hostsim	github.com/z5labs/bedrock/host	bedrock's simulated network, with TCP's semantics, routed names, its faults and the scripted controls.
humussim	root	humus on the harness. A node from an end user's Init, run as an App, on a humus.Host built from its hostsim host. In-memory telemetry with seeded IDs, which the oracles read. Recovered panics failing a run. Rollouts that wait on a node's readiness. Swarm dimensions for the Settings fields documented as tuning. The models for the root module's packages: the lifecycle, httpserver, httphealth, openapi, httpclient, health and job.
grpc/grpcsim	grpc	As before.
queue/kafka/kafkasim	queue/kafka	As before, with kfake listening on a hostsim host.
kube/kubesim	kube	As before.
internal/simhook	root	As before.
build tag humus.verify	all	As before.

The paragraph under it becomes:

Labels and attributes are bedrock's: sim.seed, sim.node and sim.class. The names humus invents for its components, telemetry and hooks keep the humus. prefix. humus.Host's interface doesn't change, and neither do humustest's documented semantics, fault or bedrock/fit.

A simulation test

sim.Run, sim.Fuzz, sim.Operation and the scripted events are bedrock's. Two things in the sketches change:

• r.Workload(…) becomes sim.Workload(r, m, …), a function, because a method can't take a type parameter.
• op.HTTP("orders") becomes the end user's own client over op.Dial, the workload's host. bedrock's Op has no client helper, as bedrock#614 decided for hosttest.

One run

Steps 1 to 4 and 6 to 10 are bedrock's sim.Run, as its One run says. humus adds three things:

• in step 3, humus's in-memory telemetry with seeded IDs is made in the bubble;
• in step 5, a node is an end user's Init, run as an App with humustest.Start's semantics, on a humus.Host built from its hostsim host;
• in step 7, humus's models and hooks report through the run.

A sweep's seeds come from BEDROCK_SIM_SEEDS, not HUMUS_SIM_SEEDS.

Determinism at the seam

The table and its four rules are bedrock's. Three rows say what humus adds:

Source of nondeterminism	Seam	Status
Network and files	humus.Host, built from a hostsim host	Covered. The end user's own clients take its dial and resolver, as pgx's DialFunc and LookupFunc do in the walkthroughs
OTel trace and span IDs	sdktrace.WithIDGenerator	Seeded in humussim's telemetry
humus's retry jitter, from bedrock/retry.Exponential	a generator seeded from crypto/rand (z5labs/bedrock#628, question 3)	Seeded with crypto

The row for franz-go's RNGs stands.

The reproducibility claim: two run modes

The claim and the run modes are bedrock's (z5labs/bedrock#628, question 1), and humus claims nothing beyond them. What this section says of humus's bugs stands: #959 failed only at -cpu 2 and 4.

Nodes and lifecycle

The events are bedrock's. humus adds two things through humussim:

• a rollout's readiness is the node's own readiness check;
• a crash fences the node's telemetry, as well as its host and its streams.

The network

The network is bedrock's hostsim, with everything this section said: the cut at net.Conn, TCP's semantics checked against loopback, only the faults TCP can produce, protocol faults from real peers, zero latency in the swarm, and the scripted and routing controls. humustest's network and its documented semantics stay as they are.

What end users' code must do

The rules are bedrock's, in its "What an App's code must do", and rule 1's Host is Setup.Host. humus adds one: a Kafka simulation builds with franz-go's -tags synctests.

Conflicts with prior decisions

• C6 is dropped. Seeding retry's jitter is bedrock's question 3, and its C3 takes up fix(deps): update module github.com/jackc/pgx/v5 to v5.11.0 #606's rule. If bedrock keeps that rule, C6 and open question 3 come back here.
• C11 moves to bedrock, as its C4. humustest's documented semantics don't change.
• C7 and C14 stand. Whether crypto is seeded is bedrock's question 2 now, and bedrock's C6 is C14's counterpart there.

The rest stand.

Testing

Under "How the toolkit is tested":

• the network's conformance tests, and the planted-bug and determinism canaries, are bedrock's;
• humus keeps the fleet canary, a differential test for every stand-in, the regression benchmark, unit tests with and without humus.verify, and the rule that correct code must pass every oracle.

Phases

Phase 1 builds on bedrock's phase 1. That brings the seeds, the swarm, the run modes, the self-check, hostsim delivering at once, nodes, liveness and the canaries. humus builds:

• humussim's nodes, telemetry and recovered-panic rule;
• internal/simhook, the invariant policy and humus.verify;
• liveness bounds from Settings;
• the deterministic gate in the Test jobs.

Its first users and pins don't change.

Phase 2 builds on bedrock's phase 2. That brings the network's seeded schedule, its faults and controls, and crashes. humus builds:

• the fence on a crashed node's telemetry;
• a CA that covers the simulated span, since humustest's certificates last 24 h;
• the transport fuzzers of Shutdown ordering and of openapi's went-away classification.

Its first users and pins don't change.

Phases 3 and 4 don't change.

Migration from today

"humustest, fault, bedrock and fit keep their behaviour" becomes:

humustest, fault and fit keep their behaviour. bedrock gains sim and hostsim, and retry's jitter is seeded from crypto/rand (z5labs/bedrock#628). Each phase requires the bedrock release that brings its part.

Open questions

• Question 1 moves to bedrock, as its question 1. This proposal takes its answer.
• Question 2 moves to bedrock, as its question 2. C7 still applies to humus's tests.
• Question 3 moves to bedrock, as its question 3. If bedrock keeps fix(deps): update module github.com/jackc/pgx/v5 to v5.11.0 #606's rule, the question comes back here with its options (b) to (d).
• Question 4 becomes: where does humus's part live, and what is it called? As revised: humussim in the root module, plus kafkasim, grpcsim and kubesim in their modules, all on bedrock's sim. The root module can't import kfake or client-go. The alternative extends humustest and the *servertest packages instead. Recommendation: as revised, which answers Proposal: queue processing and the Kafka runtime #683's reason (C2). humussim is named after humustest.
• Question 7 moves to bedrock, as its question 5.
• Question 10 stands, next to Proposal: simulation testing shared by frameworks built on bedrock bedrock#628, which holds what bedrock ships. Recommendation: one proposal, whose phases order its stories.
• Question 11's prototype is of bedrock's harness, with queue/kafka and orderstatus as its first users, and it settles the signatures of both proposals. bedrock's question 8 asks the same. Recommendation: yes, one prototype, posted as a revision of both.

Questions 5, 6, 8 and 9 stand.

Out of scope

"Changes to bedrock or bedrock/fit" becomes "Changes to bedrock/fit". The rest stands.

References

bedrock's simulation testing, host and retry proposals are added.