scenarios

Pre-Alpha. This page describes behavior that may change.

A chaos scenario in Ze is a configuration of ze-chaos: the number of mock peers, the address families they negotiate, the duration, the chaos seed and rate, and any explicit fault triggers. You build one by combining ze-chaos command-line flags, and you can save and replay the exact fault sequence through event logs.

This page covers the operator view of writing and running scenarios. If you want the deeper material about the orchestrator design, the convergence tracker, or the property model, read the in-tree chaos testing guide.

The two running modes

ze-chaos has two modes, and the choice matters because they target different kinds of bugs.

Real-network mode. ze-chaos runs mock peers over real TCP sockets against a real ze binary. Slower than in-process, but it exercises the kernel network path, which is where a handful of Ze's known edge cases live.

ze-chaos --ze ./bin/ze --seed 42 --peers 8 --duration 60s | ./bin/ze -

ze-chaos writes the Ze config to stdout and the diagnostics to stderr, so the pipe feeds the config directly into the daemon. If you prefer a file:

ze-chaos --config-out chaos.conf --seed 42 --peers 8
ze start chaos.conf

In-process mode. Mock network, virtual clock, fully deterministic. This is the mode you want in CI: it is fast, it does not need root, and it produces the same fault sequence on every run.

ze-chaos --in-process --seed 42 --duration 30s

Building a scenario

The most common flags, in the order you usually reach for them:

Flag	Purpose
--seed <N>	PRNG seed. Same seed plus same rate equals same fault sequence.
--peers <N>	Number of mock peers.
--duration <dur>	How long the run lasts.
--families <list>	Comma-separated families to negotiate.
--chaos-rate <0-1>	Per-operation fault probability.
--convergence-deadline <dur>	Deadline for the convergence property check.
--properties <list>	Which property validators to run. all runs all of them.
--event-log <file>	Record the run as an NDJSON event log.
--replay <file>	Replay a recorded event log.
--shrink <file>	Reduce a recorded run to its minimal reproduction.

A full-family deterministic run that validates all properties and records everything:

ze-chaos --in-process --seed 42 --duration 60s \
         --peers 8 --families ipv4/unicast,ipv6/unicast \
         --chaos-rate 0.2 \
         --properties all \
         --convergence-deadline 5s \
         --event-log run.ndjson

Replay and shrink

A failing run from the event log can be replayed deterministically. The same seed and the same event stream reproduce the same failure every time.

ze-chaos --replay run.ndjson

When a failing run is too long to debug, shrink it. The shrinker repeatedly removes events and replays the result to find the smallest subset that still reproduces the failure. The output is a new NDJSON file with the minimal reproduction.

ze-chaos --shrink run.ndjson > minimal.ndjson
ze-chaos --replay minimal.ndjson

The shrunk scenario is what you put in the test suite.

Properties

--properties all validates five protocol-level invariants while the run is happening. You can also list them individually.

Property	What it checks
convergence-deadline	Every route reaches every peer that should see it within the deadline, despite the faults.
route-consistency	The RIB matches the peers' adj-RIB-out after recovery.
no-duplicate-routes	No peer receives duplicate announcements for the same prefix.
hold-timer-enforcement	Hold timers are enforced correctly under fault conditions.
message-ordering	Messages are delivered in the correct order.

ze-chaos --properties list shows the current list from the running binary. Adding a property in the code makes it appear here automatically.

Fault categories

The faults ze-chaos knows how to inject are grouped into five categories.

Category	Examples
Network I/O	Connection failures, partial writes, read timeouts.
Wire parsing	Malformed messages, truncated packets.
FSM	Invalid state transitions, unexpected messages.
Route processing	Dropped updates, corrupted NLRI.
Event delivery	Lost messages, out-of-order events.

The --chaos-rate flag controls how often any of these trigger. 0.0 means never. 1.0 means every operation. 0.1 (the default) is the one you want for a reasonable balance between stress and progress.

When to use which

The honest list.

• Deterministic repro of a failing test. --in-process --seed <N> --replay failing.ndjson.
• Fast CI smoke test. --in-process --seed <fixed> with a short duration and --properties convergence-deadline.
• Exercising the real network path before a release. Real-network mode, longer duration, --properties all.
• Hunting a specific bug. Record with --event-log, fail, shrink, commit the shrunk scenario as a test.

Additional flags

These flags extend the scenario model with comparison, backpressure, load, and observability controls.

Event log comparison

--diff and --diff2 compare two event logs side by side, showing where they diverge. Useful for understanding what changed between two runs with different seeds or code versions.

ze-chaos --diff run-a.ndjson --diff2 run-b.ndjson

Slow-reader simulation

--slow-peers and --slow-read-delay simulate peers that read slowly, exercising backpressure paths in the route server. The specified number of peers will delay every read by the given duration.

ze-chaos --seed 42 --peers 8 --slow-peers 2 --slow-read-delay 1s

Heavy peer simulation

--heavy-peers (percentage) and --heavy-routes (count) designate a fraction of peers as heavy senders. Heavy peers inject far more routes than the base count, simulating full-table peers in a mixed environment.

ze-chaos --seed 42 --peers 10 --heavy-peers 10 --heavy-routes 1000000

Route churn

--churn-rate sets the percentage of routes churning per second per peer. When --route-rate is not explicitly set, it is derived from the churn rate. This controls how much route dynamics stress the convergence path.

ze-chaos --seed 42 --peers 8 --churn-rate 0.05

Warmup delay

--warmup sets the delay before chaos events start firing. This gives the route server time to establish sessions and converge before faults are injected.

ze-chaos --seed 42 --peers 8 --warmup 10s

Observability endpoints

--metrics starts a Prometheus metrics endpoint for ze-chaos itself, exposing per-peer and per-property counters. --pprof starts a pprof HTTP server for profiling ze-chaos during long runs.

ze-chaos --seed 42 --peers 8 --metrics :9090 --pprof :6060

See also

• Chaos testing index for the why and the overall shape.
• Dashboard for the live web UI that watches runs in real time.
• Property verification for what the invariants actually check.

Adapted from main/docs/guide/chaos-testing.md.