chaos testing
Pre-Alpha. This page describes behavior that may change.
Chaos testing is built into Ze. It is not a side project or a separate repo. The daemon ships with a chaos mode that can inject faults at runtime, and there is a separate orchestrator (ze-chaos) that drives multiple BGP peers against a Ze instance, validates protocol-level invariants under fault, and ships with a web dashboard for watching it happen.
The point is to find bugs that the unit tests cannot. Race conditions between the FSM and the RIB. Convergence regressions. Half-open sockets that the parser handles wrong. Plugins that crash when an event arrives in an order they did not expect. The kind of thing you only find in production unless something pushes the system into that state on purpose.
This section is the operator's view of chaos testing. The deeper architecture reference lives in the in-tree chaos testing guide and the chaos web dashboard architecture.
There are two ways to enable chaos in Ze.
The first is the in-process flag pair on the daemon itself.
ze --chaos-seed 42 --chaos-rate 0.1 config.conf--chaos-seed is the PRNG seed. 0 disables chaos entirely with zero overhead. -1 is time-based (non-reproducible). Any other value seeds the PRNG and gives you a reproducible fault sequence. --chaos-rate is the per-operation fault probability between 0.0 and 1.0.
The second is ze-chaos, the orchestrator. It runs a configurable number of mock peers, generates routes, validates that the route table converges, and injects faults along the way. It has both a real-network mode and an in-process mode with a virtual clock for fully deterministic runs.
ze-chaos --ze ./bin/ze --seed 42 --peers 8 --duration 60s | ./bin/ze -
ze-chaos --in-process --seed 42 --duration 30sThe in-process mode is the one you want in CI. Real network mode is the one you want when you suspect the bug is in the OS network path.
Every chaos run is reproducible from its seed. The same seed and the same rate produce the same fault sequence on every run. That is what makes chaos testing useful for debugging: when the dashboard shows convergence breaking on run 12, you take the seed and you reproduce the failure on demand.
ze --chaos-seed 42 --chaos-rate 0.1 config.conf
ze --chaos-seed 42 --chaos-rate 0.1 config.conf # identical fault sequenceCombined with event logging and replay, this is the part of the chaos surface that pays for itself. You record a failing run, you reduce it to its minimal reproduction with --shrink, and you put the smaller scenario in the test suite.
ze-chaos --event-log run.ndjson --seed 42 | ./bin/ze -
ze-chaos --replay run.ndjson
ze-chaos --shrink run.ndjsonze-chaos --properties all validates five protocol-level invariants while it is running.
| Property | What it checks |
|---|---|
| Convergence deadline | Every route reaches every peer that should see it within the deadline, despite the faults. |
| Route consistency | The RIB matches the peers' adj-RIB-out after recovery. |
| No duplicate routes | No peer receives duplicate announcements for the same prefix. |
| Hold timer enforcement | Hold timers are enforced correctly under fault conditions. |
| Message ordering | Messages are delivered in the correct order. |
Use --convergence-deadline 5s to set the deadline for the convergence check.
| Category | Examples |
|---|---|
| Network I/O | Connection failures, partial writes, read timeouts. |
| Wire parsing | Malformed messages, truncated packets. |
| FSM | State machine violations, invalid transitions. |
| Route processing | Dropped updates, corrupted NLRI. |
| Event delivery | Lost messages, out-of-order events. |
The honest list is shorter than you might expect.
- Before deployment, to make sure your config tolerates peer failures.
- In CI, to catch races and edge cases on every commit.
- When debugging an intermittent failure, to reproduce it with a fixed seed.
- When benchmarking, to measure convergence under fault.
The make targets give you both the unit and functional chaos suites at once.
make ze-chaos-test # Unit + functional
make ze-chaos # Build the orchestrator- In-tree chaos testing guide for the long version.
- Chaos web dashboard architecture for how the dashboard is built.
- Development: building for the make targets that wire chaos into the test suite.
Adapted from main/docs/guide/chaos-testing.md and main/docs/architecture/chaos-web-dashboard.md.