chaos testing
Pre-Alpha. This page describes behavior that may change.
Chaos testing is built into Ze. It is not a side project or a separate repo. The daemon ships with a chaos mode that can inject faults at runtime, and there is a separate orchestrator (ze-chaos) that drives multiple BGP peers against a Ze instance, validates protocol-level invariants under fault, and ships with a web dashboard for watching it happen.
The point is to find bugs that the unit tests cannot. Ze-chaos also includes an MCP server for AI-driven chaos testing and a Watchdog consumer for automated monitoring. Race conditions between the FSM and the RIB. Convergence regressions. Half-open sockets that the parser handles wrong. Plugins that crash when an event arrives in an order they did not expect. The kind of thing you only find in production unless something pushes the system into that state on purpose.
This section is the operator's view of chaos testing. The deeper architecture reference lives in the in-tree chaos testing guide and the chaos web dashboard architecture.
There are two ways to enable chaos in Ze.
The first is the in-process flag pair on the daemon itself.
ze --chaos-seed 42 --chaos-rate 0.1 config.conf--chaos-seed is the PRNG seed. 0 disables chaos entirely with zero overhead. -1 is time-based (non-reproducible). Any other value seeds the PRNG and gives you a reproducible fault sequence. --chaos-rate is the per-operation fault probability between 0.0 and 1.0.
The second is ze-chaos, the orchestrator. It runs a configurable number of mock peers, generates routes, validates that the route table converges, and injects faults along the way. It has both a real-network mode and an in-process mode with a virtual clock for fully deterministic runs.
By default ze-chaos forks the daemon as a child process: you no longer need to pipe a config into a separately started ze. Point it at a binary with --binary (it auto-discovers one in PATH if you do not), and it generates the config, starts the child, drives the run, and detects a crash if the child dies. Auto port allocation (--port 0, the default) asks the kernel for a free port so parallel runs do not collide.
# Fork mode (default): ze-chaos starts ze as a child process
ze-chaos --seed 42 --peers 8 --duration 60s
ze-chaos --binary ./bin/ze --seed 42 --peers 8 --duration 60s
# In-process: mock network + virtual clock, fully deterministic
ze-chaos --in-process --seed 42 --duration 30s
# Pipe mode (legacy): config on stdout, you start ze yourself
ze-chaos --pipe --seed 42 --peers 8 --duration 60s | ./bin/ze -The in-process mode is the one you want in CI. Real network mode is the one you want when you suspect the bug is in the OS network path.
The same chaos scenario can run against FRR (bgpd) or BIRD instead of Ze. ze-chaos generates the matching daemon config and, in fork mode, starts the daemon itself, so one fault sequence can be replayed across different BGP implementations.
# Generate config to inspect (no run)
ze-chaos --config-only --application frr --seed 42 --peers 4
ze-chaos --config-only --application bird --seed 42 --peers 4
# Fork the daemon and run the scenario
ze-chaos --application frr --seed 42 --peers 4 --duration 60s
ze-chaos --application bird --binary /usr/sbin/bird --seed 42 --peers 4 --duration 60sFRR and BIRD use a single BGP port and identify peers by source IP (127.0.0.x), unlike Ze's per-peer port model. On macOS, create loopback aliases (sudo ifconfig lo0 alias 127.0.0.$i) for each peer first; Linux routes 127.0.0.0/8 to loopback by default. The in-tree chaos testing guide covers running FRR or BIRD in Docker and validating generated configs without a live session.
Every chaos run is reproducible from its seed. The same seed and the same rate produce the same fault sequence on every run. That is what makes chaos testing useful for debugging: when the dashboard shows convergence breaking on run 12, you take the seed and you reproduce the failure on demand.
ze --chaos-seed 42 --chaos-rate 0.1 config.conf
ze --chaos-seed 42 --chaos-rate 0.1 config.conf # identical fault sequenceCombined with event logging and replay, this is the part of the chaos surface that pays for itself. You record a failing run, you reduce it to its minimal reproduction with --shrink, and you put the smaller scenario in the test suite.
ze-chaos --event-log run.ndjson --seed 42 | ./bin/ze -
ze-chaos --replay run.ndjson
ze-chaos --shrink run.ndjsonZe-chaos exposes an MCP server that allows AI assistants to configure and run chaos scenarios, inspect results, and make decisions about fault injection. The Watchdog consumer monitors per-family convergence during runs, detecting address-family-specific regression that per-peer convergence checks miss.
ze-chaos --properties all validates five protocol-level invariants while it is running.
| Property | What it checks |
|---|---|
| Convergence deadline | Every route reaches every peer that should see it within the deadline, despite the faults. |
| Route consistency | The RIB matches the peers' adj-RIB-out after recovery. |
| No duplicate routes | No peer receives duplicate announcements for the same prefix. |
| Hold timer enforcement | Hold timers are enforced correctly under fault conditions. |
| Message ordering | Messages are delivered in the correct order. |
Use --convergence-deadline 5s to set the deadline for the convergence check.
Ze-chaos v2 introduced 6 parameterized chaos action types, replacing the flat fault categories. Each action type has configurable parameters for fine-grained fault injection.
| Category | Examples |
|---|---|
| Network I/O | Connection failures, partial writes, read timeouts. |
| Wire parsing | Malformed messages, truncated packets. |
| FSM | State machine violations, invalid transitions. |
| Route processing | Dropped updates, corrupted NLRI. |
| Event delivery | Lost messages, out-of-order events. |
The honest list is shorter than you might expect.
- Before deployment, to make sure your config tolerates peer failures.
- In CI, to catch races and edge cases on every commit.
- When debugging an intermittent failure, to reproduce it with a fixed seed.
- When benchmarking, to measure convergence under fault.
The make targets give you both the unit and functional chaos suites at once.
make ze-chaos-test # Unit + functional
make ze-chaos # Build the orchestrator- In-tree chaos testing guide for the long version.
- Chaos web dashboard architecture for how the dashboard is built.
- Development: building for the make targets that wire chaos into the test suite.
Adapted from main/docs/guide/chaos-testing.md and main/docs/architecture/chaos-web-dashboard.md.