chaos tested peering

Pre-Alpha. This page describes behavior that may change. Not for production use.

Set up a peering session and verify that it survives a battery of chaos scenarios. The goal is not to prove Ze is bug-free (it is not) but to give you confidence that a specific configuration holds up under fault conditions you expect to see in production. Flaky TCP, slow peers, congestion, partial writes, FSM edge cases.

The approach: build the peering session with ze-chaos as the mock peer instead of a real one, run a scripted scenario with a fixed seed, and assert that the invariants hold. If the run fails, the NDJSON event log shrinks to a minimal reproduction that goes in the test suite.

The workflow

Six steps, end to end.

1. Build the config. Use the target production config, with the mock peer's details substituted in for the real peer.
2. Run ze-chaos as the peer. Point it at the production config and let it drive the session.
3. Run the test with a fixed seed. Deterministic fault injection.
4. Validate properties. --properties all to check convergence deadline, route consistency, no duplicate routes, hold timer enforcement, and message ordering.
5. Shrink any failure. --shrink on the recorded NDJSON to get a minimal reproduction.
6. Commit the shrunk scenario. The test stays in the suite after you fix the bug.

The scenario

ze-chaos --in-process \
         --seed 42 \
         --duration 60s \
         --peers 1 \
         --families ipv4/unicast,ipv6/unicast \
         --chaos-rate 0.1 \
         --properties all \
         --convergence-deadline 5s \
         --event-log run.ndjson \
         --config-out ze.conf

ze start ze.conf

--in-process runs mock peers against a virtual clock, so the run is fully deterministic. For a real-network variant that exercises the OS network path, drop --in-process and pipe the config into a real Ze binary.

--seed 42 is the reproducibility knob. Same seed, same rate, same fault sequence, every time. If the run fails, the same invocation reproduces the failure.

--chaos-rate 0.1 is the default fault probability. Ten percent of operations are candidate fault points. Tune this to your use case: 0.01 for a stability soak, 0.3 for aggressive stress.

--properties all turns on every validator: convergence-deadline, route-consistency, no-duplicate-routes, hold-timer-enforcement, message-ordering.

--event-log run.ndjson captures every event so you can replay the run later.

What the properties check

Convergence deadline. Every route that should reach the mock peer does reach it within the deadline, despite the faults. A failure here usually means the reactor dropped an update or the forwarding pool congested enough to miss one.

Route consistency. After recovery, the RIB matches the mock peer's Adj-RIB-Out. A failure here usually means an ordering bug in the recovery path.

No duplicate routes. No peer receives duplicate announcements for the same prefix. A failure here usually means a race between withdrawal and announcement.

Hold timer enforcement. Hold timers are enforced correctly under fault conditions. A failure here usually means the FSM is not tracking timers properly.

Message ordering. Messages are delivered in the correct order. A failure here usually means an ordering bug in the event pipeline.

When a run fails

ze-chaos --replay run.ndjson

The replay reproduces the failure deterministically. Once you have confirmed the failure reproduces, shrink the scenario to its minimal reproduction.

ze-chaos --shrink run.ndjson > minimal.ndjson
ze-chaos --replay minimal.ndjson

The shrunk file is the one to commit as a test. It stays in the suite even after the fix.

Integrating with CI

The pattern that keeps chaos tests useful is "one small canned scenario that runs on every PR, plus a longer nightly sweep".

ze-chaos-smoke:
	./bin/ze-chaos --in-process --seed 42 --duration 10s \
	               --properties all --peers 4

ze-chaos-nightly:
	./bin/ze-chaos --in-process --seed $$(date +%s) --duration 10m \
	               --properties all --peers 40 --chaos-rate 0.2

The smoke target runs in seconds and catches regressions in the happy path. The nightly target runs for ten minutes with a time-based seed and finds the "works at low rate, fails at high rate" kind of bug.

Property-specific tuning

A few of the properties have knobs that matter in practice.

• convergence-deadline: the --convergence-deadline flag caps how long a route can take to reach every expected peer. Tighten it to catch slow convergence; relax it if your scenario includes heavy fault rates that naturally extend convergence.
• route-consistency: no knobs. Fails on state divergence regardless of the rate.
• no-duplicate-routes: no knobs. Checks every announced prefix for duplicates.
• hold-timer-enforcement: checks that hold timers fire correctly under fault conditions.
• message-ordering: checks that messages are delivered in the correct order.

What could go wrong

False positives. A convergence-deadline that is too tight will fail under high chaos rates even when the implementation is correct. Start with the default and tune down only after you have a baseline.

Seed regressions. A change in the PRNG or in the fault list can change the fault sequence for a given seed. Commit the shrunk scenarios rather than bare seeds, because the shrunk scenario is self-contained.

Flakes. A property that fails once out of ten runs at the same seed is a flake, and a flake is a bug. Do not mark it "flaky" and move on. Run with --count 10 to reproduce the flake, then debug.

Overfitting. Shrinking a scenario gives you the smallest input that still fails. The fix should address the underlying bug, not the specific input. If your fix makes minimal.ndjson pass but the general case still fails, you fixed the wrong thing.

See also

• Chaos testing index for the orchestrator overview.
• Scenarios for the full flag reference.
• Property verification for the property details.