testing

Pre-Alpha. This page describes behavior that may change.

Ze has five test suites: unit, functional, fuzz, chaos, and mutation. You run them through the Makefile. Each suite covers a different layer, and they are independent enough that you can run one without the others while you are iterating, and combined enough that make ze-verify runs lint, unit, functional, and ExaBGP tests when you are ready to submit. Mutation testing is advisory and runs separately.

The suites, in order

Unit tests. Go unit tests in *_test.go files alongside the code. These run in milliseconds to seconds and cover the small-scale correctness of every exported function. make ze-unit-test runs them.

Functional tests. .ci files under test/, driven by ze-test. They start real or mocked Ze instances, feed them configuration, drive sessions, and assert against the wire bytes, log lines, and exit codes. make ze-functional-test runs them.

Fuzz tests. Go fuzz targets, one per external-input parser. Peer wire input, config files, command-line arguments, plugin JSON, and every other input from outside the trust boundary has a fuzz harness. make ze-fuzz-test runs them in a loop.

Chaos tests. Deterministic, seed-driven fault injection orchestrated by ze-chaos. Validates protocol-level invariants under fault. make ze-chaos-test runs the unit and functional chaos suites, and chaos web tests.

The Make targets

make ze-unit-test               # Unit tests
make ze-unit-test-cover         # Unit tests with coverage
make ze-functional-test         # Full functional suite
make ze-encode-test             # Encoding-only functional tests
make ze-plugin-test             # Plugin behavior functional tests
make ze-reload-test             # Reload-path functional tests
make ze-parse-test              # Config parsing tests
make ze-fuzz-test               # All fuzz targets, one-shot
make ze-fuzz-one FUZZ=<name> PKG=<path> TIME=<duration>  # A single fuzz target, longer run
make ze-chaos-test              # Unit + functional chaos, and chaos web tests
make ze-mutation-test            # Mutation testing, all non-excluded packages (advisory)
make ze-mutation-changed        # Mutation testing, changed files only (advisory)
make ze-mutation-report         # Mutation testing with HTML report
make ze-mutation-pkg PKG=<path> # Mutation testing, specific package(s)
make ze-test                    # Lint, unit, functional, ExaBGP, and fuzz combined
make ze-verify                  # Lint, unit, functional, and ExaBGP, pre-submission
make ze-verify-changed          # Same as ze-verify, only on packages your branch touched
make ze-interop-test            # Docker interop against FRR, BIRD, GoBGP
make ze-live-test               # Live BGP tests
make ze-stress-test             # Stress suite

make ze-verify-changed is the one you want to run on every save. It is fast enough that it does not break flow, and it catches most problems before you push.

Running specific tests

ze-test drives the functional suite. Listing and running individual tests is a handful of flags.

ze-test bgp encode --list               # List the encode tests
ze-test bgp encode 4 5 6                # Run specific tests by index
ze-test bgp encode --all                # Run all encode tests
ze-test bgp plugin --list
ze-test bgp plugin --all
ze-test bgp reload --all

Stress-testing a flaky test:

ze-test bgp encode --count 10 0 1       # Run tests 0 and 1, ten times each

If a test passes nine times and fails once, it is flaky and it is a bug. Do not land a PR that introduces a flake.

Writing functional tests

Functional tests are .ci files. The format is a small scripting language: stdin=..., cmd=..., expect=... lines, with terminator-delimited blocks for config and expected output.

A minimal parse test:

stdin=config:terminator=EOF_CONF
bgp {
    peer test-peer {
        remote {
            ip 127.0.0.1;
            as 65533;
        }
        router-id 10.0.0.2;
        local-as  65533;
    }
}
EOF_CONF

cmd=foreground:seq=1:exec=ze bgp validate -:stdin=config
expect=exit:code=0

The runner parses the file, executes the commands in order, and asserts the expectations. If an expectation fails, the runner prints a diff showing what it expected and what it saw.

The existing .ci files under test/ are the best reference for how to write a new one. Start with the closest existing test, copy it, and edit.

Exit code 0 is not an assertion

A test whose only assertion is expect=exit:code=0 proves a config was accepted, never that it parsed to the correct tree. A parser that accepts interval 300 but stores 0, or silently drops a block, passes such a test green. Add a readback: assert the value the config produced, not merely that nothing errored.

A lint keeps this class from growing. TestCIAcceptOnlyLint walks every test/**/*.ci and fails on a new accept-only test that is neither strengthened nor annotated. Existing ones are grandfathered in test/.accept-only-baseline, a sorted allow-list that only ever shrinks: strengthening or annotating a test removes its line. A test whose real check lives in a set -e tmpfs script, and any reject= test, are correctly excluded.

Linux-only tests

Two directives matter when a test touches the kernel:

Directive	Effect
option=needs-linux	The test boots a daemon that exercises real kernel features. It skips on non-Linux hosts and runs automatically in the QEMU Alpine VM under make ze-qemu-all-test.
option=needs-linux:caps=net-admin	Also needs privileged network configuration (creating interfaces, bringing links up, netlink). Without the capability the test skips rather than hanging.
option=netns-link:name=<if>[:address=<cidr>]	Provisions an interface inside the per-test network namespace before Ze launches: a dummy link, assigned the CIDR when given, then brought up. Only consumed under netns mode; inert and host-safe otherwise.

netns-link exists because the namespace setup brings up loopback and nothing else, so a test that routes through an interface the daemon never creates has nothing to resolve against. A policy-routing next-hop needs a connected route to resolve its gateway, for example.

"This needs real hardware" is almost never true. A Linux-only test that fails rather than skips on macOS is missing its needs-linux marker, not blocked on the environment.

The same applies to Go unit tests for Linux-only packages. The installer suite is cross-compiled for Linux, so it cannot exec on a macOS host. make ze-unit-test runs it for real only when the host is Linux, and elsewhere type-checks the tag-guarded files with go vet, which compiles the _test.go files without running them. The real execution happens in the Alpine VM under make ze-qemu-integration-test, so the rescue, console, and bootstrap logic is genuinely exercised on every platform's evidence path rather than skipped off Linux. Before that, make ze-unit-test was red on every macOS machine, and with it make all, make ze-test, and make ze-smoke.

Writing fuzz harnesses

func FuzzParseWireUpdate(f *testing.F) {
    // Seed with known-good inputs.
    f.Add([]byte{0x00, 0x18, 0x0a, 0x00, 0x00})

    f.Fuzz(func(t *testing.T, data []byte) {
        // Must not panic, must not infinite-loop.
        _, _ = wireu.ParseUpdate(data)
    })
}

Fuzz seeds are defined inline in fuzz test functions. Adding a known-bad input that used to crash the parser keeps the regression covered.

Interop testing

The interop suite runs Ze against FRR, BIRD, and GoBGP in Docker containers and asserts that every pair negotiates cleanly and exchanges routes correctly. make ze-interop-test runs it. This is the slowest suite, and the one that catches "technically RFC-compliant but nobody else expects that" bugs.

Live testing

make ze-live-test runs tests against real RPKI caches (stayrtr container). It requires network access and external services that may be unavailable. It is optional for development and mandatory before a release.

Mutation testing

Coverage tells you which lines ran. It does not tell you whether the tests would notice if those lines did something different. gomu fills that gap: it rewrites the AST (arithmetic, conditional, logical, bitwise, branch, return value, and error handling operators), runs the test suite against each mutation, and reports which mutations survived.

gomu uses overlay-based execution, so it never modifies source files on disk. It runs mutants in parallel and supports incremental mode against a base branch, which keeps iteration fast on a codebase this size.

make ze-mutation-test           # All non-excluded packages (slow on full repo)
make ze-mutation-changed        # Changed files only, incremental (fast)
make ze-mutation-pkg PKG=./internal/core/textbuf/   # One package
make ze-mutation-report         # Full run with HTML report

Mutation testing is advisory. It never gates ze-verify or CI. A surviving mutant is a signal that a test could be stronger, not a blocking failure. The tool is young (single maintainer, no build tag support), and treating it as a gate would erode trust before a baseline is established.

gomu is vendored in tools.go and runs via go run, so no separate install is needed. Files with custom build tags and cmd/ze/ are excluded via .gomuignore because gomu cannot handle them. Reports land in tmp/ (gitignored). Mutation score history is tracked in test/mutation/history.ndjson, committed alongside the run that produced it.

Tuning: GOMU_WORKERS (default: half the CPU cores), GOMU_TIMEOUT (default: 120s per test), GOMU_THRESHOLD (default: 0%).

Coverage

make ze-unit-test-cover produces a coverage profile. Coverage is a sanity check, not a target: the goal is that every non-trivial code path has a test, not that a percentage number stays above a threshold. The maintainer cares about whether the test actually exercises the logic, not whether the line is counted.

Test health

A suite can grow indefinitely while the share of behaviour it would actually catch a regression in falls, and no count of tests can show that. make ze-test-health regenerates a dashboard that asks whether the testing is correct rather than whether it is large. Every metric on it belongs to one of three questions: would something go red if the code were wrong, what is not covered at all, and what is knowingly broken. Anything that belongs to none of the three is volume and is deliberately left off.

The structural facts on that page (which test files nothing runs, which RFCs have no test pair, every metric's status) are gated by make ze-verify and cannot lag the tree. The volume counters are not gated on purpose: a check that fired on the majority of commits that add a test would be routed around rather than read.

Two ratchets enforce sensitivity from the tree itself:

Ratchet	Fires when	Fix
assert-nothing	A Test* function has no reachable failure call, no assertion-library call, no compile-time assertion, and no panic. It executes code and passes unconditionally.	Add a real assertion, or annotate // test-asserts-nothing: <why> when the oracle is genuinely implicit, such as a must-not-panic smoke test.
tag-orphan	A _test.go build constraint needs a ze_* tag that no go test invocation in the Makefile supplies, so nothing ever builds it.	Add the tag to a go test invocation, or delete the file.

make ze-test-sensitivity-check   # The two ratchets. A ze-verify stage in both modes.
make ze-test-health              # Regenerate the dashboard
make ze-test-health-record       # Append one KPI sample to the history

A test file nothing runs and a test that cannot fail both read as coverage while providing none. That is what these catch.

Pre-submission checklist

Before you submit a patch:

1. make ze-verify-changed is green.
2. make ze-verify on the full tree is green.
3. Any new code has unit tests.
4. Any new user-facing behaviour has a functional test.
5. Any new external-input parser has a fuzz harness.
6. Documentation updates land in the same commit.

If all six are true, run /ze-review-deep in Claude Code and fix whatever it surfaces. Then submit.

See also

• Building for the make targets that produce the binaries.
• Debugging for what to do when a test fails.
• CI for what runs automatically on every PR.
• Performance for the ze-perf BGP propagation benchmark tool.
• Mock Servers for the ze-test built-in mock servers used in functional tests.
• In-tree functional tests for the full .ci reference.

Adapted from main/docs/functional-tests.md and main/Makefile.