v0.2.0-beta.3 — chaos-hardened durability + cluster scale-out

Everything in this release came out of chaos testing the previous beta on a live cluster under continuous soak traffic: force-kills of followers, leaders, two nodes at once, and kills mid-rolling-restart. Two more members of the stale-replica bug family were found and fixed, the last known disk-hygiene follow-up shipped, and the cluster can now scale out.

Fixed

Dispatcher could discard accepted records on a stale replica view (#73)

The produce dispatcher discarded WAL records — 202-acked, durable, undelivered — when their topic was absent from the local metastore replica. A replica restored from an old Raft snapshot is missing every topic created after the snapshot point, so a restarted node could permanently destroy accepted records for live topics. Discard now requires local absence and a caught-up replica and the leader confirming the topic is gone; every failure mode keeps the records.

A freshly elected leader could trust its still-replaying FSM (#74)

The subtlest one, caught by a simultaneous two-pod kill: election guarantees a new leader's log is complete — not that its FSM has applied it. The node that won the election after a force-kill confirmed a dead fan-out attach epoch from its own snapshot-restored FSM, tail-anchored its cursors, and silently skipped ~2,100 due deliveries; the follower that restarted alongside it asked the real leader, was refused, and lost nothing. Every "I am the leader, my state is authority" shortcut now runs a Raft Barrier and re-reads local state after it. The fan-out cursor-file sweep also now requires leader confirmation per removal, and consumer offsets recover lazily from the per-partition file (disk is ground truth) instead of a boot-time scan over possibly-stale assignments.

Added

Ingress WAL auto-reclaim (#73)

A fully-dispatched active WAL segment was pinned on disk until new appends overflowed it — never, once its topics went quiet (observed: 5–51 MB of dead WAL per node). Compaction now rotates a fully-dispatched active segment past a 1 MiB floor and reclaims it at the normal checkpoint cadence; recovery preserves the sequence space through an empty rotated log. No manual cleanup, no startup pass.

Cluster scale-out (#75, #76)

Raft membership was bootstrap-only; scaling the StatefulSet produced phantom clusters. Now: initialClusterSize (set once) gates bootstrap, any other fresh node starts join-only — empty Raft configuration, walks its peers with the new OpJoinCluster RPC until the leader admits it via AddVoter, and holds readiness until admitted so an unjoined node never receives traffic. Scaling out is --set replicaCount=N. Existing partitions are never rebalanced (single-owner design); new nodes take assignments for topics created after they join. Verified live 3→5: admission in seconds, perfect round-robin spread of a new topic, and a leader+new-node double-kill at 5 nodes held quorum with zero message loss.

Verification

Chaos matrix on a live cluster at 100 msg/s × 3 paths (direct, fan-out child, 60s delay child) with a Redis-ledger harness detecting any lost message: leader kill, child-partition-owner kill, two-node quorum loss (three runs), kill during rolling restart, 3→5 scale-out, and a five-node double-kill — zero lost messages end-state in every scenario, bounded at-least-once duplicates only.

Upgrade notes: no API or on-disk format changes; drop-in from v0.2.0-beta.2. To enable scale-out on an existing install, upgrade the chart (adds NARAD_CLUSTER_INITIAL_MEMBERS; default initialClusterSize: 3 matches existing clusters), then raise replicaCount. Scale-down/decommission is not yet supported.