Skip to content

v0.2.0-beta.2 — crash-recovery hardening

Pre-release
Pre-release

Choose a tag to compare

@DebanganThakuria DebanganThakuria released this 08 Jul 11:35
7d4dda6

Two data-loss bugs in crash recovery, found by force-killing pods under live soak traffic and verified fixed the same way. Both share one root cause: a replica restored from a Raft snapshot reads as "caught up" against its own local indexes while its FSM is hours stale. Both fixes share one philosophy: destructive actions require leader confirmation, and every failure mode keeps the data.

Fixed

Startup orphan sweep could delete live topic data (#71)

A freshly restarted node's sweep compared on-disk topic directories against its stale topic set and reclaimed live topics' partition data seconds after boot. A directory is now deleted only when the leader confirms the topic is absent (new OpGetTopic peer RPC). No leader, unreachable leader, non-404 answer — the directory stays.

Fan-out cursors could silently rewind to tail, dropping the delay backlog (#72)

The fan-out runner reconciling against a stale FSM spawned cursors under dead attach epochs, tail-anchored on the epoch mismatch, and overwrote the good offset files — so the post-catch-up cursors re-anchored at the current tail and silently skipped everything pending (measured live: ~2,000 lost delay-child deliveries at 100 msg/s with a 60s delay). Three layers now prevent it:

  • AppliedCaughtUp additionally requires fresh leader contact (raft LastContact ≤ 5s) on followers.
  • The runner skips reconcile passes — spawn and cleanup — until the replica is caught up, and a stopped cursor may only remove an offset file whose epoch it owns.
  • Tail-anchoring (destructive by definition: skips records, overwrites the offset file) requires the leader to confirm the attach epoch; any failure defers the cursor and retries at reconcile cadence. Clean resume from a matching file needs no RPC.

Verification

Force-kill under 300 msg/s soak traffic (parent + fan-out child + 60s delay child, Redis-ledger loss detection): topic dirs kept, cursors resumed from their offset files through a mid-catch-up stale window, delay backlog delivered in full — zero lost messages, OVERDUE=0, only bounded at-least-once duplicates from the in-flight slab.

Upgrade notes: no API, envelope, or on-disk format changes; drop-in upgrade from v0.2.0-beta.1.