Skip to content

Narad v2.0.0 — Partition Rebalance & Node Decommission

Choose a tag to compare

View all tags
@DebanganThakuria DebanganThakuria released this 19 Jul 18:55
· 27 commits to master since this release
v2.0.0
f817221
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The headline feature: the cluster can now rebalance and shrink on its own. New nodes automatically absorb existing partitions, and nodes can be drained and removed — with no record loss.

What's new

Auto-rebalance on scale-out. When a node joins, the leader computes the minimal set of partition moves that re-balances owned-partition count and relocates them — copying each partition verbatim (same offsets, HWM, consumer position) to its new owner and cutting over with a millisecond freeze at the very end. Watch it with narad cluster moves / narad cluster members.

Node decommission. narad cluster decommission <node> drains a node's partitions onto the others and removes it from the Raft voter set, behind two guards: never drop below 3 voters, and transfer leadership away before removing a leader.

Last-moment cutover, never a long freeze. The copy is two-phase — a freeze-free bulk catch-up (a GB partition copies with produce still flowing), then a millisecond freeze for the final tail. Bounded pre-copy: a partition written faster than it copies still cuts over (stop-and-copy fallback), never loops forever.

Force-promote. If a source node dies mid-move and stays dead, a caught-up destination promotes its copy rather than stalling — strictly gated so it never exposes a truncated partition.

Operator surface. POST/DELETE /v1/cluster/members/{id}/decommission, GET /v1/cluster/moves, GET /v1/cluster/members, and the narad cluster CLI.

Validated on live infrastructure

  • 23,728 / 23,728 records durable through an extreme-chaos soak (scale-up rebalance + full rolling restart + decommission + pod kill + forced recovery, under load).
  • 53,824 / 53,824 records durable, zero produce rejections through a clean decommission drain under sustained load.
  • Replay-audit methodology (durability, not consumption); 1,637 unit/integration tests.

Docs

Rebalance & Decommission internals · Scaling & Recovery runbook

Upgrading: drop-in — no data migration, no config changes. Rebalance is automatic; decommission is operator-driven.

Assets 2
Loading