Skip to content

v0.6.0 — Membership Changes (Feature Complete)

Pre-release
Pre-release

Choose a tag to compare

View all tags
@jamesgober jamesgober released this 08 Jun 11:50
· 8 commits to main since this release
v0.6.0
7afe12d

raft-io v0.6.0 — Membership Changes (Feature Complete)

Reconfigure a running cluster. v0.6.0 adds membership changes — add or remove
a voting server one at a time — and leadership transfer, the last pieces of the
core protocol. With this, raft-io is feature complete: hardening and the
API/protocol freeze are all that remain before 1.0. Building the membership tests
also surfaced and fixed a latent snapshot edge case from v0.5.

What is raft-io?

A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.

What's new in 0.6.0

Add and remove servers, one at a time

The leader reconfigures the cluster with two events:

use raft_io::{Event, RaftConfig, RaftNode};

let mut node = RaftNode::new(RaftConfig::single(1));
while !node.is_leader() {
    let _ = node.step(Event::Tick).unwrap();
}
let _ = node.step(Event::AddServer(2)).unwrap();
assert_eq!(node.members(), &[1, 2]);

A membership change is a special configuration log entry. The node adopts the new
configuration immediately on append — Raft's rule, which combined with
changing one server at a time guarantees the old and new quorums always overlap,
so no two disjoint majorities can form. Only one change is in flight at a time: a
request made while a previous change is still uncommitted returns
Error::ConfigInProgress. A leader removed from the configuration steps down once
its removal commits. Action::MembershipChanged tells the application the new
membership so it can update its transport.

The configuration survives everything that the log does: it is recovered from the
log on restart, carried in snapshots (Snapshot.config) so a node catching up via
InstallSnapshot still knows the membership, and persisted by WalLog.

Leadership transfer

use raft_io::{Event, RaftNode, RaftConfig};

# let mut leader = RaftNode::new(RaftConfig::new(1, [2, 3]));
// On the leader:
let _ = leader.step(Event::TransferLeadership(2));

The leader brings the target fully up to date, then sends it the new TimeoutNow
message so it campaigns immediately and takes over with minimal disruption, rather
than waiting out an election timeout.

Leader stickiness — no disruption from removed servers

A removed or partitioned server stops hearing heartbeats, times out, and would
otherwise keep forcing elections with ever-higher terms. v0.6 implements the Raft
§4.2.3 mitigation: a node ignores a RequestVote (not even adopting its term)
while a leader it recognises is still active. A genuine new leader still displaces
the old one through AppendEntries; only vote solicitations are suppressed. A
leadership transfer sets a force flag on its vote request to bypass stickiness,
so an authorised hand-off is never blocked.

A latent bug, fixed

Writing the membership churn proptest stressed scheduling enough to surface a
snapshot edge case present since v0.5: a follower that had already caught up past
an InstallSnapshot's index (via normal replication) would install the older
snapshot and reset its applied state backwards. The follower now installs a
snapshot only when it actually advances beyond what it already holds, and otherwise
acknowledges the index it already covers.

New surface

  • Event::AddServer, Event::RemoveServer, Event::TransferLeadership.
  • Action::MembershipChanged; RaftNode::members.
  • Message::TimeoutNow; RequestVote.force.
  • EntryKind (Normal / Config) on LogEntry, with LogEntry::config /
    LogEntry::members; Snapshot.config with Snapshot::with_config.
  • Error::ConfigInProgress.
  • membership example.

Breaking changes

Pre-1.0 shape changes (the wire/trait surface freezes at v0.7):

  • LogEntry gains a kind field, RequestVote a force field, and Snapshot
    a config field. Code that constructs these with struct literals must set the
    new field (use EntryKind::Normal / false / an empty config for the prior
    behaviour, or the LogEntry::new / Snapshot::new constructors which do).
  • Message gains a TimeoutNow variant; Event and Action gain the
    membership and transfer variants (Action is #[non_exhaustive]).

Verification

Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2 Ubuntu)
and via the CI matrix:

cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo bench

All green. Test counts at this tag:

  • --all-features: 100 unit + 16 integration / property tests
    (safety.rs, replication.rs, recovery.rs, snapshot.rs, membership.rs)
    • 52 doctests.

Property suites were additionally run at elevated case counts
(PROPTEST_CASES=1500+) with no failures. loom is still not exercised: the core
is a single-threaded, owned state machine with no lock-free or shared-state path.

What's next

  • v0.7.0 — Hardening + API freeze. Jepsen-style adversarial scheduling, fuzzing
    the decode path, cross-platform verification of the persistent path, a normative
    docs/PROTOCOL.md, and freezing the public traits and wire protocol.

Installation

[dependencies]
raft-io = "0.6"

# Optional features:
raft-io = { version = "0.6", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.6", features = ["framing"] }     # pack-io wire framing

MSRV: Rust 1.85 (edition 2024).

Documentation

Full diff: v0.5.0...v0.6.0.
Changelog: CHANGELOG.md.

Assets 2
Loading