Releases: jamesgober/raft-io
Release list
v1.0.0 — Stable API
raft-io v1.0.0 — Stable
A complete, frozen Raft. From the first scaffold, raft-io was built to be the
consensus engine three projects could stake their correctness on. With 1.0 the
protocol is complete, every safety property is proven under adversarial fault
injection, the performance baseline is set, and the public API, wire format, and
durable log format are frozen — no backward-incompatible change before 2.0.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.
The complete protocol
Everything Raft needs to run a real replicated system, and nothing it does not:
- Leader election with term and vote safety, randomized timeouts, leader
stickiness, and pre-vote so a partitioned node cannot disrupt a healthy
cluster when it rejoins. - Log replication — batched
AppendEntries, per-follower progress with
optimistic pipelining, conflict-hint backtracking, and commit on a quorum. - Durable crash recovery — term, vote, and log persisted before each RPC; a
restarted node recovers and rejoins without violating safety (theWalLog,
under thepersistencefeature). - Snapshots with log compaction and
InstallSnapshotcatch-up for a
far-behind follower, driven by a snapshot-policy hint. - Membership changes — single-server add/remove with safe sequencing, and
non-voting learners that catch up without affecting any quorum, then get
promoted to voters. - Leadership transfer — hand off to a caught-up peer with
TimeoutNow. - Linearizable reads — the ReadIndex protocol: a read that reflects every
committed write, confirmed against a quorum, with no log append.
Proven, not just written
- A kitchen-sink adversarial suite turns every fault mode on at once —
partitions, message loss, reordering, duplication, membership churn, and
snapshotting under one randomised schedule — and asserts all five Raft safety
properties continuously: Election Safety, Leader Append-Only, Log Matching,
Leader Completeness / State Machine Safety, and apply ordering. Run sustained to
PROPTEST_CASES=6000+. - An application-level suite drives a replicated key-value store to identical
state on every node — and serves stale-free linearizable reads — under the same
faults. - The decode path is fuzzed: arbitrary bytes off the wire or off disk decode
to a valid value or fail cleanly, never panicking or over-allocating. - Determinism: no wall clock, no I/O in the core; an entire cluster run is
reproducible from a seed and a sequence of events.
Built to a standard
#![forbid(unsafe_code)]; nounwrap/expect/panic/todoon any production
path (enforced bydenylints); every fallible operation returnsResult.- Allocation-free steady-state hot paths; committed performance baselines for
every shape ofstepindocs/BENCHMARKS.md(a follower tick
is ~8 ns). - A three-tier API: trivial single-node use needs no generic to name; a builder
tunes timing; traits plug in real storage and transport. - The full surface is documented item-by-item in
docs/API.md, and the
protocol is specified normatively indocs/PROTOCOL.md.
The freeze
As of 1.0, the public API, the Message set and its pack-io framing, the
WalLog record format, and the configuration encoding are frozen and will not
change incompatibly before 2.0. Future additions stay compatible through
#[non_exhaustive] enums and tagged encodings.
Breaking changes
None versus 0.10.1. This release is documentation, an added example, and the
final specification pass; behaviour is unchanged.
Verification
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. At this tag: 122 unit + 22 integration / property tests + 60 doctests,
nine runnable examples, and the hardening suite soaked at PROPTEST_CASES=6000.
MSRV: Rust 1.85 (edition 2024). loom is not exercised: the core is a
single-threaded, owned state machine with no lock-free or shared-state path.
Installation
[dependencies]
raft-io = "1.0"
# Optional features:
raft-io = { version = "1.0", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "1.0", features = ["framing"] } # pack-io wire framingDocumentation
Full diff: v0.10.1...v1.0.0.
Changelog: CHANGELOG.md.
v0.10.1 — RC hardening
raft-io v0.10.1 — RC hardening: benchmarks + soak
Numbers, and confidence. With the protocol feature-complete as of v0.10.0,
v0.10.1 is the first run-up to a 1.0: a committed performance baseline across
every hot path, and an extended soak of the property suite. No library code or
public API changed — this is tooling, measurement, and documentation.
What's new in 0.10.1
A committed performance baseline
RaftNode::step is the single entry point the whole system funnels through, so it
is the path whose cost matters. The criterion suite (benches/raft_bench.rs) now
covers every shape of step that a running cluster exercises — not just the three
it started with:
- steady state: the follower clock tick (~8 ns, allocation-free) and the leader
heartbeat tick (~19 ns); - elections: granting an inbound vote (~93 ns, persists the vote);
- reads: releasing a linearizable read via ReadIndex on a single-node leader
(~165 ns); - snapshots: a follower installing a leader's snapshot (~170 ns);
- replication: a leader committing on a follower's acknowledgement (~188 ns) and a
follower applying a batch of eight entries (~820 ns); - proposals: a single-node append + commit + apply (~205 ns).
Because the core is sans-I/O, these are pure protocol costs — no clock, socket,
or disk — so they are stable and reproducible. docs/BENCHMARKS.md records the
reference numbers, the methodology, and how to gate regressions; REPS treats a
regression beyond 5% on a tracked metric as a blocker.
Extended soak
The full property suite was run at elevated case counts with no failure: the
jepsen-style hardening nemesis (every fault mode at once, all five safety
properties asserted) at PROPTEST_CASES=6000, and the whole suite — replication,
recovery, snapshots, membership churn, and the key-value consumer (convergence and
stale-free linearizable reads) — at PROPTEST_CASES=2000.
Breaking changes
None. This release adds benchmarks, a soak, and documentation only.
Verification
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts are unchanged from v0.10.0 (122 unit + 25 integration /
property + 60 doctests). MSRV: Rust 1.85 (edition 2024).
What's next
- 1.0. The protocol is complete and the performance baseline is set. What
remains is a final pass over the docs and the protocol specification, and the
1.0freeze.
Installation
[dependencies]
raft-io = "0.10"Documentation
Full diff: v0.10.0...v0.10.1.
Changelog: CHANGELOG.md.
v0.10.0 — Beta: learner members
raft-io v0.10.0 — Beta: learner members
Grow the cluster without holding your breath. v0.10.0 adds non-voting
learner members: a node you can add to the cluster that replicates the log and
catches up like a follower but counts toward no quorum until you promote it. So
adding a far-behind node — even one that has to pull a whole snapshot — never
shrinks your fault tolerance or stalls commits while it catches up. Everything
here is MINOR-compatible, and the on-disk encoding is byte-identical to v0.9 for
any cluster that uses no learners.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.
What's new in 0.10.0
Learner (non-voting) members
Adding a fresh voter to a cluster has always carried a risk: the new member counts
toward the quorum the moment it joins, but it starts with an empty log. Until it
catches up, every commit and election needs a majority that now includes a node
with nothing in it — so a single other failure can wedge the cluster. The standard
remedy (Raft thesis §4.2.1) is to add the node as a learner first.
A learner:
- receives the replicated log —
AppendEntries, andInstallSnapshotwhen it
is far enough behind — and applies committed entries, exactly like a follower; - counts toward no quorum — not commit, not elections, not linearizable-read
confirmation — so the voter majority is unchanged while it catches up; - never campaigns — a node that finds itself listed as a learner follows
quietly and starts no elections, so it cannot disrupt the cluster.
The workflow:
use raft_io::Event;
// On the leader: add a far-behind node as a learner. The voter set — and the
// quorum — is unchanged, so availability is not affected while it catches up.
let _ = node.step(Event::AddLearner(4))?;
// ... the leader replicates the backlog (and a snapshot if needed) to node 4;
// you can watch its progress. Once it has caught up:
let _ = node.step(Event::PromoteLearner(4))?; // now a full voting member
# Ok::<(), raft_io::Error>(())Event::RemoveServer drops a learner just as it drops a voter, and the new
RaftNode::learners() accessor reports the current learner set (members()
continues to report only voters). Promotion and addition follow the same
one-change-at-a-time rule as every other membership change.
Byte-compatible encoding — the freeze holds
Learners are durable, replicated cluster state, so they ride in the same
configuration entries (and snapshot membership) as voters. The encoding extends
the existing format with a reserved sentinel (u64::MAX, never a valid node id)
that separates voters from learners — and only appears when learners are
present. A cluster that uses no learners produces configuration entries and
snapshots that are byte-for-byte identical to v0.9, so existing logs, snapshots,
and deployments are wholly unaffected and the v0.7 format freeze holds.
Verified end to end
A new tests/membership.rs scenario adds a learner to a running cluster, confirms
the voter set and quorum are unchanged, keeps committing proposals while the
learner catches up (proving availability is never stalled), then promotes the
caught-up learner and checks the whole cluster — now four voters — agrees. Unit
tests cover that a learner never forms a commit quorum, never campaigns, is
recovered from a snapshot's configuration, and that promotion and removal behave.
Breaking changes
None. Event::AddLearner / Event::PromoteLearner and RaftNode::learners()
are additive; Event::RemoveServer gains the ability to remove a learner but is
otherwise unchanged; and the configuration encoding is identical to v0.9 unless a
learner is actually present. Existing code, wire bytes, and WAL records are
unaffected.
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2 Ubuntu)
and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 124 unit + 25 integration / property tests
(safety.rs,replication.rs,recovery.rs,snapshot.rs,membership.rs,
hardening.rs,kv_consumer.rs) + 60 doctests.
The hardening suite was additionally run at PROPTEST_CASES=6000 with no failures
— adding the learner split between voting and non-voting replicas disturbed no
safety property. loom is not exercised: the core is a single-threaded, owned
state machine with no lock-free or shared-state path.
What's next
- RC → 1.0. With the protocol now complete — election, replication,
persistence, snapshots, membership with learners, leadership transfer, and
linearizable reads — what remains is the run to 1.0: committed benchmark
baselines for the hot paths, an extended soak, a final pass over the docs and the
protocol spec, and the 1.0 freeze.
Installation
[dependencies]
raft-io = "0.10"
# Optional features:
raft-io = { version = "0.10", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.10", features = ["framing"] } # pack-io wire framingMSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.9.0...v0.10.0.
Changelog: CHANGELOG.md.
v0.9.0 — Beta
raft-io v0.9.0 — Beta: linearizable reads
Reads you can trust. v0.9.0 adds the one capability a database-grade Raft
still needed: linearizable reads via the ReadIndex protocol. A leader now
answers a read only after confirming it still leads a quorum and has applied
through the read point — so a read never returns stale state from a leader that
has silently lost its mandate. Everything here is MINOR-compatible: no existing
signature, wire encoding, or WAL record changed.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.
What's new in 0.9.0
Linearizable reads (the ReadIndex protocol)
Through v0.8 a leader could only serve a read from whatever it happened to have
applied, with no guarantee it was still the leader — a node deposed by a partition
it had not yet noticed could answer with stale state. That is exactly the caveat
the v0.8 notes flagged. v0.9 closes it.
Ask for a read with the new event, carrying a caller token:
use raft_io::{Action, Event};
// On the leader: request a linearizable read.
for action in node.step(Event::Read { id: 7 })? {
if let Action::ReadReady { id, index } = action {
// The state machine now reflects every command committed before this
// read was requested. Read it and answer the client holding `id`.
let _ = (id, index);
}
}
# Ok::<(), raft_io::Error>(())Under the hood this is the ReadIndex protocol (Raft thesis §6.4):
- The leader records its commit index as the read point.
- It confirms it is still the leader by exchanging a
ReadProberound with a
quorum — a deposed leader cannot complete this, so it never answers stale. - Once a quorum has acknowledged and the leader has applied through the read
point, it emitsAction::ReadReady { id, index }.
The read is served entirely off the existing log — no entry is appended — so
reads do not grow the log or compete with writes for replication. If leadership is
lost before confirmation the read is simply dropped (no action is emitted) and the
client retries against the new leader. On a single-node cluster the leader is its
own quorum, so a read is ready immediately.
The confirmation round rides on two new messages, Message::ReadProbe and
Message::ReadProbeReply. Like v0.8's pre-vote messages, they are additive
#[non_exhaustive] enum variants — every existing message keeps its exact wire
encoding, so the v0.7 freeze holds.
A no-op on election, for an accurate commit index
ReadIndex is only correct if the leader's commit index truly reflects everything
committed before it took over. A freshly elected leader that inherited an
uncommitted tail of earlier-term entries cannot prove those committed by replica
count alone (§5.4.2). So, following the thesis (§6.4 / §8), a new leader that took
over with such a tail now appends a no-op entry of its own term; committing it
carries the earlier entries over the commit line and pins down an accurate commit
index.
The no-op is represented as a configuration entry that re-asserts the current
membership — which means it is never surfaced to the application, exactly like a
real membership entry. One consequence, shared with membership entries and worth
stating plainly: applied indices increase strictly but are not contiguous. A
consumer keys its applied state by the index in each Action::Apply, never by
counting applies. (The bundled WalLog and the example state machines already do
this; the recovery test harness was updated to match.)
Verified end to end
tests/kv_consumer.rs now issues a linearizable read on the healed cluster after
its full partition/snapshot fault schedule and asserts the read is never stale —
its read index always covers everything committed before the request. The
deterministic suites additionally check that a read after a partition heal reflects
the majority's writes, and that a read on a quiesced cluster returns exactly the
committed model. examples/kv_store.rs demonstrates a linearizable read of a key.
Breaking changes
None. The read-probe messages and Event::Read / Action::ReadReady are
additive; the election no-op changes no public signature and is invisible to the
application. Existing code, wire bytes, and WAL records are unaffected. The one
behavioural note — non-contiguous applied indices — already held for any cluster
that used membership changes.
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2 Ubuntu)
and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 112 unit + 24 integration / property tests
(safety.rs,replication.rs,recovery.rs,snapshot.rs,membership.rs,
hardening.rs,kv_consumer.rs) + 57 doctests.
The hardening suite was additionally run at PROPTEST_CASES=6000 and the
key-value consumer suite at PROPTEST_CASES=2000, both with no failures — the
election no-op did not disturb any safety property. loom is not exercised: the
core is a single-threaded, owned state machine with no lock-free or shared-state
path.
What's next
- v0.9.1+ → RC → 1.0. Bug fixes only, broader soak, and final committed
benchmarks toward the 1.0 freeze. A remaining candidate addition is witness /
learner (non-voting) members for safe catch-up before promotion.
Installation
[dependencies]
raft-io = "0.9"
# Optional features:
raft-io = { version = "0.9", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.9", features = ["framing"] } # pack-io wire framingMSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.8.0...v0.9.0.
Changelog: CHANGELOG.md.
v0.8.0 — Alpha: first consumer, and pre-vote
raft-io v0.8.0 — Alpha: first consumer, and pre-vote
The first real consumer, and the liveness fix it surfaced. v0.8.0 builds a
replicated key-value store on the core — end to end, with snapshots and faults —
and turns it into a property test. That test found a real disruption corner, and
the fix is pre-vote elections. Everything here is MINOR-compatible: no
existing signature, wire encoding, or WAL record changed.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.
What's new in 0.8.0
Pre-vote elections — no disruption on rejoin
A node partitioned away from the cluster keeps timing out and, under plain Raft,
keeps incrementing its term. When it rejoins, that inflated term forces the
sitting leader to step down — even though the rejoining node is behind and cannot
win — and the cluster churns through elections before it re-settles. This is the
classic rejoin-disruption problem, and the standard remedy is pre-vote (Raft
thesis §9.6).
A node now runs a pre-vote round before a real election. It asks each peer
whether it would grant a vote at the next term — without incrementing its own
term or casting a vote — and a peer grants only if it has no active leader and
the candidate's log is at least as up to date as its own. The node campaigns for
real, finally advancing its term, only once a quorum of pre-votes agrees. A
partitioned node never collects that quorum, so its term never climbs; on rejoin
it slots back in behind the existing leader without disrupting it.
The probe rides on two new messages, Message::PreVote and
Message::PreVoteReply. They are additive #[non_exhaustive] enum variants —
every existing message keeps its exact wire encoding, so the freeze declared in
v0.7 holds. TimeoutNow (leadership transfer) still triggers an immediate real
election, bypassing pre-vote, because the leader has already vouched for the
target.
A real consumer: a replicated key-value store
examples/kv_store.rs is the library's first end-to-end consumer. The application
supplies a KvStore state machine and an in-memory transport and drives the node
with step; committed commands arrive as Action::Apply and are decoded and
applied, and the snapshot hooks serialize and restore the store. The demo elects a
leader, replicates a series of writes so every node converges, then adds a fourth
node that catches up entirely from a snapshot and ends with the identical state.
Application-level convergence as a property
tests/kv_consumer.rs lifts that consumer into a property test. Where the
protocol suites check that committed commands never diverge, this checks the
layer an application actually cares about: the materialized state machine. A
key-value store is driven through writes, partitions, and snapshotting; once the
cluster heals and settles, every node's map must equal a single-threaded model
built by applying the committed command sequence in order — including any node
whose state was rebuilt from a snapshot.
This is the test that surfaced the rejoin disruption: under heavy fault injection
it caught a cluster that would not re-converge promptly after a partition healed,
because a term-inflated node kept unseating the leader. Pre-vote fixes it; the
test now passes at elevated case counts (PROPTEST_CASES=4000).
A prelude
use raft_io::prelude::*; now brings in the everyday surface in one line —
RaftNode, RaftConfig, the Event/Action vocabulary, Error/Result, and
the RaftLog/RaftTransport seams with their in-memory implementations. The
message and other value types remain at the crate root for when you implement a
transport or inspect a LogEntry.
A note on reads
A leader serves application state from what it has applied. raft-io does not (yet)
implement read-index or lease-based linearizable reads, so a client that reads
from a node which has just lost leadership without knowing it could observe stale
state. For strongly-consistent reads today, route them through the log as
commands; first-class linearizable reads are a candidate for a later milestone.
Breaking changes
None. The pre-vote messages are additive enum variants, the prelude is new,
and the example and test are additions. Existing code, wire bytes, and WAL records
are unaffected.
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2 Ubuntu)
and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 107 unit + 22 integration / property tests
(safety.rs,replication.rs,recovery.rs,snapshot.rs,membership.rs,
hardening.rs,kv_consumer.rs) + 55 doctests.
The hardening suite was additionally run at PROPTEST_CASES=6000 and the
key-value consumer suite at PROPTEST_CASES=4000, both with no failures. loom
is not exercised: the core is a single-threaded, owned state machine with no
lock-free or shared-state path.
What's next
- v0.9.x — Beta → RC. Broaden consumer integration, capture final benchmarks,
and soak toward the 1.0 freeze. Candidate hardening items include linearizable
reads (read-index / leader lease) and witness/learner (non-voting) members.
Installation
[dependencies]
raft-io = "0.8"
# Optional features:
raft-io = { version = "0.8", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.8", features = ["framing"] } # pack-io wire framingMSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.7.0...v0.8.0.
Changelog: CHANGELOG.md.
v0.7.0 — Hardening + API/Protocol Freeze
raft-io v0.7.0 — Hardening + API/Protocol Freeze
Proven, and frozen. v0.7.0 adds no features — it proves the ones already
there. A single kitchen-sink test now turns every fault mode on at once and
asserts all five Raft safety properties under sustained, adversarial runs; the
decode path is fuzzed; and the public traits and the wire and WAL formats are
frozen behind a normative specification. From here to 1.0 it is alpha/beta
soak and bug fixes only.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.
What's new in 0.7.0
One harness, every fault, every safety property
tests/hardening.rs is a jepsen-style nemesis: a single randomised schedule that
combines partitions, message loss, reordering, duplication, membership churn,
and snapshotting on one cluster, and after every step asserts the complete set
of Raft safety properties:
- Election Safety — at most one leader per term.
- Leader Append-Only — a leader never overwrites or deletes an entry in its
own log; it only appends. - Log Matching — if two logs hold an entry with the same index and term, the
logs agree on every entry up through that index. - Leader Completeness / State Machine Safety — no two nodes ever apply a
different command at the same index, across leader changes, snapshots, and
reconfiguration. - Apply ordering — each node applies in strictly increasing index order.
The earlier suites each stressed one dimension and checked a subset of these; this
turns them all on together and adds the two properties they did not check
explicitly — Leader Append-Only and Log Matching. It has been run sustained
(PROPTEST_CASES=8000) with no violation.
The decode path is fuzzed
The only untrusted input a node sees is bytes off the wire. Both decoders —
framing::decode and the WalLog record decoder — are now covered by proptest
no-panic properties that run cross-platform in the default suite: arbitrary bytes
must decode to a valid value or fail cleanly, never panic, and anything that
decodes must re-encode to identical bytes (the wire format is canonical). A
cargo-fuzz target in fuzz/ provides coverage-guided fuzzing on nightly.
A normative protocol specification
docs/PROTOCOL.md specifies the protocol in RFC-2119 terms: the state model, the
message set and their semantics, the pack-io wire framing, the durable WAL
record format (byte-level), the snapshot and membership-change rules, and the five
safety invariants. It is the contract a second implementation would honour to
interoperate.
Freeze
The public traits (RaftLog, RaftTransport), the Message set and its framing,
and the WalLog record format are frozen as of v0.7 — no backward-incompatible
change before 2.0. Future additions stay compatible via the #[non_exhaustive]
enums and tagged encodings. Cross-platform verification of the persistent path
runs on the Linux / macOS / Windows CI matrix.
Breaking changes
None. This release is hardening, tests, and documentation; the public API is
unchanged from v0.6 and is now frozen.
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2 Ubuntu)
and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 103 unit + 17 integration / property tests
(safety.rs,replication.rs,recovery.rs,snapshot.rs,membership.rs,
hardening.rs) + 52 doctests.
The hardening suite was additionally run at PROPTEST_CASES=8000 with no failures.
loom is not exercised: the core is a single-threaded, owned state machine with no
lock-free or shared-state path.
What's next
- v0.8.0 → v0.9.x — Alpha / Beta → RC. Integrate against the first real
consumers and fix what they surface (MINOR-compatible additions only; no
breaking signatures), broaden testing, capture final benchmarks, and soak toward
the 1.0 freeze.
Installation
[dependencies]
raft-io = "0.7"
# Optional features:
raft-io = { version = "0.7", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.7", features = ["framing"] } # pack-io wire framingMSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.6.0...v0.7.0.
Changelog: CHANGELOG.md.
v0.6.0 — Membership Changes (Feature Complete)
raft-io v0.6.0 — Membership Changes (Feature Complete)
Reconfigure a running cluster. v0.6.0 adds membership changes — add or remove
a voting server one at a time — and leadership transfer, the last pieces of the
core protocol. With this, raft-io is feature complete: hardening and the
API/protocol freeze are all that remain before 1.0. Building the membership tests
also surfaced and fixed a latent snapshot edge case from v0.5.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events and it returns actions. Time,
networking, and storage are injected through trait seams, which is what makes the
core provable and reproducible from a seed. It is the consensus layer above
wal-db and the coordination substrate for Hive DB clustering.
What's new in 0.6.0
Add and remove servers, one at a time
The leader reconfigures the cluster with two events:
use raft_io::{Event, RaftConfig, RaftNode};
let mut node = RaftNode::new(RaftConfig::single(1));
while !node.is_leader() {
let _ = node.step(Event::Tick).unwrap();
}
let _ = node.step(Event::AddServer(2)).unwrap();
assert_eq!(node.members(), &[1, 2]);A membership change is a special configuration log entry. The node adopts the new
configuration immediately on append — Raft's rule, which combined with
changing one server at a time guarantees the old and new quorums always overlap,
so no two disjoint majorities can form. Only one change is in flight at a time: a
request made while a previous change is still uncommitted returns
Error::ConfigInProgress. A leader removed from the configuration steps down once
its removal commits. Action::MembershipChanged tells the application the new
membership so it can update its transport.
The configuration survives everything that the log does: it is recovered from the
log on restart, carried in snapshots (Snapshot.config) so a node catching up via
InstallSnapshot still knows the membership, and persisted by WalLog.
Leadership transfer
use raft_io::{Event, RaftNode, RaftConfig};
# let mut leader = RaftNode::new(RaftConfig::new(1, [2, 3]));
// On the leader:
let _ = leader.step(Event::TransferLeadership(2));The leader brings the target fully up to date, then sends it the new TimeoutNow
message so it campaigns immediately and takes over with minimal disruption, rather
than waiting out an election timeout.
Leader stickiness — no disruption from removed servers
A removed or partitioned server stops hearing heartbeats, times out, and would
otherwise keep forcing elections with ever-higher terms. v0.6 implements the Raft
§4.2.3 mitigation: a node ignores a RequestVote (not even adopting its term)
while a leader it recognises is still active. A genuine new leader still displaces
the old one through AppendEntries; only vote solicitations are suppressed. A
leadership transfer sets a force flag on its vote request to bypass stickiness,
so an authorised hand-off is never blocked.
A latent bug, fixed
Writing the membership churn proptest stressed scheduling enough to surface a
snapshot edge case present since v0.5: a follower that had already caught up past
an InstallSnapshot's index (via normal replication) would install the older
snapshot and reset its applied state backwards. The follower now installs a
snapshot only when it actually advances beyond what it already holds, and otherwise
acknowledges the index it already covers.
New surface
Event::AddServer,Event::RemoveServer,Event::TransferLeadership.Action::MembershipChanged;RaftNode::members.Message::TimeoutNow;RequestVote.force.EntryKind(Normal/Config) onLogEntry, withLogEntry::config/
LogEntry::members;Snapshot.configwithSnapshot::with_config.Error::ConfigInProgress.membershipexample.
Breaking changes
Pre-1.0 shape changes (the wire/trait surface freezes at v0.7):
LogEntrygains akindfield,RequestVoteaforcefield, andSnapshot
aconfigfield. Code that constructs these with struct literals must set the
new field (useEntryKind::Normal/false/ an empty config for the prior
behaviour, or theLogEntry::new/Snapshot::newconstructors which do).Messagegains aTimeoutNowvariant;EventandActiongain the
membership and transfer variants (Actionis#[non_exhaustive]).
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2 Ubuntu)
and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 100 unit + 16 integration / property tests
(safety.rs,replication.rs,recovery.rs,snapshot.rs,membership.rs)- 52 doctests.
Property suites were additionally run at elevated case counts
(PROPTEST_CASES=1500+) with no failures. loom is still not exercised: the core
is a single-threaded, owned state machine with no lock-free or shared-state path.
What's next
- v0.7.0 — Hardening + API freeze. Jepsen-style adversarial scheduling, fuzzing
the decode path, cross-platform verification of the persistent path, a normative
docs/PROTOCOL.md, and freezing the public traits and wire protocol.
Installation
[dependencies]
raft-io = "0.6"
# Optional features:
raft-io = { version = "0.6", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.6", features = ["framing"] } # pack-io wire framingMSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.5.0...v0.6.0.
Changelog: CHANGELOG.md.
v0.5.0 — Snapshots + Log Compaction
raft-io v0.5.0 — Snapshots + Log Compaction
Bounded log growth and fast catch-up. v0.5.0 adds snapshots: a node can
capture its state machine, compact the log behind it, and ship that snapshot to a
follower too far behind to replicate entry by entry. With this the protocol is
feature-complete except for membership changes (v0.6). A separate framing
feature adds pack-io wire encoding for messages. A property test that takes
snapshots throughout an adversarial, partitioned schedule found — and the fix
closed — a real post-compaction edge case before this tag shipped.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events (logical ticks, inbound messages,
client proposals, snapshots) and it returns actions (send these messages, apply
this command, take or restore a snapshot). Time, networking, and storage are
injected through trait seams, which is what makes the core provable and
reproducible from a seed. It is the consensus layer above wal-db and the
coordination substrate for Hive DB clustering.
What's new in 0.5.0
Snapshots and log compaction
The log now has a compaction boundary. After a snapshot through index I, entries
up to I are dropped and (I, term_at(I)) becomes the log's new base —
term_at(I) still answers (so consistency checks at the boundary work) while
reads below it return None. The RaftLog trait gains snapshot,
apply_snapshot, and snapshot_index; both MemoryLog and WalLog implement
them, and WalLog persists a snapshot record and physically compacts the WAL file.
A snapshot-policy hint
Snapshotting is driven by the node but performed by the application — the state
machine is yours, so only you can serialize it. Set a threshold:
use raft_io::RaftConfig;
// Ask for a snapshot once 1024 entries pile up beyond the last one.
let cfg = RaftConfig::new(1, [2, 3]).with_snapshot_threshold(1024);
# assert_eq!(cfg.snapshot_threshold(), 1024);When the applied log grows past the threshold, the node emits
Action::Snapshot { index, term }. The application serializes its state through
index and hands it back as Event::Snapshot { index, data }, and the node
compacts. 0 (the default) disables the hint, so snapshots are strictly opt-in.
InstallSnapshot — catching up a far-behind follower
When a follower needs an entry the leader has already compacted away, the leader
sends an InstallSnapshot (new Message variant) instead of an AppendEntries.
The follower installs it — the node emits Action::RestoreSnapshot { index, term, data } for the application to reset its state machine — and then resumes normal
tail replication. The whole flow, end to end:
use raft_io::{Action, Event, RaftConfig, RaftNode};
let mut node = RaftNode::new(RaftConfig::single(1).with_snapshot_threshold(4));
// … drive the node; on Action::Snapshot { index, .. } reply with the state …
# let index = 1;
let _ = node.step(Event::Snapshot { index, data: b"serialized state".to_vec() });framing feature — typed wire encoding
The protocol stays transport-agnostic, but a transport needs some codec. Behind
the framing feature, framing::encode / framing::decode serialize a Message
with pack-io; the message types derive pack_io::Serialize / Deserialize.
# #[cfg(feature = "framing")] {
use raft_io::{framing, Message, RequestVote};
let msg = Message::RequestVote(RequestVote {
term: 4, candidate: 2, last_log_index: 9, last_log_term: 3,
});
let bytes = framing::encode(&msg).unwrap();
assert_eq!(framing::decode(&bytes).unwrap(), msg);
# }A decode failure is Error::Encoding, which a transport should treat like a
dropped message rather than a crash.
Testing found a real bug
tests/snapshot.rs includes a property test that takes snapshots throughout an
adversarial schedule of ticks, proposals, reordered deliveries, and partitions,
asserting no committed entry ever diverges. It surfaced a case where a stale,
reordered AppendEntries whose prev_log_index fell below a follower's
freshly-compacted boundary slipped through the head-of-log shortcut and triggered
a non-contiguous append. The follower's consistency check now accounts for the
snapshot boundary — it acknowledges that it already holds everything through the
boundary and lets the leader resend the tail. This is exactly why the adversarial
suite exists.
Alongside it: a deterministic test that a partitioned follower catches up via
snapshot then tail, a check that compaction never exceeds the applied index, and
framing round-trip tests for every message variant.
New example
snapshot_catchup— a node is isolated while the cluster commits and
compacts a long run, then rejoins and is caught up by a snapshot.
Breaking changes
Pre-1.0, additive in spirit but two surface changes to note:
- The
RaftLogtrait gains three methods.snapshot_indexandsnapshothave
defaults;apply_snapshotdefaults to an error, so a custom backend compiles
but must implement it to support snapshots. MessagegainsInstallSnapshot/InstallSnapshotReplyvariants (it is
#[non_exhaustive], so matches already carry a wildcard arm).
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2
Ubuntu) and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 80 unit + 13 integration / property tests
(safety.rs,replication.rs,recovery.rs,snapshot.rs) + 46 doctests.
Property suites were additionally run at elevated case counts
(PROPTEST_CASES=1500) with no failures. loom is still not exercised: the core
is a single-threaded, owned state machine with no lock-free or shared-state path.
What's next
- v0.6.0 — Membership changes + feature freeze. Single-server add/remove with
joint-consensus-safe sequencing, leadership transfer, and the feature freeze.
Installation
[dependencies]
raft-io = "0.5"
# Optional features:
raft-io = { version = "0.5", features = ["persistence"] } # durable wal-db-backed log
raft-io = { version = "0.5", features = ["framing"] } # pack-io wire framingMSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.4.0...v0.5.0.
Changelog: CHANGELOG.md.
v0.4.0 — Durable Log + Crash Recovery
raft-io v0.4.0 — Durable Log + Crash Recovery
A cluster that survives restarts. v0.4.0 adds durability: under the new
persistence feature, a node backs its log with WalLog, a wal-db-backed
store whose entries and hard state (term, vote) are written before the node acts
on them and recovered on restart. A node killed at any point — mid-election,
mid-replication — comes back, recovers its log from disk, and rejoins without
violating safety. This is verified by a crash-recovery property test that
interleaves node restarts into an adversarial schedule. The in-memory path is
untouched: persistence is purely additive and off by default.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events (logical ticks, inbound messages,
client proposals) and it returns actions (send these messages, apply this
committed command). Time, networking, and storage are injected through trait
seams, which is what makes the core provable and reproducible from a seed. It is
the consensus layer above wal-db and the coordination substrate for Hive DB
clustering.
What's new in 0.4.0
WalLog — a durable, crash-recoverable log
WalLog implements the same RaftLog trait as the in-memory store, so the
protocol core does not change — you just hand the node a durable log instead:
use raft_io::{RaftConfig, RaftNode, WalLog};
let log = WalLog::open("node-1.wal")?;
let mut node = RaftNode::with_log(RaftConfig::single(1), log);
# let _ = &mut node;
# Ok::<(), raft_io::Error>(())It is log-structured. Every mutation — an appended entry, a hard-state
update, a truncation — is encoded as a record and appended to a wal-db
write-ahead log, which frames and checksums each record. An in-memory index
mirrors the current state so reads stay fast. On open, the records are replayed
in order to rebuild that index exactly:
use raft_io::{LogEntry, RaftLog, WalLog};
let mut log = WalLog::open("node-1.wal")?;
log.append(&[LogEntry::new(1, 1, b"set x = 1".to_vec())])?;
log.sync()?; // durable from here
// A restart recovers everything that was written.
let recovered = WalLog::open("node-1.wal")?;
assert_eq!(recovered.last_index(), 1);
# Ok::<(), raft_io::Error>(())raft-io uses wal-db's byte-record API directly and frames its own records, so
it does not pull in wal-db's pack-io feature. Truncated entries remain
physically in the WAL until log compaction (snapshots, v0.5); replay still
reconstructs the correct logical state.
Persist before you respond
Raft's safety rests on current_term, voted_for, and the log being durable
before the node acts on them — a node that forgot it had voted could help elect
two leaders. The node already calls sync on its log at the right moments
(before granting a vote, before acknowledging an append); with WalLog that
sync is a real fsync. A unit test pins the contract down: granting a vote
persists and syncs the vote before the reply is produced, and a no-op vote makes
no durable write.
Crash-recovery testing
tests/recovery.rs (feature-gated) is the centrepiece. Nodes are backed by
WalLog files in a temp directory and can be "crashed" — dropped and rebuilt
from the same file, exactly as a process restart would. A proptest interleaves
crashes into a schedule of ticks, proposals, and reordered deliveries across a
three-node cluster, asserting after every step that no two nodes (or
incarnations) ever apply a different command at the same index. Deterministic
tests confirm a fully replicated log survives a restart of every node — recovered
byte-for-byte — that the cluster then re-elects and keeps committing, and that a
recovered node's term never regresses.
New example
persistent_node(run with--features persistence) — a node elects
itself, commits proposals, is dropped (closing its WAL), then reopens the same
file, recovers its log and term, and carries on.
Breaking changes
None. persistence is a new, off-by-default feature; the existing in-memory
API is unchanged.
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2
Ubuntu) and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo test
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
--all-features: 65 unit + 10 integration / property tests
(safety.rs,replication.rs,recovery.rs) + 38 doctests.- default features: 56 unit + 7 integration / property tests + 37 doctests.
Property suites were additionally run at elevated case counts
(PROPTEST_CASES=600+) with no failures. loom is still not exercised: the core
is a single-threaded, owned state machine with no lock-free or shared-state path.
What's next
- v0.5.0 — Snapshots + log compaction. An
InstallSnapshotRPC, a snapshot
policy hook, log truncation after a snapshot, follower catch-up from snapshot
plus tail, andpack-ioframing for snapshot/RPC payloads under theframing
feature.
Installation
[dependencies]
raft-io = "0.4"
# For the durable, crash-recoverable log:
raft-io = { version = "0.4", features = ["persistence"] }MSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.3.0...v0.4.0.
Changelog: CHANGELOG.md.
v0.3.0 — Log Replication
raft-io v0.3.0 — Log Replication
A correct multi-node cluster. v0.3.0 builds the heart of Raft on top of v0.2's
election layer: a leader replicates its log to followers, tracks each follower's
progress, backtracks fast when a log diverges, and advances the commit index once
a quorum has an entry. The whole thing is held to account by an adversarial
property-test suite that reorders, drops, duplicates, and partitions messages and
asserts that committed entries never diverge. Durable persistence is v0.4.
What is raft-io?
A from-scratch implementation of the Raft consensus algorithm, built as a clean,
embeddable library rather than a framework. The protocol core is a deterministic,
sans-I/O state machine: you feed a node events (logical ticks, inbound messages,
client proposals) and it returns actions (send these messages, apply this
committed command). Time, networking, and storage are injected through trait
seams, which is what makes the core provable and reproducible from a seed. It is
the consensus layer above wal-db and the coordination substrate for Hive DB
clustering.
What's new in 0.3.0
AppendEntries replication with per-follower progress
A leader now carries log entries to followers, not just heartbeats. For each
follower it keeps a next_index / match_index pair and a state:
- Probe — when a follower joins or its log diverges, the leader sends
conservatively and waits for each reply, finding the match point. - Replicate — once the match point is known, the leader streams entries and
advancesnext_indexoptimistically, pipelining batches without waiting for
each acknowledgement. This is the steady-state throughput path.
Batches are bounded by RaftConfig::with_max_batch (default 64) so a far-behind
follower is caught up in steady chunks rather than one unbounded message.
use raft_io::RaftConfig;
// Larger batches for a high-throughput, low-RTT deployment.
let cfg = RaftConfig::new(1, [2, 3]).with_max_batch(256);
# assert_eq!(cfg.max_batch(), 256);Fast conflict-hint backtracking
When a follower rejects an append because its log does not match, it no longer
forces the leader to step back one entry at a time. The reply now carries a hint:
use raft_io::AppendEntriesReply;
let rejection = AppendEntriesReply {
term: 5,
success: false,
from: 2,
match_index: 0,
conflict_index: 3, // probe here next…
conflict_term: 2, // …or just past the leader's last entry of this term
};
# assert!(!rejection.success);The leader uses the hint to skip its next_index back by a whole term in a
single round trip — the fast-backtracking optimisation from the Raft thesis
(§5.3). Divergent logs reconcile in O(terms), not O(entries).
Commit on a quorum, with the current-term safety rule
The commit index advances to the highest entry a majority of nodes has stored —
but only counting an entry as committable if it was created in the current
term (Raft §5.4.2). Older-term entries ride along once a current-term entry
above them commits. This is the rule that prevents a subtle class of data loss
across leader changes, and it is enforced exactly.
use raft_io::{Action, Event, RaftConfig, RaftNode};
// A single-node cluster is a quorum of one, so it commits at once.
let mut node = RaftNode::new(RaftConfig::single(1));
while !node.is_leader() {
let _ = node.step(Event::Tick).unwrap();
}
let actions = node.step(Event::Propose(b"x".to_vec())).unwrap();
assert!(actions.iter().any(|a| matches!(a, Action::Apply { .. })));
assert_eq!(node.commit_index(), 1);Followers reconcile divergent tails
A follower that receives entries conflicting with its own log truncates the
divergent tail and appends the leader's version, then advances its commit index
from the leader and applies committed entries in order. The protocol guarantees
a leader never sends entries that conflict below the commit index, so committed
state is never discarded.
RaftLog::entries(from, to) — bulk range reads
The trait gains a range read so a backend can hand the leader a replication batch
in one call. It has a default implementation over entry, and MemoryLog
overrides it with a slice copy.
Adversarial property testing
tests/replication.rs is the centrepiece. It models a network that can reorder,
drop, duplicate, and partition messages, and proptest generates schedules of
ticks, proposals, and deliveries across 3- and 5-node clusters. After every step
it asserts:
- State Machine Safety / Log Matching — if any two nodes apply an entry at
the same index, it is the same command. No committed entry is ever contradicted,
across leader changes and partitions. - Apply ordering — each node applies entries in strict index order, no gaps.
Alongside the property tests are deterministic checks that a healthy cluster
commits on every node, that a minority partition cannot commit while the majority
keeps progressing, and that a healed partition reconciles. The suite has been run
at 3,000 cases per property with no violation.
New examples
replicated_log— a 3-node cluster proposes a series of commands and
prints each node's applied log to show they agree, entry for entry.partition_recovery— a 5-node cluster is split; the majority keeps
committing, the minority stalls, and healing brings every node back into sync.
Breaking changes
AppendEntriesReply gains two fields, conflict_index and conflict_term,
which carry the backtracking hint. Code that constructs the reply directly must
set them (use 0 for both on a success). This is a pre-1.0 wire-shape change;
the message protocol freezes at v0.7.
Verification
Run on Windows x86_64, Rust stable; the same commands pass on Linux (WSL2
Ubuntu) and via the CI matrix:
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit
cargo build --examples --all-features
cargo benchAll green. Test counts at this tag:
- 54 unit tests
- 7 integration / property tests (
tests/safety.rs,tests/replication.rs) - 37 doctests
Property suites were additionally run at PROPTEST_CASES=3000 with no failures.
loom is still not exercised: the core is a single-threaded, owned state machine
with no lock-free or shared-state path.
What's next
- v0.4.0 — Durable log + crash recovery. A
wal-db-backedRaftLogunder
thepersistencefeature, the term/vote/log durability contract on the RPC
path, and crash-recovery tests that kill a node mid-replication and verify it
rejoins without violating safety.
Installation
[dependencies]
raft-io = "0.3"MSRV: Rust 1.85 (edition 2024).
Documentation
Full diff: v0.2.0...v0.3.0.
Changelog: CHANGELOG.md.