Skip to content

CRAFT Design

Jump to bottom
Brian Szmyd edited this page Jul 2, 2026 · 21 revisions

CRAFT: Client Assisted RAFT

Scope. This page is the HomeBlocks (server) side of CRAFT. The remote client/initiator side is a separate epic; it is summarized here only at its API boundary. Audience: engineers picking up the CRAFT server work who don't yet have the homeblocks internals in their head.

Contents

What CRAFT is

CRAFT (Client Assisted RAFT) is the replication protocol for HomeBlocks block volumes. Its one defining move is to split the data path from the consensus path:

  • Data path (hot, leaderless). The single client that owns a volume assigns each write a monotonic LSN and broadcasts it directly to all replicas in parallel. A write is durable once a quorum acks the journal append. The RAFT leader plays no role here.
  • Consensus path (cold, RAFT). RAFT is used for only four things: leader election, the single-writer session (InternalLogin / InternalLogout → term + client_token + lease), recovery watermarks (SyncRSCommitLSN), and membership changes. Write data never enters the RAFT log, only LSN watermarks and session tokens do.

A second structural choice runs through everything: durability, readability, and index apply are decoupled. Appended (quorum journaled the data, durable) is what makes a write readable: an eligible replica serves appended entries directly, from the LBA index if applied, or from the journal-tail overlay if not. Committed (applied to the LBA index) happens strictly in dLSN order at the contiguous commit frontier, so every replica walks the same apply sequence and reaches the same stable state with no per-entry version checks. The client drives the frontier lazily, via commit, keep_alive, or piggybacked on the next write; it advances eagerly and independently per replica, never as a synchronous quorum step.

Glossary

Sequence numbers & watermarks

Term Meaning
LSN Log Sequence Number.
dLSN Data LSN: the per-partition number the client assigns to each write (its "position"). Dense (contiguous) within a partition. This is the only LSN CRAFT itself uses.
rLSN RAFT log index. Distinct from dLSN (data is not in the RAFT log).
last_append_lsn Highest LSN a replica has appended (present in its journal).
commit_lsn (≡ Synced) The contiguous committed prefix on a replica (Empty slots skipped): everything ≤ it is present and applied. The fill watermark.
rs_commit_lsn The replica-set commit LSN chosen at login = max(quorum.last_append).
L (login dLSN) The dLSN login returns (= rs_commit_lsn). Boundary at/below (≤) which the client cannot name a write's LBA (prior-session data).
H (readLSN / horizon) Per-read horizon: the read reflects writes ≤ H. Normally the client's contiguous quorum-acked prefix; precisely, a read may carry any H provided no unresolved slot ≤ H overlaps its range (see the failed-write note in Resync).
startLSN The joining member's promotion watermark and its backlog/tail boundary. Initialized to the replica-set dLSN at the moment of the member change; advanced to the pinned CP's commit_lsn by each snapshot (re-)pin (recorded in snapshot metadata). Promotion requires the member's commit_lsn ≥ the current startLSN.

Slot / write states (a slot is one dLSN on one replica)

State Meaning
Appended A write present on a quorum (durable), not yet applied to the index. Readable on an eligible replica via the journal-tail overlay, but only up to a read's horizon H. (A replica may physically hold a sub-quorum write above H that it received but that has not reached quorum: held, never served, until H passes it or it is resolved away.)
Committed Applied to the LBA index, strictly in dLSN order at the contiguous commit frontier. An apply/reclaim state, not a readability gate.
Missing A write known to exist but not held here; must be fetched from a peer (or declared Empty).
Empty A slot proven never quorum-durable (a quorum of nodes positively known to lack it). Declared only by the leader; the verdict rides SyncRSCommitLSN (empty_slots[]) or the login. A permanent no-op hole that commit skips. A non-responding minority may still hold data there; the reconciliation rule (see Resync) discards it.

Consensus & session

Term Meaning
RAFT / Raft Consensus protocol (leader election + replicated log). CRAFT uses it only on the cold path.
Paxos The older consensus family Raft descends from.
term The CRAFT session term, stored by InternalLogin (distinct from RAFT's internal election term). Bumped on each login, including the forced re-login after a membership change, and carried on every IO to fence stale clients.
client_token Identifies the single writer; set by InternalLogin.
quorum A majority of the replica set (2 of 3).
f Number of tolerated failures (f = 1 for a 3-node set).
RS / Replica Set The nodes holding one partition's copies (typically 3).
partition The independent replica-set CRAFT replicates. A volume comprises one or more partitions; a given LBA maps to exactly one partition, so overlapping writes always land in the same partition (which is why the per-partition dLSN is a sufficient merge key).
volume Comprises one or more partitions. CRAFT operates per-partition (dLSN).

RPCs, RAFT entries & rules

Term Meaning
SyncRSCommitLSN RAFT entry that advances the replica-set commit watermark; the primary resync trigger. Carries the client_token (session verification) and the leader's empty_slots[] verdicts; the leader resolves every slot ≤ the watermark before proposing.
InternalLogin RAFT entry that sets client_token + term (single-writer enforcement).
InternalLogout RAFT entry recording logout / lease expiry: marks "no active client" and authorizes the leader stand-in.
lease Locally-evaluated client liveness (the watchdog); acted on only once InternalLogout commits.
watchdog Per-replica client-liveness timer, reset by writes / keep_alive; expiry triggers GetRSCommitLSN + SyncRSCommitLSN, and past the lease, the logout path.
GetRSCommitLSN Non-RAFT peer query of a replica's {commit_lsn, last_append_lsn}. Carries the is_login quiesce flag. Code: get_rs_commit_lsn / get_lsns.
keep_alive Client RPC: commit + watchdog reset. Its response carries {commit_lsn, last_append_lsn}, feeding the client's Missing map and the reconfig promotion gate; its request carries all_committed_lsn so members can reclaim journal opportunistically.
FetchData Non-RAFT peer pull of missing journal slots (the resync data path).
ETERM / ENOTLEADER Rejection codes: stale session term on IO; login sent to a follower.
overlay read A read served directly from an appended-but-not-yet-applied journal entry (the journal-tail overlay). No index write happens on the read path. (Formerly CommitAndRead, before apply became strictly in-order.)
journal-tail overlay Per-replica in-memory map (LBA → highest-dLSN unapplied entry) over (commit_lsn, last_append]; rebuilt from the journal on restart. What makes appended entries readable before apply. Reads consult it only up to the horizon H; entries above H (the not-yet-quorum tail) are held but never served.
all_committed_lsn The set-wide floor: min over voting members' reported commit_lsn. Computed by the client from keep_alive/commit responses (or by the stand-in leader from polls) and piggybacked back out so members reclaim journal opportunistically.
merge key The convergence rule: highest dLSN per LBA wins, on every replica.

Components & systems

Term Meaning
HomeBlocks The block-volume store (the server side this page covers).
HomeStore The underlying storage engine (journal / index / blkdata / RAFT / CP).
homeobject A blob store on HomeStore; the template CRAFT follows for snapshot + replace_member.
CraftReplDev New per-partition CRAFT class (parallel to HomeStore's repl_dev); a volume owns one per partition.
CraftConnector Client-facing RPC frontend that translates to CraftReplDev calls.
CraftRaftListener repl_dev_listener handling the CRAFT RAFT entries + snapshot / replace_member callbacks.
CraftJournalBackend Mockable journal abstraction wrapping the HomeStore logstore.
RaftReplDev / ReplDisk HomeStore's replication device classes.
replace_member / learner / flip_learner_flag HomeStore membership-change primitives CRAFT leans on.
ublk / HomeBlkDisk The in-process test adapter used as the development scaffold.
PacificA Primary-backup replication framework (consensus only for config); a CRAFT touchstone.
CORFU Shared-log storage architecture (clients write directly to storage units); a CRAFT touchstone.

Storage & durability

Term Meaning
LBA Logical Block Address.
FUA Force Unit Access: every write is durable on completion (write-through; no writeback cache, no FLUSH).
CP HomeStore Checkpoint: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary (a consistency point).
DSN Data Sequence Number: a HomeStore marker carried in snapshot-resync metadata.
logstore HomeStore's write-through journal (backs CraftJournalBackend).

Background: Raft, Paxos, PacificA, CORFU, and CRAFT

CRAFT borrows one idea each from a few well-known systems. If these names are unfamiliar this is the quick map; if they're familiar, it pins down exactly which idea CRAFT takes from each.

Raft & Paxos: consensus protocols. Both solve agreement among distributed nodes despite failures: how a set of replicas commit to the same ordered log of operations. Paxos is the older, foundational family; Raft was designed to be easier to understand and implement while giving the same replicated-log guarantees (an elected leader replicates an ordered log to followers). They answer "how do replicas agree?"

PacificA: primary-backup replication for storage. A framework for keeping replicas of a log-based storage shard consistent. A primary orders the writes and forwards them to backups; correctness across failures is delegated to a separate, consensus-backed configuration manager that owns membership. The key idea: keep consensus off the per-write path and pay for it only on rare reconfiguration. It answers "how do I replicate a storage shard with a primary and backups?"

CORFU: a shared-log architecture. Turns a cluster of flash devices into one global append-only log shared by many clients. A sequencer hands out log positions; clients then write directly to the storage units at those positions (no primary in the data path); a separate consensus-backed layer handles reconfiguration and hole-filling. It answers "how do I expose one global durable log across many devices, with clients writing directly?"

How they layer. Raft/Paxos are general-purpose "get a consistent log." PacificA is a specific primary-backup replication scheme with consensus only for config. CORFU is a shared-log abstraction others build on, where clients write straight to storage.

System The question it answers
Paxos / Raft How do replicas agree on an ordered log?
PacificA How do I replicate a storage shard with a primary + backups, keeping consensus off the write path?
CORFU How do I expose one global durable log across many devices, with clients writing directly?
CRAFT How do I replicate a single-attach block volume when the client itself can order the writes?

How CRAFT fits. CRAFT takes one idea from each and specializes it for single-attach block volumes:

  • From PacificA: keep consensus off the write path; use a separate consensus layer only for configuration-class decisions. CRAFT's consensus layer is Raft, and it touches only the cold path: leader election, InternalLogin (who is the single writer + term), SyncRSCommitLSN (recovery watermark), and membership. Raft here plays exactly PacificA's "configuration manager" role. The twist: PacificA still has a primary node that orders and forwards writes; CRAFT has no data-path primary at all.
  • From CORFU: the writer assigns positions (LSNs) and writes directly to every storage replica, bypassing any leader/primary on the hot path. The difference: CORFU runs a separate sequencer serving many concurrent clients over a shared log; CRAFT relies on block volumes being single-attach, so the one client per volume is its own sequencer: no sequencer service and no per-write coordination.

In one line: CRAFT = CORFU-style direct-to-replica writes by a client that is its own sequencer, with Raft as a PacificA-style configuration manager that never touches the data path. That combination is only safe because a block volume has a single writer (see The contract CRAFT owes).

PacificA to CORFU to CRAFT: the same design elements

The lineage in one picture: consensus (blue) stays off the data path in all three, the data path (orange) gets more direct, and the sequencer (yellow) moves from an external service in CORFU into the single client itself in CRAFT. The bottom band places CRAFT among the block replicators: some carry the write through a leader or ordered chain (Ceph, EBS, arrays); DRBD, like CRAFT, is already leaderless single-writer broadcast, and CRAFT's delta over it is quorum-ack + staleness-routed reads + server-side resync, not leaderlessness itself.

Adjacent systems: where CRAFT sits. The four above are conceptual ancestors (where CRAFT's ideas come from). The systems CRAFT sits among are block-volume replicators. They split into two camps by data-path topology, and CRAFT is not the first into the second camp, which is the honest framing:

A leader or ordered chain carries the write:

  • Ceph RBD stripes an image into RADOS objects; CRUSH maps each to a placement group whose first OSD is the primary. Writes go client to primary to secondaries, then the primary acks; reads default to the primary. Consensus (the Paxos monitors) agrees the cluster map off the data path, which CRAFT shares, but the data path itself runs through a per-object primary. Ceph is PacificA-shaped, not CORFU-shaped.
  • Amazon EBS is the closest architectural sibling on the config axis: single-attach block with the same consensus-off-the-data-path split (its Physalia Paxos store handles reconfiguration), but the data path is chain replication (a write travels head to tail down the replica chain), not a broadcast. See "Why broadcast-quorum, not chain replication".
  • Array / controller-based replication (SCSI arrays) keeps the dirty bitmap opaque inside the controller, the property CRAFT inverts.

The single writer broadcasts to all replicas (leaderless), CRAFT's own camp:

  • DRBD / RAID1 is the closest existing system, and it is already leaderless: a single-writer primary (a diskless one, in DRBD9, is exactly CRAFT's remote client) ships every write to all storage nodes with no forwarding or election among them, and DRBD9 even adds majority quorum and distributes reads to one node. Its dirty bitmap is the one-dimensional ancestor of CRAFT's per-replica Missing map. So CRAFT's contribution is not "leaderless block replication" (DRBD ships it); it is three specific refinements on that base:
    1. Per-write quorum-ack, not sync-to-all: a write commits at a majority, so it never waits for the slowest up replica (DRBD protocol C waits for every connected peer); the laggard becomes Missing and resyncs.
    2. Staleness-routed reads from partially-synced replicas: the client's horizon + per-(replica, LBA, dLSN) Missing map let a read use a replica for the ranges it is current on, where DRBD reads only from a fully-UpToDate node. The whole horizon/overlay/eligibility apparatus is the price of that capability.
    3. Server-visible, consensus-serialized slot state: Missing/Empty/commit_lsn are first-class and resync is peer-to-peer from that state (DRBD's bitmap is writer-side and writer-driven), with membership/fencing/watermarks carried by an embedded RAFT log rather than an external cluster manager (Pacemaker/STONITH).

The honest one-liner: leaderless single-writer block replication is not new (DRBD does it, CORFU does the direct-to-storage write for a shared log). CRAFT's bet is the combination specialized for single-attach block on a shared fabric: quorum-ack for latency, staleness-routed reads to use partially-synced replicas, and server-side resync from consensus-tracked slot state, all on a log-structured homestore substrate.

Why broadcast-quorum, not chain replication. Chain replication (EBS) sends a write head to tail down the replica chain, one hop each, so the client uplink carries the write only once and replica-to-replica traffic stays in the storage tier. Its whole advantage is bandwidth locality, and it pays off only when that replica-to-replica traffic rides a separate or cheaper storage fabric (FC SAN, RDMA backend, a dedicated storage network). HomeBlocks has none: one generic Ethernet, shared top-of-rack switches for client-to-storage and storage-to-storage alike, symmetric PHYs (25 to 100 Gbps). On a shared fabric the aggregate is topology-invariant: a write of size W at RF=3 injects 3W into the same switches whether the client broadcasts it or the chain forwards it, so chain's locality advantage evaporates.

What remains is where the egress originates, and on a shared fabric that favors broadcast. Chain shifts 2W of egress onto storage NICs (the head and middle nodes each receive and forward, 2x); broadcast keeps storage NICs at 1x (receive only) and concentrates 3x on the client. For single-attach that is the better allocation: client hosts are sparse (few volumes each), storage nodes are dense (many volumes each), so sparing the storage NICs from forwarding is the right trade. CRAFT's steady-state storage-to-storage traffic is near zero (that path is resync-only); chain is storage-to-storage on every write.

With bandwidth neutralized, latency decides, and broadcast wins: one round-trip to quorum versus up to N serial hops to the chain tail. The residual cost is the client host's NIC egress (3x the aggregate write bandwidth under broadcast), a per-client provisioning ceiling of about NIC/3 (roughly 1 GB/s per client at 25 Gbps, 4 GB/s at 100 Gbps), not a fabric bottleneck; chain would only relocate those bytes onto the same shared storage NICs while adding hop latency. So CRAFT takes the latency win and accepts client-side egress concentration, which single-attach makes cheap. On a dedicated storage fabric this calculus flips and chain is the better choice: that is the deployment assumption to revisit if it ever changes.

The contract CRAFT owes (block-device semantics)

CRAFT is a single-writer block device, not a transactional or linearizable store. Holding it to the right (weaker) contract is what makes the design tractable. It owes exactly:

  • Durability: a write that was acked survives crash, up to f failures. All homeblocks I/O is FUA (see Durability model): there is no FLUSH op and no writeback cache, so "acked" already means "durable." Un-acked (in-flight) writes have undefined outcome on crash; the filesystem/application owns crash consistency for them. We do not treat in-flight writes as transactions.
  • Read-after-write: only for causally ordered ops (the app waited for the write's ack before issuing the read). Concurrent read+write to the same LBA is a race with an undefined result, by the block contract.
  • Read stability: once a newer value of an LBA has been observed, a later read does not regress to an older one.
  • Replica convergence: all replicas eventually agree on the same value per LBA.

Not owed: cross-LBA ordering, atomicity of concurrent overlapping writes, transactions.

Consequence: there is no order among in-flight I/O. All submitted I/O races naturally; the application serializes (waits for ack) when it needs ordering. So the dLSN is not preserving application order (there is none for concurrent I/O). Its job is to be a deterministic merge key: highest dLSN to an LBA wins, on every replica, so replicas converge to the same value even when two writes race. The client assigns dLSNs in submission order; "highest-LSN-per-LBA wins everywhere" is the whole rule. No overlapping-write serialization logic is needed.

Why (motivation)

  1. Kill the leader bottleneck. Classic RAFT funnels every byte through the leader (client → leader → log → fan-out → apply). The leader's egress is N× the write bandwidth and it double-writes (log, then state machine). CRAFT pushes the fan-out onto the client and writes data to the journal once, a throughput and write-amplification win.
  2. Lower write latency. Client → replica broadcast is one hop to durability (quorum append) vs. two through a leader.
  3. The single-writer constraint is essentially free here. Block volumes are single-attach (EBS-like), so there is naturally one writer per volume. Given one writer, that writer is the total-order authority. You don't need consensus to order writes. That is the load-bearing insight that makes a leaderless, consensus-free data path safe. RAFT is then only needed for the genuinely ambiguous things: who is the one writer (split-brain) and what is durable after a crash (recovery watermark).
  4. Keep RAFT small and off the hot path. Login and resync are low-frequency, so the RAFT log stays tiny and never carries data.
  5. No homestore changes. Built entirely on existing homestore journal/index/ blkdata primitives, so the blast radius stays in the HomeBlocks layer.

Where CRAFT plugs into HomeBlocks

Today (solo, data-through-log):

home_blocks API (create_volume / async_read / async_write)
  -> HomeBlocksImpl / volume_mgr
  -> Volume   (owns a homestore::repl_dev created SOLO, empty member list)
              IO via rd()->async_write / async_read / async_write_journal
  -> HBListener (repl_dev_listener) handles RAFT callbacks
  -> homestore (RaftReplDev data-through-log, journal, index, blkdata)

With CRAFT (leaderless data path):

Client (initiator) --broadcast writes--> every replica's CraftConnector
  CraftConnector            (client-facing frontend; transport-agnostic)
    -> CraftReplDev         (one per partition; multi-member RAFT group)
       - write / read / commit / keep_alive / login / ...  (CRAFT API)
       - CraftJournalBackend  -> homestore logstore journal (write-through / FUA)
       - LBA index + blkdata for reads (journal-tail overlay serves unapplied entries)
       - CraftRaftListener: RAFT only for SyncRSCommitLSN + InternalLogin/Logout
         (+ snapshot / replace_member callbacks for reconfiguration)
    -> homestore (RaftReplDev for consensus only; journal / index / blkdata / CP / snapshot)

Component map (homeblocks-concrete):

Component New/changed Location Responsibility
CraftReplDev new src/lib/craft/craft_repl_dev.* The CRAFT API + per-partition state {commit_lsn, last_append_lsn, client_token, term}
CraftJournalBackend new (mockable) src/lib/craft/craft_repl_dev.* Journal slot write/read/truncate; production impl wraps the homestore logstore
CraftRaftListener new src/lib/craft/craft_repl_dev.* on_commit dispatch for the CRAFT RAFT entry types (SyncRSCommitLSN, InternalLogin, InternalLogout); snapshot + replace_member callbacks
CraftConnector new src/lib/craft/ (TBD) Translate client RPCs ↔ CraftReplDev calls; leader redirect; term checks; transport-agnostic
Volume wiring changed src/lib/volume/volume.cpp Create a multi-member CraftReplDev for CRAFT-mode volumes; route IO
HomeBlocksImpl wiring changed src/lib/homeblks_impl.cpp create_repl_dev_listener returns a CraftRaftListener; volume lifecycle/recovery
volume_info.replication_mode exists src/include/homeblks/home_blocks.hpp DISABLED (today's solo path) vs CRAFT

Note. There is no ScstConnector in the current tree. The de-facto frontend today is the public async_read/async_write API plus the ublk adapter (src/test/homeblk_disk). "Add CraftConnector" = add the client→CraftReplDev translation layer.

Journal backing. CraftJournalBackend opens the logstore in out-of-order mode (not the append-only mode RAFT uses): dLSN maps 1:1 to logstore seq_num via write_async(seq_num) (out-of-order, no gap buffer), read_sync(seq_num) for random slot reads, fill_gap(seq_num) to mark an Empty slot (so the contiguous watermark advances past it), and truncate(seq_num) at login.

Write-once, by reference (verified against homestore). A CRAFT write does not put the payload in the journal: blocks are allocated up front, the payload is written once via the data service, and the journal slot records only the reference, {term, lsn, lba, len, blkid}. This is homestore's own replication pattern (HS_DATA_LINKED: solo_repl_dev.cpp allocates in rreq->init, writes via data_service().async_write, then journals a repl_journal_entry followed by the multi_blk_id; create_journal_entry's size formula is header + key + serialized blkids, with no payload term, so the payload never enters the journal buffer). HS_DATA_INLINED exists only for header-only entries, not small-payload inlining. The ack fires on journal-append completion, which starts only after the data write completes, so an acked write has both payload and reference on media; a crash between the two leaves only uncommitted blks, which homestore recovery reclaims. Apply is a metadata-only index insert (plus reclaim of superseded blocks); FetchData serves by reading the referenced blocks; truncation and Empty discard free them. The payload is written exactly once: the log-then-state-machine double write never happens. The journal is bounded: the logstore reclaims below a checkpointed point, with CRAFT's floor = min(checkpointed commit_lsn, all_committed_lsn). The first bound is local crash-safety (the journal backs the overlay and index replay, so never reclaim above the checkpointed apply frontier). The second is the peers' FetchData needs, advanced two ways: exactly at a login (Phase 2 converges the whole set to rs_commit_lsn), and opportunistically between logins via piggyback: the client sees every member's commit_lsn in each keep_alive/commit response, computes the set-wide min (all_committed_lsn, voting members only), and carries it on its next keep_alive/write; each member may then truncate below it (a member never re-fetches below its own reported commit_lsn, which is monotonic, so a stale min is merely conservative). A silent member freezes the min until the retention window relegates it to the snapshot path (baseline resync), which is what lets the journal stay bounded without keeping entries or membership static forever. Learners are excluded from the min (the snapshot path covers them); when detached, the stand-in leader computes and distributes the same value via its polls.

The snapshot trigger for a returning voter is the RAFT log itself. CRAFT compacts its (tiny) RAFT log below the SyncRSCommitLSN entry at the reclaim floor, so a member that missed more than the retention window is necessarily behind the compacted log, and nuraft's standard too-far-behind snapshot machinery fires with the same four listener callbacks reconfiguration uses. Data-behind alone would never trip nuraft (data is not in the log); tying compaction to the floor is what makes the existing machinery see it. After the snapshot, the tail fills via the normal FetchData loop; no promotion gate applies (the member is already a voter).

Durability model: FUA write-through

All homeblocks I/O is FUA (write-through): every write is durable the moment it is acked. A guest FLUSH is accepted and acked immediately as a satisfied no-op: under FUA-always nothing is ever cache-dirty, so there is nothing to flush (a filesystem journal commit that issues PREFLUSH still gets success, correctly, because everything it would flush is already durable). FLUSH is also not an ordering barrier, and never was: since Linux removed hardware I/O barriers (2.6.37, 2010) the block layer provides no reorder fence at all. Ordering is the guest's, enforced by waiting for a write's completion before issuing the next, and CRAFT honors exactly that, because a completion (ack) means quorum-durable: the earlier write is durable before the next is even submitted. So no ordering among in-flight I/O, acked = durable is not a weakening, it is the literal block contract. (The only media-level guarantee beyond durability is single-write atomicity via RWF_ATOMIC / atomic write units, which is nascent and which CRAFT does not yet expose.) Durability must be write-through; otherwise a simultaneous restart of all replicas could lose acknowledged writes.

This maps directly onto homestore's logstore, which is already write-through (verified against homestore dev/v8.x):

  • append_async only buffers the record (log_dev.cpp:288, flush_if_necessary at :297). Flushing is group-commit (size threshold or timer).
  • LogDev::flush() issues a synchronous device write (m_vdev_jd->sync_pwritev(...), log_dev.cpp:532) and calls on_flush_completion() only after it returns (:539).
  • on_flush_completion (log_dev.cpp:545) raises each append's completion callback (on_write_completion at :583).

So the awaited append completion fires only after the bytes are synchronously written to the journal device, never on buffering. Group-commit just batches concurrent appends into one sync write and acks them together (ideal for CRAFT's broadcast-and-await-quorum pattern).

Requirement on CRAFT: CraftReplDev::write must await the logstore append completion before acking the client. Then quorum-ack ⇒ quorum-durable ⇒ an all-replicas restart cannot lose an acked write.

Device-level power-safety: the journal device is opened O_DIRECT (device_manager.cpp:47) and homestore skips fsync in direct-IO mode. O_DIRECT

  • sync write lands data on the controller; power-safety relies on the drive being FUA / write-cache-off / power-loss-protected. This is settled operationally: homeblocks already runs the journal this way in production.

Login

Login establishes a new term, converges replica state to a starting dLSN, and enforces single-writer exclusivity, all before any IO is accepted. It is leader-only; a follower replies ENOTLEADER plus the leader endpoint.

sequenceDiagram
    autonumber
    participant C as Client (initiator)
    participant L as Leader (S1)
    participant S2 as Replica S2
    participant S3 as Replica S3

    C->>L: login(client_token, vol_id)
    Note over L: Leader-only. Follower replies<br/>ENOTLEADER + leader endpoint.

    Note over L,S3: Phase 1: collect replica LSN state (non-RAFT broadcast)
    L->>S2: GetRSCommitLSN(is_login=true, term, my_commit, my_append)
    L->>S3: GetRSCommitLSN(is_login=true, term, my_commit, my_append)
    S2-->>L: {commit=5, last_append=7}
    S3-->>L: {commit=10, last_append=11}
    Note over L: rs_commit_lsn = max(quorum.last_append) = 11

    opt Leader behind (self.last_append < 11)
    L->>S3: FetchData(missing slots up to 11)
    S3-->>L: JournalSlot[] (lba,len,data)
    Note over L: append to own journal
    end

    Note over L,S3: Phase 2: SyncRSCommitLSN(11) via RAFT (data NOT in log)
    L->>S2: RAFT replicate SyncRSCommitLSN(11, token)
    L->>S3: RAFT replicate SyncRSCommitLSN(11, token)
    Note over S2,S3: on apply, if behind FetchData up to 11.<br/>commit_lsn = 11. (no truncation here, login-only step below)

    Note over L,S3: Phase 3: InternalLogin(token, term+1) via RAFT
    L->>S2: RAFT replicate InternalLogin(token, T+1)
    L->>S3: RAFT replicate InternalLogin(token, T+1)
    Note over S2,S3: store client_token, term=T+1.<br/>reject IO with any other term

    Note over L,S3: Phase 4: truncate stale tail (login-only, after InternalLogin commits)
    L->>S2: truncate(11) if last_append > 11
    L->>S3: truncate(11) if last_append > 11
    Note over L: leader truncates its own tail too<br/>if last_append > 11
    Note over S2,S3: drop appended-but-not-quorum entries above rs_commit_lsn

    L-->>C: LoginResult{members, dLSN=11, term=T+1}
    Note over C: open a queue per member, begin IO at dLSN+1
Loading

The watermark rs_commit_lsn = max(quorum.last_append) is forced, not a tuning choice; see the recovery-watermark deep-dive. The poll set includes the leader itself (its {my_commit, my_append} ride the request), so the max is taken over a quorum of the full member set. Only last_append feeds the watermark; the polled commit_lsn serves the other jobs of this RPC: it is a contiguity certificate (everything ≤ it is applied on that member, no holes), so it bounds the leader's fetch-or-verdict work to (min commit, rs_commit], seeds the journal-reclaim floor (all_committed_lsn, the leader's only source when no client is attached), and carries the reconfig promotion gate on watchdog polls.

Everything on this page is per-partition: one CraftReplDev, one RAFT group, one term, one login, one dLSN space per partition. A client attached to a multi-partition volume runs an independent CRAFT session per partition. (The development scaffold provisions one partition per volume, so vol_id doubles as the partition id in v1 code.)

Fencing the old session (quiesce barrier). A replica quiesces old-session writes the moment it receives the login poll: GetRSCommitLSN carries an is_login flag, and on it a replica stops acking data writes from any prior term, then reports last_append. This makes max(quorum.last_append) capture every write quorum-acked before the barrier (quorum intersection), so login truncation above rs_commit_lsn can never discard an acked write, and a would-be zombie writer is rejected (explicit error) rather than acked-then-silently-truncated. Watchdog/periodic GetRSCommitLSN polls carry is_login=false and never quiesce (they ride the live tail). The quiesce releases on either of: an explicit login-aborted signal the leader sends when it cannot reach quorum, or a replica-side quiesce timeout (a small multiple of the login round-trip) that fires if that signal is lost, so a failed or partitioned login can never wedge the current owner. Assumption: the attach layer (CSI) should also hard-fence the prior client on failover (reservation / network revoke); the quiesce barrier is CRAFT-side defense-in-depth.

Leader behind (Phase 1b) fallback. The leader fetches missing slots ≤ rs_commit_lsn from any peer holding each; a slot ≤ rs_commit_lsn absent from the whole responding quorum was never quorum-durable and is marked Empty. If a quorum cannot be reached, the login fails and the client retries (or RAFT elects a new leader).

Write, Commit, Read, and readLSN protection

Write. The client assigns ++next_lsn, broadcasts write(term, lsn, lba, len, data) to all replicas. Each validates term, appends the data to its journal at slot lsn (zero-copy, may arrive out of order), advances last_append_lsn, and acks, per write, since in-flight I/O is unordered. Durable at quorum-ack (Appended), and readable once Appended on an eligible replica: served from the journal-tail overlay until it is applied. A replica may receive a write before it is quorum-acked (the client broadcasts to all) and does journal it, but that sub-quorum entry sits above the horizon H, and a replica serves only versions ≤ H (see Read): it is held, never read, until the write becomes quorum-durable (H passes it) or is resolved away. That is exactly what makes eager broadcast safe: a lone replica holding a not-yet-quorum write cannot leak it. No LBA-index update happens on the write itself.

Commit (apply): strictly in dLSN order. In-flight I/O has no block-level order, but replicas must converge, so they need an agreed order, and dLSN is that order (client-assigned, total). CRAFT enforces it by apply order, not by version checks: the index applies entries only at the contiguous commit frontier (commit_lsn), in dLSN order, skipping Empty. Blind overwrite is then correct by construction, every replica reaches the same stable state, the index at any CP is an exact prefix of the write history (clean snapshots), and crash recovery is deterministic (prefix, then in-order replay, then overlay). The rejected alternative, out-of-order apply, would need a dLSN version in every index entry (overwrite-iff-higher) plus partial-overlap splits and stale-block accounting. Superseded blocks are reclaimed at apply time. The client drives the frontier (lazily, via commit / keep_alive / piggyback); readability is per-write and does not wait for it: a higher LSN is readable (overlay) while a lower one is still a hole, with overlaps resolved by the merge key (highest LSN per LBA wins, enforced by the apply order and, above the frontier, by the overlay tracking the highest dLSN per LBA).

The per-partition scalar commit_lsn (≡ Synced) is the contiguous applied prefix, the highest LSN with no unresolved hole below it (Empty slots are skipped). It is the fill watermark used by resync and reconfiguration, and the read gate in the ≤ L region (below); individual writes above it are readable from the overlay, which is why readability is not tied to it.

commit() is best-effort per replica: it advances commit_lsn to just below the first Missing hole (applying present entries in order, skipping Empty) and returns the achieved watermark; a stall at a hole is not an error (resync fills it), and it pauses only apply + block reclaim above the hole, never reads. Recovery: the journal is FUA-durable and is only reclaimed below the checkpointed frontier, so on restart the replica rebuilds the overlay (its appended-above-commit_lsn set) from the journal; every appended entry is again readable, so no readable slot (≤ H, hence covered by the horizon guarantee) is ever lost.

Read: the readLSN protection. Reads are unicast to one replica, and the client decides where it is safe to read: it knows the LBA range of every write it issued. Each read carries a horizon H (the readLSN) = the client's contiguous quorum-acked prefix: every LSN ≤ H above L is quorum-durable (acked, not merely submitted), and everything ≤ L is covered by the login watermark guarantee (held in full by the login leader). Either way, a read never reflects a write that could vanish, and a servable replica always exists. The replica returns the latest version ≤ H for the range: if that write is not yet applied, it serves it straight from the journal-tail overlay (an overlay read: it already holds the data, so no fetch, and no out-of-order index write); writes above H are ignored even if the replica holds them (the sub-quorum tail, see below). A replica may serve read([lba, lba+len), H) iff:

  1. it is filled up to the login dLSN L (Synced ≥ L; L = the dLSN login returned = rs_commit_lsn). At/below L are prior-session writes whose LBAs the client can't name, so it must hold all of them; and
  2. above L, it has no Missing slot ≤ H overlapping [lba, lba+len); the client checks this against its LBA-aware per-replica Missing map.

So readability is per-LBA-overlap, not "no holes": a hole on an unrelated LBA does not block the read. Because the client only routes a read to a replica that can serve it, the read path never fetches from a peer. Inline fetch is resync-only. Consequence: login cannot complete until ≥1 member is filled to L (the leader fetches itself whole during login), and that member is the sole reader until each peer independently reaches Synced ≥ L.

Two-sided read safety (why a sub-quorum write can't leak). The eligibility rule above is the client side: never route a read to a replica missing a needed write ≤ H. The replica side is the horizon clamp: it serves only the latest version ≤ H, so a not-yet-quorum write it happens to hold (an entry above H, e.g. a broadcast only it received) is held but never returned. Both guards are needed and both key off H. There is no race: H is fixed when the read is issued and is monotonic (the client advances it past a write only once that write is quorum-acked), so a read that reaches a replica while a higher write is still sub-quorum simply ignores it, and the next read carries a higher H. The replica never needs to know quorum status; H carries that knowledge for it.

sequenceDiagram
    autonumber
    participant C as Client (initiator)
    participant R1 as Replica (overlapping hole)
    participant R2 as Replica (eligible)

    Note over C: horizon H = client's contiguous quorum-acked prefix.<br/>Client knows every write's LBA and each replica's Missing map.
    Note over C,R1: R1 has a Missing slot <= H overlapping [lba, lba+len) -> INELIGIBLE, skip
    Note over C,R2: R2 filled to L (Synced >= L) and no overlapping Missing <= H -> eligible
    C->>R2: read(term, H, lba, len)
    Note over R2: serve latest <= H from the index, or from the<br/>journal-tail overlay if not yet applied (no fetch). ignore anything above H
    R2-->>C: data
    Note over C,R2: Client routes only to a servable replica, so the READ PATH NEVER FETCHES.<br/>Peer fetch is resync-only.
Loading

CRAFT quorum-but-not-in-sync: LBA-overlap read routing

Worked example: three replicas with independent apply frontiers (everything quorum-durable is readable via the overlay, up to H); a read of LBA 13 that the client routes around S1's hole, served by S2 with no read-path fetch; and a sub-quorum in-flight write (106, received by S1 only, above H): next_lsn has moved to 107, but H holds at 105, and a replica serves only ≤ H, so S1 holds 106 without ever returning it until it quorum-acks or is resolved.

Two regions, split at the login dLSN L. At/below L are prior-session writes the client can't reason about per-LBA, so a member needs a full contiguous fill (Synced ≥ L) before it serves any read. Above L are this session's writes, gated per read by LBA-overlap. Synced ≥ L is the same "reach the watermark" gate as reconfig promotion (commit_lsn ≥ startLSN).

The client's job: keep an accurate LBA-aware Missing map per replica. It removes entries on write-ACKs, and learns resync-fills from {commit_lsn, last_append_lsn} carried in each keep_alive / get_lsns response (pruning any Missing slot ≤ that replica's commit_lsn); staying conservatively stale when unsure (treat a slot as Missing, route elsewhere) is safe, at the cost of under-using a replica. From the same reports it computes all_committed_lsn (the set-wide min) and piggybacks it on its next keep_alive/write, letting members reclaim journal opportunistically (see the journal-backing note). The map is interval-compressed and bounded: on overflow it collapses that replica to a single dirty-window summary (ineligible for reads until the commit_lsn certificate clears it; see the degraded-member note in Resync). After login the client initializes the horizon H := L: every slot ≤ L is held by the fully-filled login leader, and the Synced ≥ L gate routes reads at that horizon only to filled members, so reads are safe there while Phase-2 resync restores full replication in the background. (A slot ≤ L held by the leader alone means its other holder is already unavailable; losing the leader too before resync heals is a second failure, the same dual-failure accounting as the reconfiguration live-tail note.) Above L, the client advances H with its own contiguous quorum-acked prefix. H is per-read, and the precise safety rule is: a read may carry any H provided no unresolved slot ≤ H overlaps its range (a failed sub-quorum write is unresolved until the leader's round fills or Empties it; see Resync). So a failed write fences only the ranges it overlaps, never the whole horizon: acked writes above it stay readable.

Resync / catch-up: the FetchData loop

SyncRSCommitLSN is the primary recovery mechanism. It carries only an LSN watermark plus Empty verdicts (no data). The leader resolves every unresolved slot ≤ the watermark before proposing (below); on apply, a replica that is behind fills its missing slots from its peers, obeys the verdict list, then advances its commit watermark.

sequenceDiagram
    autonumber
    participant L as Leader (S1)
    participant S3 as Lagging replica (S3)
    participant P as Ahead peer (S2)

    Note over L,P: Trigger: login / keep_alive watchdog / periodic checkpoint<br/>(every N LSNs) / client request (failed write)
    Note over L: leader resolves every unresolved slot <= N first:<br/>fetch it from a holder, or record an Empty verdict on<br/>quorum-lacks evidence. never proposes past an unresolved slot
    L->>S3: RAFT replicate SyncRSCommitLSN(N, token, empty_slots[])
    L->>P: RAFT replicate SyncRSCommitLSN(N, token, empty_slots[])

    Note over S3: on apply: last_append < N<br/>=> missing slots {m1..mk} <= N
    loop until no missing slot <= N
        S3->>P: FetchData([missing lsns])
        S3->>L: FetchData([missing lsns])
        P-->>S3: JournalSlot[] {lsn,lba,len,data | is_empty | omitted}
        L-->>S3: JournalSlot[] {lsn,lba,len,data | is_empty | omitted}
        Note over S3: fill present slots (leader guaranteed to hold<br/>every non-Empty slot <= N). slots in empty_slots[] are<br/>permanent no-op holes: discard any local data there
    end
    Note over S3: commit_lsn = N (contiguous Synced<br/>watermark advances through N)

    Note over L,P: New/replacement member: bulk BACKLOG (<= startLSN) arrives first<br/>via the snapshot path (CP-sourced index+blocks). THIS loop fills the<br/>tail. Promote learner -> voter when commit_lsn >= startLSN.
Loading

Declaring Empty (leader verdict). FetchData returns one of present+data, not-present-here, or known-Empty, per slot. The peer's obligation is exact: it returns is_empty=true only for a slot it has positively marked Empty (via a prior verdict); for a slot merely absent from its journal it omits the entry from the response (that reads as not-present-here). So a responder never reports Empty for data it simply lacks. The verdict itself is made in exactly one place: the leader, as a precondition of proposing SyncRSCommitLSN(N). The leader resolves every unresolved slot ≤ N: it fetches the slot from any holder (after which it can serve it to the others), or, when a quorum of nodes is positively known to lack it (itself included; non-responders never count), records an Empty verdict. With N=3 the leader plus one confirming peer suffices, and it is safe by quorum intersection: any quorum known to lack a slot intersects every possible write quorum, so a quorum-durable write can never be declared Empty. If the leader can assemble neither a copy nor a confirming quorum, it does not propose past the slot. Verdicts ride the entry (empty_slots[]), so fill-vs-Empty is RAFT-serialized: replicas never declare Empty unilaterally (two replicas racing under asymmetric reachability could otherwise diverge, one filling from a holder while the other declares Empty off a timeout), they fill present slots and obey the verdict list. Commit advances past Empty slots but never past an unresolved Missing one. This makes mid-session resync structurally identical to login Phase 1b.

Reconciliation (Empty beats held data). A replica can hold Appended data in a slot the leader has since declared Empty (e.g. it was unreachable during the verdict round). On applying the entry that carries the verdict (or the login entries, for login-declared Empties) it discards that data. This is safe for the same reason login truncation is safe: a verdict-Empty slot was provably never quorum-durable, so the data was never acked, and discarding it is what keeps replicas convergent. Held data can never have been applied before the verdict arrives, because of the resolved-prefix rule below.

Resolving a failed (sub-quorum) write. A write that reaches only a minority (slot k lands on A alone; B and C never ack) is surfaced to the app as an IO error: it was never acked, so its outcome is undefined and both futures are legal. The client does not undo anything (it cannot even distinguish "B never appended" from "B appended and the ack was lost"); it requests an immediate resolution round (the client-request SyncRSCommitLSN trigger) instead of waiting for the periodic cadence. The leader resolves k like any other slot: fetch it from A, in which case the failed write simply completes late (the same benign false-include as the login watermark), or verdict it Empty if A is unreachable (A discards its copy via reconciliation when it rejoins). The failed slot is never reused: a retry of the same app write gets a fresh dLSN. Two invariants keep the unresolved window safe:

  1. Resolved-prefix-only. The client's commit targets and the leader's sync watermarks only ever cover resolved slots, so no replica, including the holder A, applies k before the verdict. No un-apply path is ever needed.
  2. Unresolved-slot read fencing (client). An unresolved slot acts like a Missing slot on every replica in the client's map. A read of a range it overlaps either runs at a horizon below it (serving the old data, which is stable under both futures) or, if the range also carries a later acked write, stalls until resolution: stalling is contract-compliant, while serving stale data for an acked write, or serving the unresolved data that an Empty verdict would then take back, is not. Ranges that do not overlap the failed slot are unaffected: acked writes above it stay readable throughout (per-read H + overlay).

Degraded member: bounded Missing tracking and re-admission. Missing sets are interval-compressed on both sides (a contiguous outage is one range, not one entry per write), and the client's per-replica map is bounded: past the bound it collapses to a single summary, "dirty over (X, Y], LBAs unknown", keeping live tracking above Y while the member is reachable. A summarized member serves no reads (an unknown-LBA window can overlap anything): the window behaves exactly like the below-L region, a stretch whose LBAs the client cannot name, healed only by a full contiguous fill. The client keeps broadcasting writes and keep_alive to it while reachable (the live edge stays warm, so resync owes only the window) and may fire the client-requested sync trigger to hurry healing. Re-admission needs no new machinery: the commit_lsn contiguity certificate in each keep_alive / get_lsns response clears the summary once it reaches Y, the same rule that prunes individual Missing entries. A member whose window fell past the journal retention floor takes the snapshot path instead; see the journal-backing note for the compaction tie that triggers it.

Truncation is login-only. SyncRSCommitLSN apply (watchdog / periodic) never truncates: above-watermark appends are the live tail the current client is still writing. Truncation above rs_commit_lsn is a login-only action, performed by the leader on every member (itself included) after InternalLogin commits. A member that missed the login truncates itself: applying the login entries from the RAFT log reveals a stale-term tail above the synced watermark, which it drops before anything else. That ordering is load-bearing because the new session reuses slot numbers above rs_commit_lsn: a replica cannot accept a new-term write before applying InternalLogin (the term check rejects it), and applying InternalLogin is exactly where it truncates, so stale-session and new-session data can never coexist in a slot.

Reconfiguration (membership change)

Principle: lean on homestore for the mechanics; CRAFT owns only the data-plane gating that homestore (being RAFT-log-based) cannot see. Verified against homestore dev/v8.x.

Constraint: single-member-at-a-time (add-one / remove-one; compose for bigger moves). This is the data-plane safety mechanism across a change: a quorum-acked write is held by ≥2 members of the old set and at most one member leaves, so at least one holder survives into the new set; the resync loop then restores it to full width (see the live-tail note below). No joint consensus.

Reduced-availability window is control-plane only. homestore flips OUT to a learner, commits HS_CTRL_START_REPLACE, then adds IN as a learner, so RAFT voting runs 2-of-2 from the OUT flip until IN promotes, i.e. for the catch-up duration (a voter failure in that window stalls the cold path, elections and logins, until recovery). The data plane is unaffected: the client keeps writing 2-of-3 to the still-synced {A,B,C} (the outgoing member is a RAFT learner but still holds and journals data, and the client refreshes its member set only at re-login on completion), and the data path is leaderless, so a running session keeps flowing even if a voter fails in the window. This is the data/consensus split doing its job.

What homestore gives us (lean fully):

  • replace_member: a complete 6-step state machine with a learner phase: flip OUT to learner → HS_CTRL_START_REPLACE → add IN as learner → monitor catch-up → remove OUT → HS_CTRL_COMPLETE_REPLACE. Leader/control-plane driven; does not need a CRAFT client. Listener gets on_start_replace_member / on_complete_replace_member / on_remove_member. nuraft supports learners natively via flip_learner_flag.
  • Baseline resync / snapshot for the backlog (≤ startLSN): nuraft triggers it past resync_log_idx_threshold, drives chunking/transport/truncation; the listener provides the payload via read_snapshot_obj/write_snapshot_obj.
  • The CP (checkpoint) subsystem is the consistency primitive: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary, so a CP is an atomic point-in-time of index+blocks. CRAFT sources its snapshot payload by forcing a CP flush at the snapshot LSN (cp_mgr().trigger_cp_flush), pinning that CP, and serializing the index + referenced blocks. homestore's resync metadata carries only a DSN marker; the listener owns the index+block payload.

Catch-up is pull-based (so the client never broadcasts to a catching-up member): the backlog comes via the snapshot path; the tail + small gaps come via the existing SyncRSCommitLSN → FetchData loop above. If the tail grows too large for FetchData (the missing list runs past the journal retention window), fall back to a fresh snapshot, which advances startLSN; this mirrors homestore's resync_log_idx_threshold fall-back-to-baseline.

Promotion (learner → voter) is gated by CRAFT's commit_lsn ≥ startLSN, not homestore's rLSN-based monitor (which is blind to CRAFT data, since data isn't in the log). startLSN starts as the dLSN at the moment of the change and advances with each snapshot pin (see Frozen view); the member reports commit_lsn in its keep_alive response. This gate is the data-plane quorum-safety mechanism for the backlog: a member must hold data through startLSN before it can count as a poll-quorum participant.

The live tail across removal: healed by resync, not fenced. Promotion protects the backlog. For the live tail: a write acked by {A, OUT} just before OUT leaves still has A as its surviving holder in the new set, and the other members were already sent it (the client broadcasts to all; a member without it is either acking late or already faulted) or will pull it via the normal SyncRSCommitLSN → FetchData loop. Losing that write now requires A to fail before resync lands it on a second member, on top of whatever kept the broadcast copy from landing in the first place: a dual failure, outside the f = 1 contract. A login can only miss the write if A is down at poll time (the poll is a broadcast; every responder counts). The identical window exists if OUT dies instead of being removed, and no fence can be placed on a death, so CRAFT does not fence removal: the data plane stays unaffected by a planned change, and the brief single-holder exposure is the same degraded-then-heal window every unplanned failure already creates.

Snapshot callbacks (modeled on homeobject's ReplicationStateMachine): the closest existing template. Mapping: homeobject PG → CRAFT partition; its shard+blob hierarchy → CRAFT's flat LBA index + block data (one level, simpler). Transfer is a resumable, objId-cursored message stream (volume metadata → index+data batches → LAST_OBJ_ID), with a leader-side iterator in snp_obj->user_ctx, a follower-side receive handler that persists progress to a superblock for mid-resync crash recovery, async block prefetch with bounded inflight, and per-message CRC. Inherited gotchas: consistent-LSN capture (homeobject needed verify/re-read because a concurrent higher commit can overwrite an LBA mid-stream; CRAFT sidesteps this by serializing from the pinned CP below), never block in the callbacks, keep the member not-readable until the last object (this is the startLSN gate).

Frozen view (decided): CP-pinned and bounded. On snapshot need, trigger_cp_flush, pin that CP, and serialize the committed index+blocks as of it. Because the payload is read from a pinned CP (a point-in-time freeze), writes that land after the CP cannot mutate it, so no verify-on-read is needed for what the snapshot ships. The snapshot's startLSN = commit_lsn at the pinned CP, stored in the snapshot metadata so the tail FetchData loop resumes correctly. Bound the pin (time / size); a transfer that exceeds the bound releases the CP, re-pins a fresher one, and restarts with the new startLSN. Restartable: the follower checkpoints its receive cursor + startLSN to its superblock; a leader failure mid-transfer restarts the snapshot, and the follower resumes from its cursor or discards and restarts cleanly.

Self-elected stand-in client (recovery while detached). homestore's replace_member runs without a client, but CRAFT data-plane convergence normally rides client activity: the RAFT leader always proposes SyncRSCommitLSN (a client cannot propose a RAFT entry), triggered by the client's keep_alive / commit traffic, its watchdog, or the periodic checkpoint. When detached there is no traffic to ride, so the leader self-elects as a stand-in client to drive just that. No app IO when detached means no live tail, so it is pure backlog catch-up. Leader-only, so no stand-in split-brain.

Logout / lease (decided): hybrid. A locally-evaluated lease (the keep_alive watchdog) detects that the client is gone; a committed RAFT InternalLogout / lease-expiry entry grants the authority to act. The leader self-elects only after that entry commits (serialized, survives leader failover, and avoids a new leader re-deciding blindly), and yields on the next committed InternalLogin. The committed logout also marks "no active client", which sharpens the login fencing boundary (a following login is unambiguously a fresh session).

Net CRAFT-owned build list (small; everything else is homestore):

  1. The four snapshot callbacks shipping CRAFT's committed index + blocks.
  2. The commit_lsn ≥ startLSN promotion gate (the keep_alive response carries commit_lsn), interposed on complete_replace_member.
  3. The committed logout / lease record + leader self-election.
  4. Term-bump / re-login on HS_CTRL_COMPLETE_REPLACE so an attached client refreshes its member set. Re-login is triggered by ETERM: the client re-logins to the leader presenting its client_token; the leader accepts only if that token is the current owner (returns fresh term + members) and rejects a superseded token, so a deposed client cannot reclaim the volume. In-flight writes are re-driven under the new term.

Correctness deep-dives

1. The recovery watermark max(quorum.last_append) is forced

Login computes rs_commit_lsn = max over a polled quorum of last_append_lsn. It can look like over-committing (promoting an LSN only one node holds), but under block semantics it is the only correct choice; there is no decision to make.

Indistinguishability. N=3, quorum=2. An LSN k shows up on exactly one node of the polled quorum. Two scenarios are observationally identical from {S2,S3}:

  • Case A: k reached S1+S2 (true quorum, durable, client acked the app), then S1 went unavailable. Poll {S2,S3}: only S2 has k.
  • Case B: k reached only S2, never made quorum, client never got its ack. Poll {S2,S3}: only S2 has k.

To never drop the durable one (A) you must include k, which means you sometimes also include the non-durable one (B).

Cost asymmetry makes "include" the safe direction. false-exclude = drop an LSN the app was told was durable → silent committed-data loss (catastrophic). false-include = land an in-flight write whose absence was never promised to anyone → benign (the filesystem already handles crash consistency for in-flight writes; the new term fences the old client out).

The theorem to write into the design. If k was ever quorum-acked, ≥2 nodes hold it; any polled quorum of 2 intersects that set, so at least one polled node reports last_append ≥ k, hence max(quorum.last_append) ≥ k. The real constraint this names: write-quorum ∩ poll-quorum ≠ ∅, always, automatic on fixed membership. Across a membership change the guarantee is carried differently: at most one member leaves, so every quorum-acked write keeps at least one surviving holder, and resync restores full width; losing the tail then takes a dual failure. See the live-tail note in Reconfiguration.

2. Read eligibility: LBA-overlap, not a contiguous gate

Reads are unicast to one client-chosen replica, and readability is per-LBA, not per-contiguous-LSN. A replica may serve read([lba, lba+len), H) iff it is filled to L (the two-region split below) and it holds every write ≤ H overlapping the range; equivalently, no Missing slot ≤ H overlaps [lba, lba+len). A hole on an unrelated LBA is irrelevant. Gating reads on a contiguous Synced prefix would be wrong twice over: it linearizes independent in-flight I/O (holding a higher LSN's readability for an unrelated lower one) and it lets one slow or lost write stall all higher reads, which is poison for a block device.

The client enforces it. It knows the LBA range of every write it issued and keeps a per-replica, LBA-aware Missing map, so it only routes a read to a replica that can serve it. Hence the read path never fetches. Peer fetch is resync-only.

Mental model (from the RAID1 it replaces). This per-replica Missing map acts as a multi-dimensional dirty-bitmap. A RAID1 mirror keeps a one-dimensional dirty-region log so a stale mirror resyncs only the dirty regions instead of the whole disk; CRAFT keys the same idea by (replica, LBA range, dLSN). Reads route around exactly the stale ranges of each replica and resync fetches exactly those, so a partially-behind replica still serves every range it is clean on.

Crucially, unlike a RAID1 bitmap the dirty state is not opaque to the replicas. A RAID1 dirty-region log lives in the writer (the md host, the DRBD primary) or an array controller, so the copies are dumb and that one writer drives every resync. In CRAFT the same information is materialized in each replica's own slot state (Missing / Empty / the commit_lsn watermark), so replicas resync from each other via SyncRSCommitLSN + FetchData with no client in the loop. That moves resync traffic off the client and is why catch-up is server-side, not writer-driven. (This, plus quorum-ack and staleness-routed reads, is what separates CRAFT from DRBD, which is otherwise the same leaderless single-writer broadcast; see Background.)

The one place a contiguous watermark is right: the login dLSN L. At/below L are prior-session writes whose LBAs the client cannot name, so it can't run the overlap check there: a member must be fully filled to L (Synced ≥ L) before it serves any read. Above L (this session's writes) LBA-overlap governs. So the true rule is a two-region split at L, and Synced ≥ L is the same "reach the watermark" gate as reconfig promotion (commit_lsn ≥ startLSN).

Consequence: login can't finish until ≥1 member is filled to L (the leader syncs itself whole), and that member is the sole reader until each peer reaches Synced ≥ L, a window kept short by there usually being few in-flight I/Os, and by the leader-stand-in having pre-driven resync while detached.

Decisions already made

(So delegates don't relitigate.)

  • Recovery watermark = max(quorum.last_append): forced, not a tuning choice (see the deep-dive). There is no safe "conservative" commit_lsn variant.
  • Consistency target = single-writer block semantics (causal read-after-write + read stability + convergence), not linearizability/transactions.
  • Read eligibility: client-driven, per-LBA-overlap. Each read carries a horizon H (client's contiguous quorum-acked prefix); a replica may serve iff filled to the login dLSN L (Synced ≥ L) and it has no Missing slot ≤ H overlapping the range. The read path never peer-fetches (fetch is resync-only); post-login the leader is the sole reader until each peer reaches Synced ≥ L.
  • Durability: FUA write-through; no FLUSH; homestore logstore already provides it; device power-safety guaranteed by deployment.
  • dLSN = deterministic merge key (highest-per-LBA wins everywhere); no overlapping-write serialization.
  • Broadcast-quorum, not chain replication: chain's only real edge is bandwidth locality on a dedicated storage fabric; HomeBlocks runs one shared Ethernet (25 to 100 Gbps, same top-of-rack for client and storage), where aggregate bytes are topology-invariant, so CRAFT takes the latency win (1 RTT vs N serial hops) and keeps storage-node NICs forwarding-free. Revisit only if a dedicated storage fabric appears. See Background.
  • Durability / ordering: FUA write-through (acked = quorum-durable); FLUSH is an immediately-acked no-op (nothing is ever cache-dirty) and is not an ordering barrier (the block layer provides none since 2010); ordering is the guest's via wait-for-completion, which CRAFT honors because ack = durable.
  • Scope = CRAFT replicates a single partition. A volume comprises one or more partitions; dLSN is per-partition (dense) and does all the CRAFT work (merge key, watermarks, Missing/Empty, recovery).
  • Reconfiguration: single-member-at-a-time; lean on homestore replace_member/learner/snapshot/CP; CRAFT owns 4 items; catch-up is pull-based; promotion gated by commit_lsn ≥ startLSN; detached recovery via a leader self-elected stand-in client.
  • No removal fence: across a replace, acked-tail safety = broadcast to all + at least one surviving holder (single-member-at-a-time) + server-side resync. Losing tail data requires a dual failure, outside the f = 1 contract; a fence would pause the data plane during a planned change and cannot close the equivalent unplanned-death window.
  • Index apply is strictly in dLSN order at the contiguous commit frontier (skipping Empty); readability attaches to Appended via the journal-tail overlay. No per-entry index versions, exact-prefix snapshots, deterministic recovery; a Missing hole pauses apply + block reclaim above it, never reads.
  • Journal reclaim floor = min(checkpointed apply frontier, all_committed_lsn), with all_committed_lsn computed by the client and piggybacked on keep_alive/writes (exact convergence at login); a silent member freezes it until the retention window relegates that member to baseline resync.
  • Empty is a leader verdict (replaces per-replica declaration, which races a concurrent fill under asymmetric reachability): the leader resolves every slot ≤ N (fetch from a holder, or verdict on quorum-lacks evidence, itself included) before proposing SyncRSCommitLSN(N); verdicts ride the entry (empty_slots[]); replicas obey and never declare unilaterally. Mid-session resync = login Phase 1b.
  • Failed-write (sub-quorum) resolution: error to the app, then resolve forward (fill-or-Empty) via a client-requested sync round; the dLSN is never reused; commit targets and sync watermarks cover only resolved slots (so a holder never applies pre-verdict); the client read-fences ranges overlapping an unresolved slot (stall beats stale-or-unstable).
  • Bounded Missing tracking: interval-compressed maps; on overflow the client collapses a replica to a dirty-window summary (read-ineligible, still receiving writes and keep_alive), re-admitted by the commit_lsn contiguity certificate. RAFT-log compaction is tied to the journal reclaim floor, so a voter that falls past the retention window becomes log-behind and nuraft's standard snapshot machinery (the same four callbacks) catches it up: no separate trigger to build.
  • Session fencing: a login-scoped quiesce barrier (an is_login poll flag) fences the prior writer at the login poll, turning a would-be silent ack-then-truncate into an explicit reject; the attach layer is also expected to hard-fence on failover (defense in depth). See Login.

Open items

(The logout/lease record and the snapshot frozen-view mechanism are decided; see Reconfiguration.)

  • Real transport (IP/TCP): planned (CRAFT-1). The in-process multi-HomeBlkDisk + direct-API-call path is a development scaffold to build and validate the protocol in parallel with transport selection, not the end state. CraftConnector stays transport-agnostic so the chosen transport drops in behind it.

Implementation work breakdown

Full per-story acceptance criteria live in the repo at docs/craft/subtasks.md; this is the shape.

Story Scope Depends on
S1 CraftReplDev foundation: class, per-partition state, journal backend, multi-member RAFT group, listener skeleton (landed) none
S2 Write path (write: term check, journal append, last_append update, out-of-order tolerant, zero-copy) S1
S3 Commit + Read path (commit/keep_alive advance the in-order apply frontier + reclaim; read carries horizon H, client-routed by LBA-overlap, served from the index or the journal-tail overlay, no read-path fetch, no index write on reads; watchdog) S2
S4 Truncate (drop journal entries above an LSN; clear Missing above it) S1
S5 RAFT entries: SyncRSCommitLSN (leader pre-resolution + empty_slots[] verdicts) + InternalLogin apply; checkpoint triggers (periodic / watchdog / client-requested) S1, S6
S6 Peer data exchange: get_lsns/get_rs_commit_lsn, fetch_data S1
S7 Login orchestration (leader-side; watermark = max(quorum.last_append)) S2, S4, S5, S6
S8 CRAFT volume lifecycle: create multi-member group, recover state from journal/superblock; journal reclaim floor + RAFT-log compaction tied to it (the returning-voter snapshot trigger) S1
S9 CraftConnector frontend (transport-agnostic; in-process scaffold first) S2, S3, S7
S10 Reconfiguration: replace_member integration + 4 snapshot callbacks (CP-sourced payload, homeobject-modeled) + commit_lsn ≥ startLSN promotion gate + logout/lease record + leader self-election + re-login on complete S5, S6, S8

Development scaffold: all of this is implementable and testable now with multiple in-process HomeBlkDisk instances making direct API calls (catch-up is just inter-instance calls) in parallel with real-transport selection.

HomeBlocks

CRAFT

Clone this wiki locally