Skip to content

CRAFT Design

Jump to bottom
Brian Szmyd edited this page Jul 1, 2026 · 21 revisions

CRAFT: Client Assisted RAFT

Scope. This page is the HomeBlocks (server) side of CRAFT. The remote client/initiator side is a separate epic; it is summarized here only at its API boundary. Audience: engineers picking up the CRAFT server work who don't yet have the homeblocks internals in their head.

Contents

What CRAFT is

CRAFT (Client Assisted RAFT) is the replication protocol for HomeBlocks block volumes. Its one defining move is to split the data path from the consensus path:

  • Data path (hot, leaderless). The single client that owns a volume assigns each write a monotonic LSN and broadcasts it directly to all replicas in parallel. A write is durable once a quorum acks the journal append. The RAFT leader plays no role here.
  • Consensus path (cold, RAFT). RAFT is used for only three things: leader election, establishing the single-writer session (InternalLogin → term + client_token), and agreeing recovery watermarks (SyncRSCommitLSN). Write data never enters the RAFT log, only LSN watermarks and login tokens do.

A second structural choice runs through everything: durability and readability are decoupled. Appended (quorum journaled the data, durable) ≠ Committed (applied to the LBA index, readable). The client drives commit lazily, via commit, keep_alive, or piggybacked on the next write/read. Commit advances eagerly and independently per replica; it is not a synchronous quorum step.

Glossary

Sequence numbers & watermarks

Term Meaning
LSN Log Sequence Number.
dLSN Data LSN: the per-partition number the client assigns to each write (its "position").
gLSN Global (volume-level) LSN. In CRAFT a volume maps to one partition, so gLSN and dLSN coincide.
rLSN RAFT log index. Distinct from dLSN (data is not in the RAFT log).
last_append_lsn Highest LSN a replica has appended (present in its journal).
commit_lsn (≡ Synced) The contiguous committed prefix on a replica (Empty slots skipped): everything ≤ it is present and applied. The fill watermark.
rs_commit_lsn The replica-set commit LSN chosen at login = max(quorum.last_append).
L (login dLSN) The dLSN login returns (= rs_commit_lsn). Boundary below which the client cannot name a write's LBA (prior-session data).
H (readLSN / horizon) The client's contiguous quorum-acked prefix, stamped on each read; the read reflects writes ≤ H.
startLSN The dLSN at the moment of a member change; the promotion watermark for the joining member.

Slot / write states (a slot is one dLSN on one replica)

State Meaning
Appended Data is in the journal and durable at quorum, but not yet readable.
Committed Applied to the LBA index; readable.
Missing A write known to exist but not held here; must be fetched from a peer (or declared Empty).
Empty A write that reached no replica and can't be fetched; a permanent no-op hole that commit skips.

Consensus & session

Term Meaning
RAFT / Raft Consensus protocol (leader election + replicated log). CRAFT uses it only on the cold path.
Paxos The older consensus family Raft descends from.
term RAFT term; bumped on each client login and carried on every IO to fence stale clients.
client_token Identifies the single writer; set by InternalLogin.
quorum A majority of the replica set (2 of 3).
f Number of tolerated failures (f = 1 for a 3-node set).
RS / Replica Set The nodes holding one partition's copies (typically 3).
partition A replicated region of a volume; in CRAFT, partition ≈ volume.

RPCs, RAFT entries & rules

Term Meaning
SyncRSCommitLSN RAFT entry that advances the replica-set commit watermark; the primary resync trigger.
InternalLogin RAFT entry that sets client_token + term (single-writer enforcement).
GetRSCommitLSN Non-RAFT peer query of a replica's {commit_lsn, last_append_lsn}.
FetchData Non-RAFT peer pull of missing journal slots (the resync data path).
CommitAndRead A read that commits (materializes) the touched appended entry into the index, then serves.
merge key The convergence rule: highest dLSN per LBA wins, on every replica.

Components & systems

Term Meaning
HomeBlocks The block-volume store (the server side this page covers).
HomeStore The underlying storage engine (journal / index / blkdata / RAFT / CP).
homeobject A blob store on HomeStore; the template CRAFT follows for snapshot + replace_member.
CraftReplDev New per-volume CRAFT class (parallel to HomeStore's repl_dev).
CraftConnector Client-facing RPC frontend that translates to CraftReplDev calls.
CraftRaftListener repl_dev_listener handling the two CRAFT RAFT entries + snapshot / replace_member callbacks.
CraftJournalBackend Mockable journal abstraction wrapping the HomeStore logstore.
RaftReplDev / ReplDisk HomeStore's replication device classes.
replace_member / learner / flip_learner_flag HomeStore membership-change primitives CRAFT leans on.
ublk / HomeBlkDisk The in-process test adapter used as the development scaffold.
PacificA Primary-backup replication framework (consensus only for config); a CRAFT touchstone.
CORFU Shared-log storage architecture (clients write directly to storage units); a CRAFT touchstone.

Storage & durability

Term Meaning
LBA Logical Block Address.
FUA Force Unit Access: every write is durable on completion (write-through; no writeback cache, no FLUSH).
CP HomeStore Checkpoint: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary (a consistency point).
DSN Data Sequence Number: a HomeStore marker carried in snapshot-resync metadata.
logstore HomeStore's write-through journal (backs CraftJournalBackend).

Background: Raft, Paxos, PacificA, CORFU, and CRAFT

CRAFT borrows one idea each from a few well-known systems. If these names are unfamiliar this is the quick map; if they're familiar, it pins down exactly which idea CRAFT takes from each.

Raft & Paxos: consensus protocols. Both solve agreement among distributed nodes despite failures: how a set of replicas commit to the same ordered log of operations. Paxos is the older, foundational family; Raft was designed to be easier to understand and implement while giving the same replicated-log guarantees (an elected leader replicates an ordered log to followers). They answer "how do replicas agree?"

PacificA: primary-backup replication for storage. A framework for keeping replicas of a log-based storage shard consistent. A primary orders the writes and forwards them to backups; correctness across failures is delegated to a separate, consensus-backed configuration manager that owns membership. The key idea: keep consensus off the per-write path and pay for it only on rare reconfiguration. It answers "how do I replicate a storage shard with a primary and backups?"

CORFU: a shared-log architecture. Turns a cluster of flash devices into one global append-only log shared by many clients. A sequencer hands out log positions; clients then write directly to the storage units at those positions (no primary in the data path); a separate consensus-backed layer handles reconfiguration and hole-filling. It answers "how do I expose one global durable log across many devices, with clients writing directly?"

How they layer. Raft/Paxos are general-purpose "get a consistent log." PacificA is a specific primary-backup replication scheme with consensus only for config. CORFU is a shared-log abstraction others build on, where clients write straight to storage.

System The question it answers
Paxos / Raft How do replicas agree on an ordered log?
PacificA How do I replicate a storage shard with a primary + backups, keeping consensus off the write path?
CORFU How do I expose one global durable log across many devices, with clients writing directly?
CRAFT How do I replicate a single-attach block volume when the client itself can order the writes?

How CRAFT fits. CRAFT takes one idea from each and specializes it for single-attach block volumes:

  • From PacificA: keep consensus off the write path; use a separate consensus layer only for configuration-class decisions. CRAFT's consensus layer is Raft, and it touches only the cold path: leader election, InternalLogin (who is the single writer + term), SyncRSCommitLSN (recovery watermark), and membership. Raft here plays exactly PacificA's "configuration manager" role. The twist: PacificA still has a primary node that orders and forwards writes; CRAFT has no data-path primary at all.
  • From CORFU: the writer assigns positions (LSNs) and writes directly to every storage replica, bypassing any leader/primary on the hot path. The difference: CORFU runs a separate sequencer serving many concurrent clients over a shared log; CRAFT relies on block volumes being single-attach, so the one client per volume is its own sequencer: no sequencer service and no per-write coordination.

In one line: CRAFT = CORFU-style direct-to-replica writes by a client that is its own sequencer, with Raft as a PacificA-style configuration manager that never touches the data path. That combination is only safe because a block volume has a single writer (see The contract CRAFT owes).

PacificA to CORFU to CRAFT: the same design elements

The lineage in one picture: consensus (blue) stays off the data path in all three, the data path (orange) gets more direct, and the sequencer (yellow) moves from an external service in CORFU into the single client itself in CRAFT.

The contract CRAFT owes (block-device semantics)

CRAFT is a single-writer block device, not a transactional or linearizable store. Holding it to the right (weaker) contract is what makes the design tractable. It owes exactly:

  • Durability: a write that was acked survives crash, up to f failures. All homeblocks I/O is FUA (see Durability model): there is no FLUSH op and no writeback cache, so "acked" already means "durable." Un-acked (in-flight) writes have undefined outcome on crash; the filesystem/application owns crash consistency for them. We do not treat in-flight writes as transactions.
  • Read-after-write: only for causally ordered ops (the app waited for the write's ack before issuing the read). Concurrent read+write to the same LBA is a race with an undefined result, by the block contract.
  • Read stability: once a newer value of an LBA has been observed, a later read does not regress to an older one.
  • Replica convergence: all replicas eventually agree on the same value per LBA.

Not owed: cross-LBA ordering, atomicity of concurrent overlapping writes, transactions.

Consequence: there is no order among in-flight I/O. All submitted I/O races naturally; the application serializes (waits for ack) when it needs ordering. So the dLSN is not preserving application order (there is none for concurrent I/O). Its job is to be a deterministic merge key: highest dLSN to an LBA wins, on every replica, so replicas converge to the same value even when two writes race. The client assigns dLSNs in submission order; "highest-LSN-per-LBA wins everywhere" is the whole rule. No overlapping-write serialization logic is needed.

Why (motivation)

  1. Kill the leader bottleneck. Classic RAFT funnels every byte through the leader (client → leader → log → fan-out → apply). The leader's egress is N× the write bandwidth and it double-writes (log, then state machine). CRAFT pushes the fan-out onto the client and writes data to the journal once, a throughput and write-amplification win.
  2. Lower write latency. Client → replica broadcast is one hop to durability (quorum append) vs. two through a leader.
  3. The single-writer constraint is essentially free here. Block volumes are single-attach (EBS-like), so there is naturally one writer per volume. Given one writer, that writer is the total-order authority. You don't need consensus to order writes. That is the load-bearing insight that makes a leaderless, consensus-free data path safe. RAFT is then only needed for the genuinely ambiguous things: who is the one writer (split-brain) and what is durable after a crash (recovery watermark).
  4. Keep RAFT small and off the hot path. Login and resync are low-frequency, so the RAFT log stays tiny and never carries data.
  5. No homestore changes. Built entirely on existing homestore journal/index/ blkdata primitives, so the blast radius stays in the HomeBlocks layer.

Where CRAFT plugs into HomeBlocks

Today (solo, data-through-log):

home_blocks API (create_volume / async_read / async_write)
  -> HomeBlocksImpl / volume_mgr
  -> Volume   (owns a homestore::repl_dev created SOLO, empty member list)
              IO via rd()->async_write / async_read / async_write_journal
  -> HBListener (repl_dev_listener) handles RAFT callbacks
  -> homestore (RaftReplDev data-through-log, journal, index, blkdata)

With CRAFT (leaderless data path):

Client (initiator) --broadcast writes--> every replica's CraftConnector
  CraftConnector            (client-facing frontend; transport-agnostic)
    -> CraftReplDev         (one per volume; multi-member RAFT group)
       - write / read / commit / keep_alive / login / ...  (CRAFT API)
       - CraftJournalBackend  -> homestore logstore journal (write-through / FUA)
       - LBA index + blkdata for reads (CommitAndRead materializes on demand)
       - CraftRaftListener: RAFT only for SyncRSCommitLSN + InternalLogin
         (+ snapshot / replace_member callbacks for reconfiguration)
    -> homestore (RaftReplDev for consensus only; journal / index / blkdata / CP / snapshot)

Component map (homeblocks-concrete):

Component New/changed Location Responsibility
CraftReplDev new src/lib/craft/craft_repl_dev.* The CRAFT API + per-partition state {commit_lsn, last_append_lsn, client_token, term}
CraftJournalBackend new (mockable) src/lib/craft/craft_repl_dev.* Journal slot write/read/truncate; production impl wraps the homestore logstore
CraftRaftListener new src/lib/craft/craft_repl_dev.* on_commit dispatch for the two RAFT entry types; snapshot + replace_member callbacks
CraftConnector new src/lib/craft/ (TBD) Translate client RPCs ↔ CraftReplDev calls; leader redirect; term checks; transport-agnostic
Volume wiring changed src/lib/volume/volume.cpp Create a multi-member CraftReplDev for CRAFT-mode volumes; route IO
HomeBlocksImpl wiring changed src/lib/homeblks_impl.cpp create_repl_dev_listener returns a CraftRaftListener; volume lifecycle/recovery
volume_info.replication_mode exists src/include/homeblks/home_blocks.hpp DISABLED (today's solo path) vs CRAFT

Note. There is no ScstConnector in the current tree. The de-facto frontend today is the public async_read/async_write API plus the ublk adapter (src/test/homeblk_disk). "Add CraftConnector" = add the client→CraftReplDev translation layer.

Durability model: FUA write-through

All homeblocks I/O is FUA. There is no FLUSH operation and no writeback cache to flush or barrier. Every write must be durable-on-ack (write-through); otherwise a simultaneous restart of all replicas could lose acknowledged writes.

This maps directly onto homestore's logstore, which is already write-through (verified against homestore dev/v8.x):

  • append_async only buffers the record (log_dev.cpp:288, flush_if_necessary at :297). Flushing is group-commit (size threshold or timer).
  • LogDev::flush() issues a synchronous device write (m_vdev_jd->sync_pwritev(...), log_dev.cpp:532) and calls on_flush_completion() only after it returns (:539).
  • on_flush_completion (log_dev.cpp:545) raises each append's completion callback (on_write_completion at :583).

So the awaited append completion fires only after the bytes are synchronously written to the journal device, never on buffering. Group-commit just batches concurrent appends into one sync write and acks them together (ideal for CRAFT's broadcast-and-await-quorum pattern).

Requirement on CRAFT: CraftReplDev::write must await the logstore append completion before acking the client. Then quorum-ack ⇒ quorum-durable ⇒ an all-replicas restart cannot lose an acked write.

Device-level power-safety: the journal device is opened O_DIRECT (device_manager.cpp:47) and homestore skips fsync in direct-IO mode. O_DIRECT

  • sync write lands data on the controller; power-safety relies on the drive being FUA / write-cache-off / power-loss-protected. This is settled operationally: homeblocks already runs the journal this way in production.

Login

Login establishes a new term, converges replica state to a starting dLSN, and enforces single-writer exclusivity, all before any IO is accepted. It is leader-only; a follower replies ENOTLEADER plus the leader endpoint.

sequenceDiagram
    autonumber
    participant C as Client (initiator)
    participant L as Leader (S1)
    participant S2 as Replica S2
    participant S3 as Replica S3

    C->>L: login(client_token, vol_id)
    Note over L: Leader-only. Follower replies<br/>ENOTLEADER + leader endpoint.

    Note over L,S3: Phase 1: collect replica LSN state (non-RAFT broadcast)
    L->>S2: GetRSCommitLSN(term, my_commit, my_append)
    L->>S3: GetRSCommitLSN(term, my_commit, my_append)
    S2-->>L: {commit=5, last_append=7}
    S3-->>L: {commit=10, last_append=11}
    Note over L: rs_commit_lsn = max(quorum.last_append) = 11

    opt Leader behind (self.last_append < 11)
    L->>S3: FetchData(missing slots up to 11)
    S3-->>L: JournalSlot[] (lba,len,data)
    Note over L: append to own journal
    end

    Note over L,S3: Phase 2: SyncRSCommitLSN(11) via RAFT (data NOT in log)
    L->>S2: RAFT replicate SyncRSCommitLSN(11, token)
    L->>S3: RAFT replicate SyncRSCommitLSN(11, token)
    Note over S2,S3: on apply, if behind FetchData up to 11.<br/>commit_lsn = 11. if last_append > 11, truncate(11)

    Note over L,S3: Phase 3: InternalLogin(token, term+1) via RAFT
    L->>S2: RAFT replicate InternalLogin(token, T+1)
    L->>S3: RAFT replicate InternalLogin(token, T+1)
    Note over S2,S3: store client_token, term=T+1.<br/>reject IO with any other term

    L-->>C: LoginResult{members, dLSN=11, term=T+1, gLSN}
    Note over C: open a queue per member, begin IO at dLSN+1
Loading

The watermark rs_commit_lsn = max(quorum.last_append) is forced, not a tuning choice; see the recovery-watermark deep-dive. (dLSN is this partition's LSN; gLSN is the volume-global LSN. In CRAFT a volume maps to a single partition, so gLSN and dLSN coincide.)

Write, Commit, Read, and readLSN protection

Write. The client assigns ++next_lsn, broadcasts write(term, lsn, glsn, lba, len, data) to all replicas. Each validates term, appends the data to its journal at slot lsn (zero-copy, may arrive out of order), advances last_append_lsn, and acks, per write, since in-flight I/O is unordered. Durable at quorum-ack (Appended), but durable ≠ readable: a write becomes readable only once it is committed (materialized into the LBA index), which the client drives, or which a read of that entry triggers on demand (CommitAndRead, see Read). No LBA-index update happens on the write itself.

Commit. Commit makes a durable write readable by materializing it into the LBA index; the client drives it (lazily, via commit / keep_alive / piggyback), and a read can trigger it on demand (CommitAndRead). Because in-flight I/O is unordered, readability is per-write, not a contiguous prefix: a higher LSN can be readable while a lower one is still a hole, with overlaps resolved by the merge key (highest LSN per LBA wins).

The per-partition scalar commit_lsn (≡ Synced) is the contiguous committed prefix, the highest LSN with no unresolved hole below it (Empty slots are skipped). It is the fill watermark used by resync and reconfiguration, and the read gate in the ≤ L region (below); individual writes above it are made readable per-LBA on demand, which is why readability is not tied to it.

Read: the readLSN protection. Reads are unicast to one replica, and the client decides where it is safe to read: it knows the LBA range of every write it issued. Each read carries a horizon H (the readLSN) = the client's contiguous quorum-acked prefix: every LSN ≤ H is quorum-durable (not merely submitted), so a read never reflects a sub-quorum write that could vanish and a servable replica always exists. The replica returns the latest version ≤ H for the range: if that write is present but not yet committed, it commits it first (CommitAndRead, it already holds the data, so no fetch) and serves from the index; writes above H are ignored. A replica may serve read([lba, lba+len), H) iff:

  1. it is filled up to the login dLSN L (Synced ≥ L; L = the dLSN login returned = rs_commit_lsn). At/below L are prior-session writes whose LBAs the client can't name, so it must hold all of them; and
  2. above L, it has no Missing slot ≤ H overlapping [lba, lba+len); the client checks this against its LBA-aware per-replica Missing map.

So readability is per-LBA-overlap, not "no holes": a hole on an unrelated LBA does not block the read. Because the client only routes a read to a replica that can serve it, the read path never fetches from a peer. Inline fetch is resync-only. Consequence: login cannot complete until ≥1 member is filled to L (the leader fetches itself whole during login), and that member is the sole reader until each peer independently reaches Synced ≥ L.

sequenceDiagram
    autonumber
    participant C as Client (initiator)
    participant R1 as Replica (overlapping hole)
    participant R2 as Replica (eligible)

    Note over C: horizon H = client's contiguous quorum-acked prefix.<br/>Client knows every write's LBA and each replica's Missing map.
    Note over C,R1: R1 has a Missing slot <= H overlapping [lba, lba+len) -> INELIGIBLE, skip
    Note over C,R2: R2 filled to L (Synced >= L) and no overlapping Missing <= H -> eligible
    C->>R2: read(term, H, lba, len)
    Note over R2: CommitAndRead the latest <= H if not yet committed<br/>(it has the data, no fetch), then serve from index. ignore anything above H
    R2-->>C: data
    Note over C,R2: Client routes only to a servable replica, so the READ PATH NEVER FETCHES.<br/>Peer fetch is resync-only.
Loading

CRAFT quorum-but-not-in-sync: LBA-overlap read routing

Worked example: three replicas, quorum-durable but each committed to a different extent (independent commit); a read of LBA 13 that the client routes around S1's hole, served by S2 with no read-path fetch.

Two regions, split at the login dLSN L. At/below L are prior-session writes the client can't reason about per-LBA, so a member needs a full contiguous fill (Synced ≥ L) before it serves any read. Above L are this session's writes, gated per read by LBA-overlap. Synced ≥ L is the same "reach the watermark" gate as reconfig promotion (commit_lsn ≥ startLSN).

The client's job: keep an accurate LBA-aware Missing map per replica (it removes entries on write-ACKs, and learns resync-fills from members' keepalive/read responses), staying conservatively stale: if unsure, treat a slot as Missing and route elsewhere. That is safe; the only cost is under-using a replica.

Resync / catch-up: the FetchData loop

SyncRSCommitLSN is the primary recovery mechanism. It carries only an LSN watermark (no data). On apply, a replica that is behind pulls the missing slots from a peer, then advances its commit watermark.

sequenceDiagram
    autonumber
    participant L as Leader (S1)
    participant S3 as Lagging replica (S3)
    participant P as Ahead peer (S2)

    Note over L,P: Trigger: login / keepalive watchdog /<br/>periodic checkpoint (every N LSNs)
    L->>S3: RAFT replicate SyncRSCommitLSN(N, token)
    L->>P: RAFT replicate SyncRSCommitLSN(N, token)

    Note over S3: on apply: last_append < N<br/>=> missing slots {m1..mk} <= N
    loop until no missing slot <= N
        S3->>P: FetchData([missing lsns])
        P-->>S3: JournalSlot[] {lsn,lba,len,data | is_empty}
        alt slot present on peer
            Note over S3: append to journal.<br/>last_append = max(last_append, lsn).<br/>drop from missing set
        else slot on no peer
            Note over S3: mark slot Empty (permanent no-op hole).<br/>commit may advance past it
        end
    end
    Note over S3: commit_lsn = N (contiguous Synced<br/>watermark advances through N)

    Note over L,P: New/replacement member: bulk BACKLOG (<= startLSN) arrives first<br/>via the snapshot path (CP-sourced index+blocks). THIS loop fills the<br/>tail. Promote learner -> voter when commit_lsn >= startLSN.
Loading

A slot found on no peer becomes Empty, a permanent no-op hole (a write that never landed anywhere; the filesystem owns its crash consistency). Commit advances past Empty slots but never past an unresolved Missing one.

Reconfiguration (membership change)

Principle: lean on homestore for the mechanics; CRAFT owns only the data-plane gating that homestore (being RAFT-log-based) cannot see. Verified against homestore dev/v8.x.

Constraint: single-member-at-a-time (add-one / remove-one; compose for bigger moves). Preserves write-quorum ∩ poll-quorum ≠ ∅ by construction, no joint consensus.

What homestore gives us (lean fully):

  • replace_member: a complete 6-step state machine with a learner phase: flip OUT to learner → HS_CTRL_START_REPLACE → add IN as learner → monitor catch-up → remove OUT → HS_CTRL_COMPLETE_REPLACE. Leader/control-plane driven; does not need a CRAFT client. Listener gets on_start_replace_member / on_complete_replace_member / on_remove_member. nuraft supports learners natively via flip_learner_flag.
  • Baseline resync / snapshot for the backlog (≤ startLSN): nuraft triggers it past resync_log_idx_threshold, drives chunking/transport/truncation; the listener provides the payload via read_snapshot_obj/write_snapshot_obj.
  • The CP (checkpoint) subsystem is the consistency primitive: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary, so a CP is an atomic point-in-time of index+blocks. CRAFT sources its snapshot payload by forcing a CP flush at the snapshot LSN (cp_mgr().trigger_cp_flush), pinning that CP, and serializing the index + referenced blocks. homestore's resync metadata carries only a DSN marker; the listener owns the index+block payload.

Catch-up is pull-based (so the client never broadcasts to a catching-up member): the backlog comes via the snapshot path; the tail + small gaps come via the existing SyncRSCommitLSN → FetchData loop above. "Advance startLSN when the missing list hits its upper bound" mirrors homestore's resync_log_idx_threshold fall-back-to-baseline.

Promotion (learner → voter) is gated by CRAFT's commit_lsn ≥ startLSN, not homestore's rLSN-based monitor (which is blind to CRAFT data, since data isn't in the log). startLSN = the dLSN at the moment of the change; the member reports commit_lsn in its keepalive response. This gate is also the data-plane quorum-safety mechanism: a member must hold data through startLSN before it can count as a poll-quorum participant, else a later login could poll a quorum that misses acked writes.

Snapshot callbacks (modeled on homeobject's ReplicationStateMachine): the closest existing template. Mapping: homeobject PG → CRAFT volume; its shard+blob hierarchy → CRAFT's flat LBA index + block data (one level, simpler). Transfer is a resumable, objId-cursored message stream (volume metadata → index+data batches → LAST_OBJ_ID), with a leader-side iterator in snp_obj->user_ctx, a follower-side receive handler that persists progress to a superblock for mid-resync crash recovery, async block prefetch with bounded inflight, and per-message CRC. Inherited gotchas: consistent-LSN capture (a concurrent higher commit can overwrite an LBA mid-stream → verify/re-read), never block in the callbacks, keep the member not-readable until the last object (this is the startLSN gate).

Self-elected stand-in client (recovery while detached). homestore's replace_member runs without a client, but CRAFT data-plane convergence (proposing SyncRSCommitLSN, driving the promotion gate) is normally client-driven. When detached, the RAFT leader self-elects to drive just that. No app IO when detached ⇒ no live tail ⇒ pure backlog catch-up. It needs a committed logout / lease-expiry record to know it is safe to act, and yields on the next committed InternalLogin (free via the term mechanism). Leader-only ⇒ no stand-in split-brain.

Net CRAFT-owned build list (small; everything else is homestore):

  1. The four snapshot callbacks shipping CRAFT's committed index + blocks.
  2. The commit_lsn ≥ startLSN promotion gate (keepalive response carries commit_lsn), interposed on complete_replace_member.
  3. The committed logout / lease record + leader self-election.
  4. Term-bump / re-login on HS_CTRL_COMPLETE_REPLACE so an attached client refreshes its member set.

Correctness deep-dives

1. The recovery watermark max(quorum.last_append) is forced

Login computes rs_commit_lsn = max over a polled quorum of last_append_lsn. It can look like over-committing (promoting an LSN only one node holds), but under block semantics it is the only correct choice; there is no decision to make.

Indistinguishability. N=3, quorum=2. An LSN k shows up on exactly one node of the polled quorum. Two scenarios are observationally identical from {S2,S3}:

  • Case A: k reached S1+S2 (true quorum, durable, client acked the app), then S1 went unavailable. Poll {S2,S3}: only S2 has k.
  • Case B: k reached only S2, never made quorum, client never got its ack. Poll {S2,S3}: only S2 has k.

To never drop the durable one (A) you must include k, which means you sometimes also include the non-durable one (B).

Cost asymmetry makes "include" the safe direction. false-exclude = drop an LSN the app was told was durable → silent committed-data loss (catastrophic). false-include = land an in-flight write whose absence was never promised to anyone → benign (the filesystem already handles crash consistency for in-flight writes; the new term fences the old client out).

The theorem to write into the design. If k was ever quorum-acked, ≥2 nodes hold it; any polled quorum of 2 intersects that set, so at least one polled node reports last_append ≥ k, hence max(quorum.last_append) ≥ k. The real constraint this names: write-quorum ∩ poll-quorum ≠ ∅, always, automatic on fixed membership; the place it bites is reconfiguration (hence single-member-at-a-time).

2. Read eligibility: LBA-overlap, not a contiguous gate

Reads are unicast to one client-chosen replica, and readability is per-LBA, not per-contiguous-LSN. A replica may serve read([lba, lba+len), H) iff it holds every write ≤ H overlapping the range; equivalently, no Missing slot ≤ H overlaps [lba, lba+len). A hole on an unrelated LBA is irrelevant. Gating reads on a contiguous Synced prefix would be wrong twice over: it linearizes independent in-flight I/O (holding a higher LSN's readability for an unrelated lower one) and it lets one slow or lost write stall all higher reads, which is poison for a block device.

The client enforces it. It knows the LBA range of every write it issued and keeps a per-replica, LBA-aware Missing map, so it only routes a read to a replica that can serve it. Hence the read path never fetches. Peer fetch is resync-only.

The one place a contiguous watermark is right: the login dLSN L. At/below L are prior-session writes whose LBAs the client cannot name, so it can't run the overlap check there: a member must be fully filled to L (Synced ≥ L) before it serves any read. Above L (this session's writes) LBA-overlap governs. So the true rule is a two-region split at L, and Synced ≥ L is the same "reach the watermark" gate as reconfig promotion (commit_lsn ≥ startLSN).

Consequence: login can't finish until ≥1 member is filled to L (the leader syncs itself whole), and that member is the sole reader until each peer reaches Synced ≥ L, a window kept short by there usually being few in-flight I/Os, and by the leader-stand-in having pre-driven resync while detached.

Decisions already made

(So delegates don't relitigate.)

  • Recovery watermark = max(quorum.last_append): forced. Delete the commit_lsn "conservative" option from the protocol doc.
  • Consistency target = single-writer block semantics (causal read-after-write + read stability + convergence), not linearizability/transactions.
  • Read eligibility: client-driven, per-LBA-overlap. Each read carries a horizon H (client's highest quorum-acked LSN); a replica may serve iff filled to the login dLSN L (Synced ≥ L) and it has no Missing slot ≤ H overlapping the range. The read path never peer-fetches (fetch is resync-only); post-login the leader is the sole reader until each peer reaches Synced ≥ L.
  • Durability: FUA write-through; no FLUSH; homestore logstore already provides it; device power-safety guaranteed by deployment.
  • dLSN = deterministic merge key (highest-per-LBA wins everywhere); no overlapping-write serialization.
  • Reconfiguration: single-member-at-a-time; lean on homestore replace_member/learner/snapshot/CP; CRAFT owns 4 items; catch-up is pull-based; promotion gated by commit_lsn ≥ startLSN; detached recovery via a leader self-elected stand-in client.

Open items

  • logout/lease record: explicit RAFT entry vs. a lease the leader evaluates locally; who proposes it (keepalive watchdog).
  • Snapshot frozen-view mechanism: CP-pinned read vs. homeobject-style verify/retry while writes continue (an S10 detail).
  • Real transport (IP/TCP): planned (CRAFT-1 / SDSTOR-22297). The in-process multi-HomeBlkDisk + direct-API-call path is a development scaffold to build and validate the protocol in parallel with transport selection, not the end state. CraftConnector stays transport-agnostic so the chosen transport drops in behind it.

Implementation work breakdown

Epic SDSTOR-22382. Full per-story acceptance criteria live in the repo at docs/craft/subtasks.md; this is the shape.

Story Scope Depends on
S1 CraftReplDev foundation: class, per-partition state, journal backend, multi-member RAFT group, listener skeleton (landed) none
S2 Write path (write: term check, journal append, last_append update, out-of-order tolerant, zero-copy) S1
S3 Commit + Read path (commit/keep_alive materialize to index; read carries horizon H, client-routed by LBA-overlap, served CommitAndRead from the index, no read-path fetch; watchdog) S2
S4 Truncate (drop journal entries above an LSN; clear Missing above it) S1
S5 RAFT entries: SyncRSCommitLSN + InternalLogin apply; periodic checkpoint S1, S6
S6 Peer data exchange: get_lsns/get_rs_commit_lsn, fetch_data S1
S7 Login orchestration (leader-side; watermark = max(quorum.last_append)) S2, S4, S5, S6
S8 CRAFT volume lifecycle: create multi-member group, recover state from journal/superblock S1
S9 CraftConnector frontend (transport-agnostic; in-process scaffold first) S2, S3, S7
S10 (new) Reconfiguration: replace_member integration + 4 snapshot callbacks (CP-sourced payload, homeobject-modeled) + commit_lsn ≥ startLSN promotion gate + logout/lease record + leader self-election + re-login on complete S5, S6, S8

Development scaffold: all of this is implementable and testable now with multiple in-process HomeBlkDisk instances making direct API calls (catch-up is just inter-instance calls) in parallel with real-transport selection.

HomeBlocks

CRAFT

Clone this wiki locally