CRAFT Design

CRAFT: Client Assisted RAFT

Scope. This page is the HomeBlocks (server) side of CRAFT. The remote client/initiator side is a separate epic; it is summarized here only at its API boundary. Audience: engineers picking up the CRAFT server work who don't yet have the homeblocks internals in their head.

Contents

• What CRAFT is
• Glossary
• Background: Raft, Paxos, PacificA, CORFU
• The contract CRAFT owes
• Why (motivation)
• Where CRAFT plugs into HomeBlocks
• Durability model: FUA write-through
• Login
• Write, Commit, Read, and readLSN protection
• Resync / catch-up: the FetchData loop
• Reconfiguration (membership change)
• Correctness deep-dives
• Decisions already made
• Open items
• Implementation work breakdown

---

What CRAFT is

CRAFT (Client Assisted RAFT) is the replication protocol for HomeBlocks block volumes. Its one defining move is to split the data path from the consensus path:

• Data path (hot, leaderless). The single client that owns a volume assigns each write a monotonic LSN and broadcasts it directly to all replicas in parallel. A write is durable once a quorum acks the journal append. The RAFT leader plays no role here.
• Consensus path (cold, RAFT). RAFT is used for only three things: leader election, establishing the single-writer session (InternalLogin → term + client_token), and agreeing recovery watermarks (SyncRSCommitLSN). Write data never enters the RAFT log, only LSN watermarks and login tokens do.

A second structural choice runs through everything: durability and readability are decoupled. Appended (quorum journaled the data, durable) ≠ Committed (applied to the LBA index, readable). The client drives commit lazily, via commit, keep_alive, or piggybacked on the next write/read. Commit advances eagerly and independently per replica; it is not a synchronous quorum step.

---

Glossary

Sequence numbers & watermarks

Term	Meaning
LSN	Log Sequence Number.
dLSN	Data LSN: the per-partition number the client assigns to each write (its "position"). Dense (contiguous) within a partition. This is the only LSN CRAFT itself uses.
rLSN	RAFT log index. Distinct from dLSN (data is not in the RAFT log).
last_append_lsn	Highest LSN a replica has appended (present in its journal).
commit_lsn (≡ Synced)	The contiguous committed prefix on a replica (Empty slots skipped): everything ≤ it is present and applied. The fill watermark.
rs_commit_lsn	The replica-set commit LSN chosen at login = max(quorum.last_append).
L (login dLSN)	The dLSN login returns (= rs_commit_lsn). Boundary below which the client cannot name a write's LBA (prior-session data).
H (readLSN / horizon)	The client's contiguous quorum-acked prefix, stamped on each read; the read reflects writes ≤ H.
startLSN	The dLSN at the moment of a member change; the promotion watermark for the joining member.

Slot / write states (a slot is one dLSN on one replica)

State	Meaning
Appended	Data is in the journal and durable at quorum, but not yet readable.
Committed	Applied to the LBA index; readable.
Missing	A write known to exist but not held here; must be fetched from a peer (or declared Empty).
Empty	A write that reached no replica and can't be fetched; a permanent no-op hole that commit skips.

Consensus & session

Term	Meaning
RAFT / Raft	Consensus protocol (leader election + replicated log). CRAFT uses it only on the cold path.
Paxos	The older consensus family Raft descends from.
term	RAFT term; bumped on each client login and carried on every IO to fence stale clients.
client_token	Identifies the single writer; set by InternalLogin.
quorum	A majority of the replica set (2 of 3).
f	Number of tolerated failures (f = 1 for a 3-node set).
RS / Replica Set	The nodes holding one partition's copies (typically 3).
partition	The independent replica-set CRAFT replicates. A volume comprises one or more partitions; a given LBA maps to exactly one partition, so overlapping writes always land in the same partition (which is why the per-partition dLSN is a sufficient merge key).
volume	Comprises one or more partitions. CRAFT operates per-partition (dLSN).

RPCs, RAFT entries & rules

Term	Meaning
SyncRSCommitLSN	RAFT entry that advances the replica-set commit watermark; the primary resync trigger.
InternalLogin	RAFT entry that sets client_token + term (single-writer enforcement).
GetRSCommitLSN	Non-RAFT peer query of a replica's {commit_lsn, last_append_lsn}.
FetchData	Non-RAFT peer pull of missing journal slots (the resync data path).
CommitAndRead	A read that commits (materializes) the touched appended entry into the index, then serves.
merge key	The convergence rule: highest dLSN per LBA wins, on every replica.

Components & systems

Term	Meaning
HomeBlocks	The block-volume store (the server side this page covers).
HomeStore	The underlying storage engine (journal / index / blkdata / RAFT / CP).
homeobject	A blob store on HomeStore; the template CRAFT follows for snapshot + replace_member.
CraftReplDev	New per-volume CRAFT class (parallel to HomeStore's repl_dev).
CraftConnector	Client-facing RPC frontend that translates to CraftReplDev calls.
CraftRaftListener	repl_dev_listener handling the two CRAFT RAFT entries + snapshot / replace_member callbacks.
CraftJournalBackend	Mockable journal abstraction wrapping the HomeStore logstore.
RaftReplDev / ReplDisk	HomeStore's replication device classes.
replace_member / learner / flip_learner_flag	HomeStore membership-change primitives CRAFT leans on.
ublk / HomeBlkDisk	The in-process test adapter used as the development scaffold.
PacificA	Primary-backup replication framework (consensus only for config); a CRAFT touchstone.
CORFU	Shared-log storage architecture (clients write directly to storage units); a CRAFT touchstone.

Storage & durability

Term	Meaning
LBA	Logical Block Address.
FUA	Force Unit Access: every write is durable on completion (write-through; no writeback cache, no FLUSH).
CP	HomeStore Checkpoint: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary (a consistency point).
DSN	Data Sequence Number: a HomeStore marker carried in snapshot-resync metadata.
logstore	HomeStore's write-through journal (backs CraftJournalBackend).

---

Background: Raft, Paxos, PacificA, CORFU, and CRAFT

CRAFT borrows one idea each from a few well-known systems. If these names are unfamiliar this is the quick map; if they're familiar, it pins down exactly which idea CRAFT takes from each.

Raft & Paxos: consensus protocols. Both solve agreement among distributed nodes despite failures: how a set of replicas commit to the same ordered log of operations. Paxos is the older, foundational family; Raft was designed to be easier to understand and implement while giving the same replicated-log guarantees (an elected leader replicates an ordered log to followers). They answer "how do replicas agree?"

PacificA: primary-backup replication for storage. A framework for keeping replicas of a log-based storage shard consistent. A primary orders the writes and forwards them to backups; correctness across failures is delegated to a separate, consensus-backed configuration manager that owns membership. The key idea: keep consensus off the per-write path and pay for it only on rare reconfiguration. It answers "how do I replicate a storage shard with a primary and backups?"

CORFU: a shared-log architecture. Turns a cluster of flash devices into one global append-only log shared by many clients. A sequencer hands out log positions; clients then write directly to the storage units at those positions (no primary in the data path); a separate consensus-backed layer handles reconfiguration and hole-filling. It answers "how do I expose one global durable log across many devices, with clients writing directly?"

How they layer. Raft/Paxos are general-purpose "get a consistent log." PacificA is a specific primary-backup replication scheme with consensus only for config. CORFU is a shared-log abstraction others build on, where clients write straight to storage.

System	The question it answers
Paxos / Raft	How do replicas agree on an ordered log?
PacificA	How do I replicate a storage shard with a primary + backups, keeping consensus off the write path?
CORFU	How do I expose one global durable log across many devices, with clients writing directly?
CRAFT	How do I replicate a single-attach block volume when the client itself can order the writes?

How CRAFT fits. CRAFT takes one idea from each and specializes it for single-attach block volumes:

• From PacificA: keep consensus off the write path; use a separate consensus layer only for configuration-class decisions. CRAFT's consensus layer is Raft, and it touches only the cold path: leader election, InternalLogin (who is the single writer + term), SyncRSCommitLSN (recovery watermark), and membership. Raft here plays exactly PacificA's "configuration manager" role. The twist: PacificA still has a primary node that orders and forwards writes; CRAFT has no data-path primary at all.
• From CORFU: the writer assigns positions (LSNs) and writes directly to every storage replica, bypassing any leader/primary on the hot path. The difference: CORFU runs a separate sequencer serving many concurrent clients over a shared log; CRAFT relies on block volumes being single-attach, so the one client per volume is its own sequencer: no sequencer service and no per-write coordination.

In one line: CRAFT = CORFU-style direct-to-replica writes by a client that is its own sequencer, with Raft as a PacificA-style configuration manager that never touches the data path. That combination is only safe because a block volume has a single writer (see The contract CRAFT owes).

The lineage in one picture: consensus (blue) stays off the data path in all three, the data path (orange) gets more direct, and the sequencer (yellow) moves from an external service in CORFU into the single client itself in CRAFT.

---

The contract CRAFT owes (block-device semantics)

CRAFT is a single-writer block device, not a transactional or linearizable store. Holding it to the right (weaker) contract is what makes the design tractable. It owes exactly:

• Durability: a write that was acked survives crash, up to f failures. All homeblocks I/O is FUA (see Durability model): there is no FLUSH op and no writeback cache, so "acked" already means "durable." Un-acked (in-flight) writes have undefined outcome on crash; the filesystem/application owns crash consistency for them. We do not treat in-flight writes as transactions.
• Read-after-write: only for causally ordered ops (the app waited for the write's ack before issuing the read). Concurrent read+write to the same LBA is a race with an undefined result, by the block contract.
• Read stability: once a newer value of an LBA has been observed, a later read does not regress to an older one.
• Replica convergence: all replicas eventually agree on the same value per LBA.

Not owed: cross-LBA ordering, atomicity of concurrent overlapping writes, transactions.

Consequence: there is no order among in-flight I/O. All submitted I/O races naturally; the application serializes (waits for ack) when it needs ordering. So the dLSN is not preserving application order (there is none for concurrent I/O). Its job is to be a deterministic merge key: highest dLSN to an LBA wins, on every replica, so replicas converge to the same value even when two writes race. The client assigns dLSNs in submission order; "highest-LSN-per-LBA wins everywhere" is the whole rule. No overlapping-write serialization logic is needed.

---

Why (motivation)

1. Kill the leader bottleneck. Classic RAFT funnels every byte through the leader (client → leader → log → fan-out → apply). The leader's egress is N× the write bandwidth and it double-writes (log, then state machine). CRAFT pushes the fan-out onto the client and writes data to the journal once, a throughput and write-amplification win.
2. Lower write latency. Client → replica broadcast is one hop to durability (quorum append) vs. two through a leader.
3. The single-writer constraint is essentially free here. Block volumes are single-attach (EBS-like), so there is naturally one writer per volume. Given one writer, that writer is the total-order authority. You don't need consensus to order writes. That is the load-bearing insight that makes a leaderless, consensus-free data path safe. RAFT is then only needed for the genuinely ambiguous things: who is the one writer (split-brain) and what is durable after a crash (recovery watermark).
4. Keep RAFT small and off the hot path. Login and resync are low-frequency, so the RAFT log stays tiny and never carries data.
5. No homestore changes. Built entirely on existing homestore journal/index/ blkdata primitives, so the blast radius stays in the HomeBlocks layer.

---

Where CRAFT plugs into HomeBlocks

Today (solo, data-through-log):

home_blocks API (create_volume / async_read / async_write)
  -> HomeBlocksImpl / volume_mgr
  -> Volume   (owns a homestore::repl_dev created SOLO, empty member list)
              IO via rd()->async_write / async_read / async_write_journal
  -> HBListener (repl_dev_listener) handles RAFT callbacks
  -> homestore (RaftReplDev data-through-log, journal, index, blkdata)

With CRAFT (leaderless data path):

Client (initiator) --broadcast writes--> every replica's CraftConnector
  CraftConnector            (client-facing frontend; transport-agnostic)
    -> CraftReplDev         (one per volume; multi-member RAFT group)
       - write / read / commit / keep_alive / login / ...  (CRAFT API)
       - CraftJournalBackend  -> homestore logstore journal (write-through / FUA)
       - LBA index + blkdata for reads (CommitAndRead materializes on demand)
       - CraftRaftListener: RAFT only for SyncRSCommitLSN + InternalLogin
         (+ snapshot / replace_member callbacks for reconfiguration)
    -> homestore (RaftReplDev for consensus only; journal / index / blkdata / CP / snapshot)

Component map (homeblocks-concrete):

Component	New/changed	Location	Responsibility
CraftReplDev	new	src/lib/craft/craft_repl_dev.*	The CRAFT API + per-partition state {commit_lsn, last_append_lsn, client_token, term}
CraftJournalBackend	new (mockable)	src/lib/craft/craft_repl_dev.*	Journal slot write/read/truncate; production impl wraps the homestore logstore
CraftRaftListener	new	src/lib/craft/craft_repl_dev.*	on_commit dispatch for the two RAFT entry types; snapshot + replace_member callbacks
CraftConnector	new	src/lib/craft/ (TBD)	Translate client RPCs ↔ CraftReplDev calls; leader redirect; term checks; transport-agnostic
Volume wiring	changed	src/lib/volume/volume.cpp	Create a multi-member CraftReplDev for CRAFT-mode volumes; route IO
HomeBlocksImpl wiring	changed	src/lib/homeblks_impl.cpp	create_repl_dev_listener returns a CraftRaftListener; volume lifecycle/recovery
volume_info.replication_mode	exists	src/include/homeblks/home_blocks.hpp	DISABLED (today's solo path) vs CRAFT

Note. There is no ScstConnector in the current tree. The de-facto frontend today is the public async_read/async_write API plus the ublk adapter (src/test/homeblk_disk). "Add CraftConnector" = add the client→CraftReplDev translation layer.

Journal backing. CraftJournalBackend opens the logstore in out-of-order mode (not the append-only mode RAFT uses): dLSN maps 1:1 to logstore seq_num via write_async(seq_num) (out-of-order, no gap buffer), read_sync(seq_num) for random slot reads, fill_gap(seq_num) to mark an Empty slot (so the contiguous watermark advances past it), and truncate(seq_num) at login. The journal is bounded: the logstore reclaims below a checkpointed point, with CRAFT's floor = min(checkpointed commit_lsn, resync-retention window); a peer past that window is caught up via the snapshot path, not FetchData.

---

Durability model: FUA write-through

All homeblocks I/O is FUA. There is no FLUSH operation and no writeback cache to flush or barrier. Every write must be durable-on-ack (write-through); otherwise a simultaneous restart of all replicas could lose acknowledged writes.

This maps directly onto homestore's logstore, which is already write-through (verified against homestore dev/v8.x):

• append_async only buffers the record (log_dev.cpp:288, flush_if_necessary at :297). Flushing is group-commit (size threshold or timer).
• LogDev::flush() issues a synchronous device write (m_vdev_jd->sync_pwritev(...), log_dev.cpp:532) and calls on_flush_completion() only after it returns (:539).
• on_flush_completion (log_dev.cpp:545) raises each append's completion callback (on_write_completion at :583).

So the awaited append completion fires only after the bytes are synchronously written to the journal device, never on buffering. Group-commit just batches concurrent appends into one sync write and acks them together (ideal for CRAFT's broadcast-and-await-quorum pattern).

Requirement on CRAFT: CraftReplDev::write must await the logstore append completion before acking the client. Then quorum-ack ⇒ quorum-durable ⇒ an all-replicas restart cannot lose an acked write.

Device-level power-safety: the journal device is opened O_DIRECT (device_manager.cpp:47) and homestore skips fsync in direct-IO mode. O_DIRECT

• sync write lands data on the controller; power-safety relies on the drive being FUA / write-cache-off / power-loss-protected. This is settled operationally: homeblocks already runs the journal this way in production.

---

Login

Login establishes a new term, converges replica state to a starting dLSN, and enforces single-writer exclusivity, all before any IO is accepted. It is leader-only; a follower replies ENOTLEADER plus the leader endpoint.

sequenceDiagram
    autonumber
    participant C as Client (initiator)
    participant L as Leader (S1)
    participant S2 as Replica S2
    participant S3 as Replica S3

    C->>L: login(client_token, vol_id)
    Note over L: Leader-only. Follower replies<br/>ENOTLEADER + leader endpoint.

    Note over L,S3: Phase 1: collect replica LSN state (non-RAFT broadcast)
    L->>S2: GetRSCommitLSN(term, my_commit, my_append)
    L->>S3: GetRSCommitLSN(term, my_commit, my_append)
    S2-->>L: {commit=5, last_append=7}
    S3-->>L: {commit=10, last_append=11}
    Note over L: rs_commit_lsn = max(quorum.last_append) = 11

    opt Leader behind (self.last_append < 11)
    L->>S3: FetchData(missing slots up to 11)
    S3-->>L: JournalSlot[] (lba,len,data)
    Note over L: append to own journal
    end

    Note over L,S3: Phase 2: SyncRSCommitLSN(11) via RAFT (data NOT in log)
    L->>S2: RAFT replicate SyncRSCommitLSN(11, token)
    L->>S3: RAFT replicate SyncRSCommitLSN(11, token)
    Note over S2,S3: on apply, if behind FetchData up to 11.<br/>commit_lsn = 11. (no truncation here, login-only step below)

    Note over L,S3: Phase 3: InternalLogin(token, term+1) via RAFT
    L->>S2: RAFT replicate InternalLogin(token, T+1)
    L->>S3: RAFT replicate InternalLogin(token, T+1)
    Note over S2,S3: store client_token, term=T+1.<br/>reject IO with any other term

    Note over L,S3: Phase 4: truncate stale tail (login-only, after InternalLogin commits)
    L->>S2: truncate(11) if last_append > 11
    L->>S3: truncate(11) if last_append > 11
    Note over S2,S3: drop appended-but-not-quorum entries above rs_commit_lsn

    L-->>C: LoginResult{members, dLSN=11, term=T+1}
    Note over C: open a queue per member, begin IO at dLSN+1

Loading

The watermark rs_commit_lsn = max(quorum.last_append) is forced, not a tuning choice; see the recovery-watermark deep-dive. (dLSN is this partition's LSN, and the only LSN CRAFT uses. A volume comprises one or more such partitions.

Fencing the old session (quiesce barrier). A replica quiesces old-session writes the moment it receives the login poll: GetRSCommitLSN carries an is_login flag, and on it a replica stops acking data writes from any prior term, then reports last_append. This makes max(quorum.last_append) capture every write quorum-acked before the barrier (quorum intersection), so login truncation above rs_commit_lsn can never discard an acked write, and a would-be zombie writer is rejected (explicit error) rather than acked-then-silently-truncated. Watchdog/periodic GetRSCommitLSN polls carry is_login=false and never quiesce (they ride the live tail). The quiesce releases on either of: an explicit login-aborted signal the leader sends when it cannot reach quorum, or a replica-side quiesce timeout (a small multiple of the login round-trip) that fires if that signal is lost, so a failed or partitioned login can never wedge the current owner. Assumption: the attach layer (CSI) should also hard-fence the prior client on failover (reservation / network revoke); the quiesce barrier is CRAFT-side defense-in-depth.

Leader behind (Phase 1b) fallback. The leader fetches missing slots ≤ rs_commit_lsn from any peer holding each; a slot ≤ rs_commit_lsn absent from the whole responding quorum was never quorum-durable and is marked Empty. If a quorum cannot be reached, the login fails and the client retries (or RAFT elects a new leader).

---

Write, Commit, Read, and readLSN protection

Write. The client assigns ++next_lsn, broadcasts write(term, lsn, lba, len, data) to all replicas. Each validates term, appends the data to its journal at slot lsn (zero-copy, may arrive out of order), advances last_append_lsn, and acks, per write, since in-flight I/O is unordered. Durable at quorum-ack (Appended), but durable ≠ readable: a write becomes readable only once it is committed (materialized into the LBA index), which the client drives, or which a read of that entry triggers on demand (CommitAndRead, see Read). No LBA-index update happens on the write itself.

Commit. Commit makes a durable write readable by materializing it into the LBA index; the client drives it (lazily, via commit / keep_alive / piggyback), and a read can trigger it on demand (CommitAndRead). Because in-flight I/O is unordered, readability is per-write, not a contiguous prefix: a higher LSN can be readable while a lower one is still a hole, with overlaps resolved by the merge key (highest LSN per LBA wins).

The per-partition scalar commit_lsn (≡ Synced) is the contiguous committed prefix, the highest LSN with no unresolved hole below it (Empty slots are skipped). It is the fill watermark used by resync and reconfiguration, and the read gate in the ≤ L region (below); individual writes above it are made readable per-LBA on demand, which is why readability is not tied to it.

commit() is best-effort per replica: it advances commit_lsn to just below the first Missing hole (applying present entries, skipping Empty) and returns the achieved watermark; a stall at a hole is not an error (resync fills it). Recovery: the journal is FUA-durable, so a CommitAndRead'd entry survives a crash even if a CP had not yet flushed the index. On restart the replica rebuilds its appended-above-commit_lsn set from the journal and reads re-CommitAndRead, so no readable slot (≤ H, hence quorum-durable) is ever lost.

Read: the readLSN protection. Reads are unicast to one replica, and the client decides where it is safe to read: it knows the LBA range of every write it issued. Each read carries a horizon H (the readLSN) = the client's contiguous quorum-acked prefix: every LSN ≤ H is quorum-durable (not merely submitted), so a read never reflects a sub-quorum write that could vanish and a servable replica always exists. The replica returns the latest version ≤ H for the range: if that write is present but not yet committed, it commits it first (CommitAndRead, it already holds the data, so no fetch) and serves from the index; writes above H are ignored. A replica may serve read([lba, lba+len), H) iff:

1. it is filled up to the login dLSN L (Synced ≥ L; L = the dLSN login returned = rs_commit_lsn). At/below L are prior-session writes whose LBAs the client can't name, so it must hold all of them; and
2. above L, it has no Missing slot ≤ H overlapping [lba, lba+len); the client checks this against its LBA-aware per-replica Missing map.

So readability is per-LBA-overlap, not "no holes": a hole on an unrelated LBA does not block the read. Because the client only routes a read to a replica that can serve it, the read path never fetches from a peer. Inline fetch is resync-only. Consequence: login cannot complete until ≥1 member is filled to L (the leader fetches itself whole during login), and that member is the sole reader until each peer independently reaches Synced ≥ L.

sequenceDiagram
    autonumber
    participant C as Client (initiator)
    participant R1 as Replica (overlapping hole)
    participant R2 as Replica (eligible)

    Note over C: horizon H = client's contiguous quorum-acked prefix.<br/>Client knows every write's LBA and each replica's Missing map.
    Note over C,R1: R1 has a Missing slot <= H overlapping [lba, lba+len) -> INELIGIBLE, skip
    Note over C,R2: R2 filled to L (Synced >= L) and no overlapping Missing <= H -> eligible
    C->>R2: read(term, H, lba, len)
    Note over R2: CommitAndRead the latest <= H if not yet committed<br/>(it has the data, no fetch), then serve from index. ignore anything above H
    R2-->>C: data
    Note over C,R2: Client routes only to a servable replica, so the READ PATH NEVER FETCHES.<br/>Peer fetch is resync-only.

Loading

Worked example: three replicas, quorum-durable but each committed to a different extent (independent commit); a read of LBA 13 that the client routes around S1's hole, served by S2 with no read-path fetch.

Two regions, split at the login dLSN L. At/below L are prior-session writes the client can't reason about per-LBA, so a member needs a full contiguous fill (Synced ≥ L) before it serves any read. Above L are this session's writes, gated per read by LBA-overlap. Synced ≥ L is the same "reach the watermark" gate as reconfig promotion (commit_lsn ≥ startLSN).

The client's job: keep an accurate LBA-aware Missing map per replica. It removes entries on write-ACKs, and learns resync-fills from {commit_lsn, last_append_lsn} carried in each keep_alive / get_lsns response (pruning any Missing slot ≤ that replica's commit_lsn); staying conservatively stale when unsure (treat a slot as Missing, route elsewhere) is safe, at the cost of under-using a replica. After login the client initializes the horizon H := L (all data through L is quorum-durable), then advances H with its contiguous quorum-acked prefix.

---

Resync / catch-up: the FetchData loop

SyncRSCommitLSN is the primary recovery mechanism. It carries only an LSN watermark (no data). On apply, a replica that is behind pulls the missing slots from a peer, then advances its commit watermark.

sequenceDiagram
    autonumber
    participant L as Leader (S1)
    participant S3 as Lagging replica (S3)
    participant P as Ahead peer (S2)

    Note over L,P: Trigger: login / keepalive watchdog /<br/>periodic checkpoint (every N LSNs)
    L->>S3: RAFT replicate SyncRSCommitLSN(N, token)
    L->>P: RAFT replicate SyncRSCommitLSN(N, token)

    Note over S3: on apply: last_append < N<br/>=> missing slots {m1..mk} <= N
    loop until no missing slot <= N
        S3->>P: FetchData([missing lsns])
        P-->>S3: JournalSlot[] {lsn,lba,len,data | is_empty}
        alt slot present on peer
            Note over S3: append to journal.<br/>last_append = max(last_append, lsn).<br/>drop from missing set
        else slot on no peer
            Note over S3: mark slot Empty (permanent no-op hole).<br/>commit may advance past it
        end
    end
    Note over S3: commit_lsn = N (contiguous Synced<br/>watermark advances through N)

    Note over L,P: New/replacement member: bulk BACKLOG (<= startLSN) arrives first<br/>via the snapshot path (CP-sourced index+blocks). THIS loop fills the<br/>tail. Promote learner -> voter when commit_lsn >= startLSN.

Loading

Declaring Empty (precise rule). FetchData returns one of present+data, not-present-here, or known-Empty, per slot. The peer's obligation is exact: it returns is_empty=true only for a slot it has positively marked Empty in a prior resync cycle; for a slot merely absent from its journal it omits the entry from the response (that reads as not-present-here). So a responder never reports Empty for data it simply lacks, which is what keeps "quorum says Empty" from being spoofed by absence. A lagging replica broadcasts FetchData to its peers and accumulates responses; it marks a slot Empty only when a quorum report not-present / known-Empty (a slot ≤ rs_commit_lsn absent from a responding quorum was never quorum-durable, so Empty is safe). A single peer's not-present is not enough, and a partial response (a peer down) stalls those slots rather than declaring Empty. Commit advances past Empty slots but never past an unresolved Missing one.

Truncation is login-only. SyncRSCommitLSN apply (watchdog / periodic) never truncates: above-watermark appends are the live tail the current client is still writing. Truncation above rs_commit_lsn is a login-only action, performed by the leader after InternalLogin commits.

---

Reconfiguration (membership change)

Principle: lean on homestore for the mechanics; CRAFT owns only the data-plane gating that homestore (being RAFT-log-based) cannot see. Verified against homestore dev/v8.x.

Constraint: single-member-at-a-time (add-one / remove-one; compose for bigger moves). Preserves write-quorum ∩ poll-quorum ≠ ∅ by construction, no joint consensus.

Reduced-availability window is control-plane only. homestore flips OUT to a learner, commits HS_CTRL_START_REPLACE, then adds IN, so RAFT voting briefly drops to 2-of-2 for the span of one RAFT commit (not shrinkable below it; a voter failure in that window stalls the cold path until recovery). The data plane is unaffected: the client keeps writing 2-of-3 to the still-synced {A,B,C} (the outgoing member is a RAFT learner but still holds and journals data, and the client refreshes its member set only at re-login on completion), and the data path is leaderless, so a running session keeps flowing even if a voter fails in the window. This is the data/consensus split doing its job.

What homestore gives us (lean fully):

• replace_member: a complete 6-step state machine with a learner phase: flip OUT to learner → HS_CTRL_START_REPLACE → add IN as learner → monitor catch-up → remove OUT → HS_CTRL_COMPLETE_REPLACE. Leader/control-plane driven; does not need a CRAFT client. Listener gets on_start_replace_member / on_complete_replace_member / on_remove_member. nuraft supports learners natively via flip_learner_flag.
• Baseline resync / snapshot for the backlog (≤ startLSN): nuraft triggers it past resync_log_idx_threshold, drives chunking/transport/truncation; the listener provides the payload via read_snapshot_obj/write_snapshot_obj.
• The CP (checkpoint) subsystem is the consistency primitive: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary, so a CP is an atomic point-in-time of index+blocks. CRAFT sources its snapshot payload by forcing a CP flush at the snapshot LSN (cp_mgr().trigger_cp_flush), pinning that CP, and serializing the index + referenced blocks. homestore's resync metadata carries only a DSN marker; the listener owns the index+block payload.

Catch-up is pull-based (so the client never broadcasts to a catching-up member): the backlog comes via the snapshot path; the tail + small gaps come via the existing SyncRSCommitLSN → FetchData loop above. "Advance startLSN when the missing list hits its upper bound" mirrors homestore's resync_log_idx_threshold fall-back-to-baseline.

Promotion (learner → voter) is gated by CRAFT's commit_lsn ≥ startLSN, not homestore's rLSN-based monitor (which is blind to CRAFT data, since data isn't in the log). startLSN = the dLSN at the moment of the change; the member reports commit_lsn in its keepalive response. This gate is also the data-plane quorum-safety mechanism: a member must hold data through startLSN before it can count as a poll-quorum participant, else a later login could poll a quorum that misses acked writes.

Snapshot callbacks (modeled on homeobject's ReplicationStateMachine): the closest existing template. Mapping: homeobject PG → CRAFT volume; its shard+blob hierarchy → CRAFT's flat LBA index + block data (one level, simpler). Transfer is a resumable, objId-cursored message stream (volume metadata → index+data batches → LAST_OBJ_ID), with a leader-side iterator in snp_obj->user_ctx, a follower-side receive handler that persists progress to a superblock for mid-resync crash recovery, async block prefetch with bounded inflight, and per-message CRC. Inherited gotchas: consistent-LSN capture (a concurrent higher commit can overwrite an LBA mid-stream → verify/re-read), never block in the callbacks, keep the member not-readable until the last object (this is the startLSN gate).

Frozen view (decided): CP-pinned and bounded. On snapshot need, trigger_cp_flush, pin that CP, and serialize the committed index+blocks as of it. Because the payload is read from a pinned CP (a point-in-time freeze), writes that land after the CP cannot mutate it, so no verify-on-read is needed for what the snapshot ships. The snapshot's startLSN = commit_lsn at the pinned CP, stored in the snapshot metadata so the tail FetchData loop resumes correctly. Bound the pin (time / size); a transfer that exceeds the bound releases the CP, re-pins a fresher one, and restarts with the new startLSN. Restartable: the follower checkpoints its receive cursor + startLSN to its superblock; a leader failure mid-transfer restarts the snapshot, and the follower resumes from its cursor or discards and restarts cleanly.

Self-elected stand-in client (recovery while detached). homestore's replace_member runs without a client, but CRAFT data-plane convergence (proposing SyncRSCommitLSN, driving the promotion gate) is normally client-driven. When detached, the RAFT leader self-elects to drive just that. No app IO when detached means no live tail, so it is pure backlog catch-up. Leader-only, so no stand-in split-brain.

Logout / lease (decided): hybrid. A locally-evaluated lease (the keepalive watchdog) detects that the client is gone; a committed RAFT InternalLogout / lease-expiry entry grants the authority to act. The leader self-elects only after that entry commits (serialized, survives leader failover, and avoids a new leader re-deciding blindly), and yields on the next committed InternalLogin. The committed logout also marks "no active client", which sharpens the login fencing boundary (a following login is unambiguously a fresh session).

Net CRAFT-owned build list (small; everything else is homestore):

1. The four snapshot callbacks shipping CRAFT's committed index + blocks.
2. The commit_lsn ≥ startLSN promotion gate (keepalive response carries commit_lsn), interposed on complete_replace_member.
3. The committed logout / lease record + leader self-election.
4. Term-bump / re-login on HS_CTRL_COMPLETE_REPLACE so an attached client refreshes its member set. Re-login is triggered by ETERM: the client re-logins to the leader presenting its client_token; the leader accepts only if that token is the current owner (returns fresh term + members) and rejects a superseded token, so a deposed client cannot reclaim the volume. In-flight writes are re-driven under the new term.

---

Correctness deep-dives

1. The recovery watermark max(quorum.last_append) is forced

Login computes rs_commit_lsn = max over a polled quorum of last_append_lsn. It can look like over-committing (promoting an LSN only one node holds), but under block semantics it is the only correct choice; there is no decision to make.

Indistinguishability. N=3, quorum=2. An LSN k shows up on exactly one node of the polled quorum. Two scenarios are observationally identical from {S2,S3}:

• Case A: k reached S1+S2 (true quorum, durable, client acked the app), then S1 went unavailable. Poll {S2,S3}: only S2 has k.
• Case B: k reached only S2, never made quorum, client never got its ack. Poll {S2,S3}: only S2 has k.

To never drop the durable one (A) you must include k, which means you sometimes also include the non-durable one (B).

Cost asymmetry makes "include" the safe direction. false-exclude = drop an LSN the app was told was durable → silent committed-data loss (catastrophic). false-include = land an in-flight write whose absence was never promised to anyone → benign (the filesystem already handles crash consistency for in-flight writes; the new term fences the old client out).

The theorem to write into the design. If k was ever quorum-acked, ≥2 nodes hold it; any polled quorum of 2 intersects that set, so at least one polled node reports last_append ≥ k, hence max(quorum.last_append) ≥ k. The real constraint this names: write-quorum ∩ poll-quorum ≠ ∅, always, automatic on fixed membership; the place it bites is reconfiguration (hence single-member-at-a-time).

2. Read eligibility: LBA-overlap, not a contiguous gate

Reads are unicast to one client-chosen replica, and readability is per-LBA, not per-contiguous-LSN. A replica may serve read([lba, lba+len), H) iff it holds every write ≤ H overlapping the range; equivalently, no Missing slot ≤ H overlaps [lba, lba+len). A hole on an unrelated LBA is irrelevant. Gating reads on a contiguous Synced prefix would be wrong twice over: it linearizes independent in-flight I/O (holding a higher LSN's readability for an unrelated lower one) and it lets one slow or lost write stall all higher reads, which is poison for a block device.

The client enforces it. It knows the LBA range of every write it issued and keeps a per-replica, LBA-aware Missing map, so it only routes a read to a replica that can serve it. Hence the read path never fetches. Peer fetch is resync-only.

Mental model (from the RAID1 it replaces). This per-replica Missing map acts as a multi-dimensional dirty-bitmap. A RAID1 mirror keeps a one-dimensional dirty-region log so a stale mirror resyncs only the dirty regions instead of the whole disk; CRAFT keys the same idea by (replica, LBA range, dLSN). Reads route around exactly the stale ranges of each replica and resync fetches exactly those, so a partially-behind replica still serves every range it is clean on.

Crucially, unlike a RAID1 bitmap the dirty state is not opaque to the replicas. A RAID1 dirty-region log lives in the controller, so the mirror members are dumb and the controller must drive every resync. In CRAFT the same information is materialized in each replica's own slot state (Missing / Empty / the commit_lsn watermark), so replicas resync from each other via SyncRSCommitLSN + FetchData with no client in the loop. That moves resync traffic off the client and is why catch-up is server-side, not client-driven.

The one place a contiguous watermark is right: the login dLSN L. At/below L are prior-session writes whose LBAs the client cannot name, so it can't run the overlap check there: a member must be fully filled to L (Synced ≥ L) before it serves any read. Above L (this session's writes) LBA-overlap governs. So the true rule is a two-region split at L, and Synced ≥ L is the same "reach the watermark" gate as reconfig promotion (commit_lsn ≥ startLSN).

Consequence: login can't finish until ≥1 member is filled to L (the leader syncs itself whole), and that member is the sole reader until each peer reaches Synced ≥ L, a window kept short by there usually being few in-flight I/Os, and by the leader-stand-in having pre-driven resync while detached.

---

Decisions already made

(So delegates don't relitigate.)

• Recovery watermark = max(quorum.last_append): forced. Delete the commit_lsn "conservative" option from the protocol doc.
• Consistency target = single-writer block semantics (causal read-after-write + read stability + convergence), not linearizability/transactions.
• Read eligibility: client-driven, per-LBA-overlap. Each read carries a horizon H (client's contiguous quorum-acked prefix); a replica may serve iff filled to the login dLSN L (Synced ≥ L) and it has no Missing slot ≤ H overlapping the range. The read path never peer-fetches (fetch is resync-only); post-login the leader is the sole reader until each peer reaches Synced ≥ L.
• Durability: FUA write-through; no FLUSH; homestore logstore already provides it; device power-safety guaranteed by deployment.
• dLSN = deterministic merge key (highest-per-LBA wins everywhere); no overlapping-write serialization.
• Scope = CRAFT replicates a single partition. A volume comprises one or more partitions; dLSN is per-partition (dense) and does all the CRAFT work (merge key, watermarks, Missing/Empty, recovery).
• Reconfiguration: single-member-at-a-time; lean on homestore replace_member/learner/snapshot/CP; CRAFT owns 4 items; catch-up is pull-based; promotion gated by commit_lsn ≥ startLSN; detached recovery via a leader self-elected stand-in client.
• Session fencing (Option A): a login-scoped quiesce barrier (an is_login poll flag) fences the prior writer at the login poll, turning a would-be silent ack-then-truncate into an explicit reject; the attach layer is also expected to hard-fence on failover (defense in depth). See Login.

---

Open items

(The logout/lease record and the snapshot frozen-view mechanism are now decided; see Reconfiguration.)

• Real transport (IP/TCP): planned (CRAFT-1 / SDSTOR-22297). The in-process multi-HomeBlkDisk + direct-API-call path is a development scaffold to build and validate the protocol in parallel with transport selection, not the end state. CraftConnector stays transport-agnostic so the chosen transport drops in behind it.

---

Implementation work breakdown

Epic SDSTOR-22382. Full per-story acceptance criteria live in the repo at docs/craft/subtasks.md; this is the shape.

Story	Scope	Depends on
S1	CraftReplDev foundation: class, per-partition state, journal backend, multi-member RAFT group, listener skeleton (landed)	none
S2	Write path (write: term check, journal append, last_append update, out-of-order tolerant, zero-copy)	S1
S3	Commit + Read path (commit/keep_alive materialize to index; read carries horizon H, client-routed by LBA-overlap, served CommitAndRead from the index, no read-path fetch; watchdog)	S2
S4	Truncate (drop journal entries above an LSN; clear Missing above it)	S1
S5	RAFT entries: SyncRSCommitLSN + InternalLogin apply; periodic checkpoint	S1, S6
S6	Peer data exchange: get_lsns/get_rs_commit_lsn, fetch_data	S1
S7	Login orchestration (leader-side; watermark = max(quorum.last_append))	S2, S4, S5, S6
S8	CRAFT volume lifecycle: create multi-member group, recover state from journal/superblock	S1
S9	CraftConnector frontend (transport-agnostic; in-process scaffold first)	S2, S3, S7
S10 (new)	Reconfiguration: replace_member integration + 4 snapshot callbacks (CP-sourced payload, homeobject-modeled) + commit_lsn ≥ startLSN promotion gate + logout/lease record + leader self-election + re-login on complete	S5, S6, S8

Development scaffold: all of this is implementable and testable now with multiple in-process HomeBlkDisk instances making direct API calls (catch-up is just inter-instance calls) in parallel with real-transport selection.