Skip to content

CRAFT on HomeBlocks

Jump to bottom
Brian Szmyd edited this page Jul 6, 2026 · 3 revisions

CRAFT on HomeBlocks

Scope. This page is the HomeBlocks binding of CRAFT: how the store-agnostic CRAFT Design protocol maps onto HomeStore's journal, index, block-data, RAFT, checkpoint, and membership primitives. The protocol page is the source of truth for what CRAFT does and why it is correct; this page is how it is realized here, and the only place that names concrete classes and cites source files. Audience: engineers picking up the CRAFT server work who don't yet have the HomeBlocks internals in their head.

Contents

Component map

The protocol's abstract roles map onto these concrete pieces.

Term Meaning
HomeBlocks The block-volume store (the server side).
HomeStore The underlying storage engine (journal / index / blkdata / RAFT / CP). Provides everything the protocol calls "the underlying store."
homeobject A blob store on HomeStore; the template CRAFT follows for snapshot + replace_member.
CraftReplDev New per-partition CRAFT class (parallel to HomeStore's repl_dev); a volume owns one per partition. Holds the per-partition state {commit_lsn, last_append_lsn, client_token, term}.
CraftConnector Client-facing RPC frontend that translates to CraftReplDev calls; leader redirect; term checks; transport-agnostic.
CraftRaftListener repl_dev_listener handling the CRAFT RAFT entries (SyncRSCommitLSN, InternalLogin, InternalLogout) plus snapshot / replace_member callbacks.
CraftJournalBackend Mockable journal abstraction wrapping the HomeStore logstore.
RaftReplDev / ReplDisk HomeStore's replication device classes.
replace_member / learner / flip_learner_flag HomeStore membership-change primitives CRAFT leans on.
ublk / HomeBlkDisk The in-process test adapter used as the development scaffold.
CP HomeStore Checkpoint: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary (a consistency point), so a CP is an atomic point-in-time of index + blocks.
DSN Data Sequence Number: a HomeStore marker carried in snapshot-resync metadata.
logstore HomeStore's write-through journal (backs CraftJournalBackend).

The protocol-level terms (dLSN, commit_lsn, last_append_lsn, rs_commit_lsn, L, H, startLSN, the slot states, SyncRSCommitLSN / InternalLogin / InternalLogout, Missing / Empty, the merge key) are defined in the Design glossary.

Where CRAFT plugs into HomeBlocks

Today (solo, data-through-log):

home_blocks API (create_volume / async_read / async_write)
  -> HomeBlocksImpl / volume_mgr
  -> Volume   (owns a homestore::repl_dev created SOLO, empty member list)
              IO via rd()->async_write / async_read / async_write_journal
  -> HBListener (repl_dev_listener) handles RAFT callbacks
  -> homestore (RaftReplDev data-through-log, journal, index, blkdata)

With CRAFT (leaderless data path):

Client (initiator) --broadcast writes--> every replica's CraftConnector
  CraftConnector            (client-facing frontend; transport-agnostic)
    -> CraftReplDev         (one per partition; multi-member RAFT group)
       - write / read / commit / keep_alive / login / ...  (CRAFT API)
       - CraftJournalBackend  -> homestore logstore journal (write-through / FUA)
       - LBA index + blkdata for reads (journal-tail overlay serves unapplied entries)
       - CraftRaftListener: RAFT only for SyncRSCommitLSN + InternalLogin/Logout
         (+ snapshot / replace_member callbacks for reconfiguration)
    -> homestore (RaftReplDev for consensus only; journal / index / blkdata / CP / snapshot)

Component map (homeblocks-concrete):

Component New/changed Location Responsibility
CraftReplDev new src/lib/craft/craft_repl_dev.* The CRAFT API + per-partition state {commit_lsn, last_append_lsn, client_token, term}
CraftJournalBackend new (mockable) src/lib/craft/craft_repl_dev.* Journal slot write/read/truncate; production impl wraps the homestore logstore
CraftRaftListener new src/lib/craft/craft_repl_dev.* on_commit dispatch for the CRAFT RAFT entry types (SyncRSCommitLSN, InternalLogin, InternalLogout); snapshot + replace_member callbacks
CraftConnector new src/lib/craft/ (TBD) Translate client RPCs to CraftReplDev calls; leader redirect; term checks; transport-agnostic
Volume wiring changed src/lib/volume/volume.cpp Create a multi-member CraftReplDev for CRAFT-mode volumes; route IO
HomeBlocksImpl wiring changed src/lib/homeblks_impl.cpp create_repl_dev_listener returns a CraftRaftListener; volume lifecycle/recovery
volume_info.replication_mode exists src/include/homeblks/home_blocks.hpp DISABLED (today's solo path) vs CRAFT

Note. There is no ScstConnector in the current tree. The de-facto frontend today is the public async_read/async_write API plus the ublk adapter (src/test/homeblk_disk). "Add CraftConnector" = add the client to CraftReplDev translation layer.

No HomeStore changes. CRAFT is built entirely on existing HomeStore journal / index / blkdata primitives, so the blast radius stays in the HomeBlocks layer.

Journal backing and write-once-by-reference

CraftJournalBackend opens the logstore in out-of-order mode (not the append-only mode RAFT uses): dLSN maps 1:1 to logstore seq_num via write_async(seq_num) (out-of-order, no gap buffer), read_sync(seq_num) for random slot reads, fill_gap(seq_num) to mark an Empty slot (so the contiguous watermark advances past it), and truncate(seq_num) at login.

Write-once, by reference (verified against homestore). A CRAFT write does not put the payload in the journal: blocks are allocated up front, the payload is written once via the data service, and the journal slot records only the reference, {term, lsn, lba, len, blkid}. This is homestore's own replication pattern (HS_DATA_LINKED: solo_repl_dev.cpp allocates in rreq->init, writes via data_service().async_write, then journals a repl_journal_entry followed by the multi_blk_id; create_journal_entry's size formula is header + key + serialized blkids, with no payload term, so the payload never enters the journal buffer). HS_DATA_INLINED exists only for header-only entries, not small-payload inlining. The ack fires on journal-append completion, which starts only after the data write completes, so an acked write has both payload and reference on media; a crash between the two leaves only uncommitted blks, which homestore recovery reclaims. Apply is a metadata-only index insert (plus reclaim of superseded blocks); FetchData serves by reading the referenced blocks; truncation and Empty discard free them. The payload is written exactly once: the log-then-state-machine double write never happens.

Thin (all_zeros) writes carry no blocks. A zero write (the protocol's thin WRITE form) never touches the data service: it allocates nothing and its journal slot records {term, lsn, lba, len, all_zeros} with no blkid. It is a real ordered slot (it participates in the merge key by dLSN like any write), just a metadata-only one. On apply it is an index delete / update over the range (deallocating any blocks that earlier writes mapped there and reclaiming them), which leaves the range unmapped, that is, a hole that reads as zero. See Thin writes and holes on HomeStore.

Journal reclaim and the returning-voter snapshot trigger

The journal is bounded: the logstore reclaims below a checkpointed point, with CRAFT's floor = min(checkpointed commit_lsn, all_committed_lsn). The first bound is local crash-safety (the journal backs the overlay and index replay, so never reclaim above the checkpointed apply frontier). The second is the peers' FetchData needs, advanced two ways: exactly at a login (login Phase 2 converges the whole set to rs_commit_lsn), and opportunistically between logins via piggyback: the client sees every member's commit_lsn in each keep_alive / commit response, computes the set-wide min (all_committed_lsn, voting members only), and carries it on its next keep_alive / write; each member may then truncate below it (a member never re-fetches below its own reported commit_lsn, which is monotonic, so a stale min is merely conservative). A silent member freezes the min until the retention window relegates it to the snapshot path (baseline resync), which is what lets the journal stay bounded without keeping entries or membership static forever. Learners are excluded from the min (the snapshot path covers them); when detached, the stand-in leader computes and distributes the same value via its polls.

The snapshot trigger for a returning voter is the RAFT log itself. CRAFT compacts its (tiny) RAFT log below the SyncRSCommitLSN entry at the reclaim floor, so a member that missed more than the retention window is necessarily behind the compacted log, and nuraft's standard too-far-behind snapshot machinery fires with the same four listener callbacks reconfiguration uses. Data-behind alone would never trip nuraft (data is not in the log); tying compaction to the floor is what makes the existing machinery see it. After the snapshot, the tail fills via the normal FetchData loop; no promotion gate applies (the member is already a voter). This is the concrete binding of the protocol's bounded-Missing / degraded-member re-admission rule.

Durability: the logstore is already write-through

The protocol requires FUA write-through: the journal-append completion must fire only after the bytes are on stable media, never on buffering (see Durability model). HomeStore's logstore already satisfies this (verified against homestore dev/v8.x):

  • append_async only buffers the record (log_dev.cpp:288, flush_if_necessary at :297). Flushing is group-commit (size threshold or timer).
  • LogDev::flush() issues a synchronous device write (m_vdev_jd->sync_pwritev(...), log_dev.cpp:532) and calls on_flush_completion() only after it returns (:539).
  • on_flush_completion (log_dev.cpp:545) raises each append's completion callback (on_write_completion at :583).

So the awaited append completion fires only after the bytes are synchronously written to the journal device, never on buffering. Group-commit just batches concurrent appends into one sync write and acks them together (ideal for CRAFT's broadcast-and-await-quorum pattern).

Requirement on CRAFT: CraftReplDev::write must await the logstore append completion before acking the client. Then quorum-ack implies quorum-durable implies an all-replicas restart cannot lose an acked write.

Device-level power-safety: the journal device is opened O_DIRECT (device_manager.cpp:47) and homestore skips fsync in direct-IO mode. O_DIRECT + sync write lands data on the controller; power-safety relies on the drive being FUA / write-cache-off / power-loss-protected. This is settled operationally: homeblocks already runs the journal this way in production.

Reconfiguration mechanics

The protocol's Reconfiguration section owns the principles (single-member-at-a-time data-plane safety, the control-plane-only availability window, pull-based catch-up, the commit_lsn >= startLSN promotion gate, the healed-not-fenced live tail, the logout / lease hybrid, and the leader stand-in). This section is the HomeStore binding of those principles.

Principle: lean on homestore for the mechanics; CRAFT owns only the data-plane gating that homestore (being RAFT-log-based) cannot see. Verified against homestore dev/v8.x.

What homestore gives us (lean fully):

  • replace_member: a complete 6-step state machine with a learner phase: flip OUT to learner -> HS_CTRL_START_REPLACE -> add IN as learner -> monitor catch-up -> remove OUT -> HS_CTRL_COMPLETE_REPLACE. Leader / control-plane driven; does not need a CRAFT client. Listener gets on_start_replace_member / on_complete_replace_member / on_remove_member. nuraft supports learners natively via flip_learner_flag.
  • Baseline resync / snapshot for the backlog (<= startLSN): nuraft triggers it past resync_log_idx_threshold, drives chunking / transport / truncation; the listener provides the payload via read_snapshot_obj / write_snapshot_obj.
  • The CP (checkpoint) subsystem is the consistency primitive: INDEX_SVC and BLK_DATA_SVC flush together at a CP boundary, so a CP is an atomic point-in-time of index + blocks. CRAFT sources its snapshot payload by forcing a CP flush at the snapshot LSN (cp_mgr().trigger_cp_flush), pinning that CP, and serializing the index + referenced blocks. homestore's resync metadata carries only a DSN marker; the listener owns the index + block payload.

The reduced-availability window (control-plane only). homestore flips OUT to a learner, commits HS_CTRL_START_REPLACE, then adds IN as a learner, so RAFT voting runs 2-of-2 from the OUT flip until IN promotes, i.e. for the catch-up duration (a voter failure in that window stalls the cold path, elections and logins, until recovery). The data plane is unaffected: the client keeps writing 2-of-3 to the still-synced {A,B,C} (the outgoing member is a RAFT learner but still holds and journals data, and the client refreshes its member set only at re-login on completion), and the data path is leaderless, so a running session keeps flowing even if a voter fails in the window.

Catch-up is pull-based (so the client never broadcasts to a catching-up member): the backlog comes via the snapshot path; the tail + small gaps come via the existing SyncRSCommitLSN -> FetchData loop. If the tail grows too large for FetchData (the missing list runs past the journal retention window), fall back to a fresh snapshot, which advances startLSN; this mirrors homestore's resync_log_idx_threshold fall-back-to-baseline.

Promotion (learner -> voter) is gated by CRAFT's commit_lsn >= startLSN, not homestore's rLSN-based monitor (which is blind to CRAFT data, since data isn't in the log). startLSN starts as the dLSN at the moment of the change and advances with each snapshot pin (see Frozen view); the member reports commit_lsn in its keep_alive response. This gate is the data-plane quorum-safety mechanism for the backlog: a member must hold data through startLSN before it can count as a poll-quorum participant.

Snapshot callbacks (modeled on homeobject's ReplicationStateMachine): the closest existing template. Mapping: homeobject PG -> CRAFT partition; its shard + blob hierarchy -> CRAFT's flat LBA index + block data (one level, simpler). Transfer is a resumable, objId-cursored message stream (volume metadata -> index + data batches -> LAST_OBJ_ID), with a leader-side iterator in snp_obj->user_ctx, a follower-side receive handler that persists progress to a superblock for mid-resync crash recovery, async block prefetch with bounded inflight, and per-message CRC. Inherited gotchas: consistent-LSN capture (homeobject needed verify / re-read because a concurrent higher commit can overwrite an LBA mid-stream; CRAFT sidesteps this by serializing from the pinned CP below), never block in the callbacks, keep the member not-readable until the last object (this is the startLSN gate).

Frozen view (decided): CP-pinned and bounded. On snapshot need, trigger_cp_flush, pin that CP, and serialize the committed index + blocks as of it. Because the payload is read from a pinned CP (a point-in-time freeze), writes that land after the CP cannot mutate it, so no verify-on-read is needed for what the snapshot ships. The snapshot's startLSN = commit_lsn at the pinned CP, stored in the snapshot metadata so the tail FetchData loop resumes correctly. Bound the pin (time / size); a transfer that exceeds the bound releases the CP, re-pins a fresher one, and restarts with the new startLSN. Restartable: the follower checkpoints its receive cursor + startLSN to its superblock; a leader failure mid-transfer restarts the snapshot, and the follower resumes from its cursor or discards and restarts cleanly.

Self-elected stand-in client (recovery while detached). homestore's replace_member runs without a client, but CRAFT data-plane convergence normally rides client activity: the RAFT leader always proposes SyncRSCommitLSN (a client cannot propose a RAFT entry), triggered by the client's keep_alive / commit traffic, its watchdog, or the periodic checkpoint. When detached there is no traffic to ride, so the leader self-elects as a stand-in client to drive just that. No app IO when detached means no live tail, so it is pure backlog catch-up. Leader-only, so no stand-in split-brain.

Logout / lease (decided): hybrid. A locally-evaluated lease (the keep_alive watchdog) detects that the client is gone; a committed RAFT InternalLogout / lease-expiry entry grants the authority to act. The leader self-elects only after that entry commits (serialized, survives leader failover, and avoids a new leader re-deciding blindly), and yields on the next committed InternalLogin. The committed logout also marks "no active client", which sharpens the login fencing boundary (a following login is unambiguously a fresh session).

Net CRAFT-owned build list (small; everything else is homestore):

  1. The four snapshot callbacks shipping CRAFT's committed index + blocks.
  2. The commit_lsn >= startLSN promotion gate (the keep_alive response carries commit_lsn), interposed on complete_replace_member.
  3. The committed logout / lease record + leader self-election.
  4. Term-bump / re-login on HS_CTRL_COMPLETE_REPLACE so an attached client refreshes its member set. Re-login is triggered by ETERM: the client re-logins to the leader presenting its client_token; the leader accepts only if that token is the current owner (returns fresh term + members) and rejects a superseded token, so a deposed client cannot reclaim the volume. In-flight writes are re-driven under the new term.

Thin writes and holes on HomeStore

The protocol makes READ and WRITE thin (see the thin WRITE / READ model). The HomeStore binding:

  • Zero (all_zeros) write: metadata only, as above, no data-service write and no blkid. On commit it is an index node delete / update over [lba, lba+len): it removes the mappings any earlier (lower-dLSN) writes installed there and reclaims those blocks, leaving the range unmapped. An unmapped range reads as zero, so a delete and a canonical zero-mapping are equivalent end states; the index keeps it unmapped (nothing allocated), which is what preserves thinness. The client emits a zero write when a client-side scan finds an all-zero buffer; the server never re-scans a data write on the write path (that would be double work), so an all-zero buffer that still arrives as a data write is stored as blocks and normalized on read (below).
  • Hole on read: the LBA index is naturally sparse (a thin volume never maps a never-written LBA). A read walks the covering slots and the index for [lba, lba+len); any sub-range with no data mapping (never written, or unmapped by a zero write) is returned as a hole, never a materialized zero buffer. As a final step the read scans the data blocks it would return and collapses any all-zero region to a hole, so a data write that happens to hold zeros reads back thin too; this read-time scan is the only zero-scan on the server. The client / guest reads a hole as zeros, and a hole is not Missing.
  • Thin resync: because reads and fetches expose holes, baseline resync and FetchData copy only the mapped extents. The receiving member allocates nothing for holes, so a thin volume stays thin across a resync or a member replace. A fetched slot that is a zero write carries the all_zeros marker (distinct from an Empty no-op slot and from an omitted / not-present slot), so the lagging member applies it as the same index delete / update.

Implementation work breakdown

Full per-story acceptance criteria live in the repo at docs/craft/subtasks.md; this is the shape.

Story Scope Depends on
S1 CraftReplDev foundation: class, per-partition state, journal backend, multi-member RAFT group, listener skeleton (landed) none
S2 Write path (write: term check, journal append, last_append update, out-of-order tolerant, zero-copy; thin all_zeros write: metadata-only slot, no data-service write, no blkid) S1
S3 Commit + Read path (commit/keep_alive advance the in-order apply frontier + reclaim, zero-write commit = index delete/update; read carries horizon H, client-routed by LBA-overlap, served from the index or the journal-tail overlay, sparse result with holes, no read-path fetch, no index write on reads; watchdog) S2
S4 Truncate (drop journal entries above an LSN; clear Missing above it) S1
S5 RAFT entries: SyncRSCommitLSN (leader pre-resolution + empty_slots[] verdicts) + InternalLogin apply; checkpoint triggers (periodic / watchdog / client-requested) S1, S6
S6 Peer data exchange: get_lsns/get_rs_commit_lsn, fetch_data (all_zeros slot kind) S1
S7 Login orchestration (leader-side; watermark = max(quorum.last_append)) S2, S4, S5, S6
S8 CRAFT volume lifecycle: create multi-member group, recover state from journal/superblock; journal reclaim floor + RAFT-log compaction tied to it (the returning-voter snapshot trigger) S1
S9 CraftConnector frontend (transport-agnostic; in-process scaffold first) S2, S3, S7
S10 Reconfiguration: replace_member integration + 4 snapshot callbacks (CP-sourced payload, homeobject-modeled) + commit_lsn >= startLSN promotion gate + logout/lease record + leader self-election + re-login on complete S5, S6, S8

Development scaffold: all of this is implementable and testable now with multiple in-process HomeBlkDisk instances making direct API calls (catch-up is just inter-instance calls) in parallel with real-transport selection.

Open items

  • Real transport (IP/TCP): planned (CRAFT-1). The in-process multi-HomeBlkDisk + direct-API-call path is a development scaffold to build and validate the protocol in parallel with transport selection, not the end state. CraftConnector stays transport-agnostic so the chosen transport drops in behind it.

Reference detail in the repo

The docs/craft/ folder in the HomeBlocks repo holds the wire-format and API reference this page and the protocol page summarize:

  • docs/craft/api.md - the CraftReplDev C++ API.
  • docs/craft/rpcs.md - all RPC wire formats (client to server and server to server).
  • docs/craft/subtasks.md - per-story acceptance criteria (the S1-S10 breakdown above).

Substrate references. HomeStore (storage engine: journal / index / blkdata / RAFT / CP) and HomeObject (blob store; the snapshot + replace_member template CRAFT follows): https://github.com/eBay/HomeStore, https://github.com/eBay/HomeObject

HomeBlocks

CRAFT

Clone this wiki locally