feat(upgrade): data-safe core of the streamed old->new handoff (part of #391)#594
Merged
Conversation
… safe core) Phase 2c of the upgrade epic (#385): stream the live keyspace from the OLD process straight into a sibling NEW process over a unix socket, so the new one can serve BEFORE the old drains, with no disk round-trip. This lands the DATA-SAFE CORE and clearly defers the live-serve wiring. Streaming protocol (reuses the reviewed ironcache-repl machinery) A localized old->new full transfer is a Redis-style full-sync + tail, which ironcache-repl already implements and proves converges (HA-7b/7c). This module ORCHESTRATES those pieces over a unix socket rather than re-deriving them: - BULK: ShardStore::snapshot_chunk pulled in bounded chunks (constant memory: pull under the store borrow, DROP the borrow, THEN await the sends), shipped as SYNCKV frames, replayed via insert_object (type/encoding/TTL/ collections round-trip). - DELTA: writes during the transfer are captured by the HA-5a ReplObserver into a shard-local ReplRing as offset-ordered StreamOps, shipped as STREAMPUT/STREAMDEL, applied by ReplicaApplier in offset order, idempotently. The cut (end_offset) is captured BEFORE the scan so any write during the scan lands in the delta (last-write-wins). Wire envelope (CRC'd + versioned + length-prefixed, fail-closed, like #530) Every message is wrapped in a 16-byte envelope (MAGIC "ICHO" | version | kind | reserved | len | crc32) then payload; the CRC (reused ironcache_persist::crc32) covers the whole message with the CRC field zeroed, so a torn header OR payload is caught. Decode is TOTAL and fail-closed: a wrong magic, an unknown version, an over-cap length, a CRC mismatch, a truncated read, or an undecodable inner frame each ABORT (never a silent mis-parse), the same posture #530 took on disk. Atomic cutover across shards (the one place shards synchronize, ADR-0002) Shared-nothing => one send_shard/recv_shard pair per shard. The pure CutoverBarrier gathers every shard's result and flips ALL-OR-NOTHING: Commit only when EVERY shard committed; Abort the instant any shard aborts (sticky, wins over any commits). No window where some shards are new and some old for a client, so no key is ever served by neither/both. Abort-safety (never lose data, never serve stale) The durable data_dir snapshot is NEVER touched, so it always remains a valid fallback. The SENDER only READS its store (snapshot_chunk) and ring; it never mutates or drops the store, so on ANY sender error the OLD process keeps serving intact. The RECEIVER builds a FRESH store and returns it only on the complete, CRC-verified, cutover-acked path; on ANY error (bad magic/version/ CRC, over-cap, truncation/EOF, undecodable frame, delta gap, offset mismatch, peer abort, delta overflow) the fresh store is DROPPED (adopt nothing) and the sibling exits without ever serving. The old process stops serving a shard only after its cutover is acked (barrier Commit), so neither side is ever down. SO_REUSEPORT handling Documented: systemd socket-activation (#389) is the SUPPORTED no-RST path (systemd owns the listener fd and passes it to the new process, so it is never closed across the handoff and no accept-queue backlog is orphaned). The non-socket-activated SO_REUSEPORT case needs a drain-and-final-accept, which is part of the deferred acceptor wiring below. Deferred to the follow-up (why: each touches the live datapath / process lifecycle and warrants its own data-loss-focused review) - the orchestrator spawning the sibling + passing the socket path; - the coordinator boot pulling over the socket instead of load-from-disk + installing the observer/ring on each old shard at handoff start; - the dispatch-layer -LOADING write-quiesce across shards + the atomic serve-flip on the sibling; - the acceptor drain-and-final-accept for the non-socket-activated case. The IRONCACHE_HANDOFF_SOCKET config/env knob is added now (default None = streamed path off, byte-unchanged) so that wiring has its rendezvous path. Determinism (ADR-0003): ops-path, off the engine decision path; no clock/RNG (the lazy-expiry `now` is the caller's env-seam clock). No unsafe. No dashes. Tests (data-safety first): a real AF_UNIX-socket multi-shard transfer serving every key after cutover; delta correctness (create/overwrite/delete during the transfer present after); THREE abort proofs (db-count mismatch, delta overflow, peer crash mid-transfer) each leaving the OLD store intact and adopting NO partial store; the framed-wire CRC/version/magic/malformed-frame fail-closed codec; and the cross-shard barrier atomicity (a single abort vetoes the flip). Existing upgrade + persist + turnkey + config tests pass; clippy -D warnings and the CI invariant + prior-art checks are clean; io_uring build is clean. Part of #391. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Zeke <ezequiel.lares@outlook.com>
perf-gate (A5)Same-runner ratchet of HEAD against the merge-base (both rebuilt and measured in this job).
Overall: PASS
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Upgrade epic #385 Phase 2c: the data-safe CORE of the near-zero-downtime streamed handoff (stream state old->new over a unix socket so the new sibling serves before the old drains). Landed as Part of #391: the protocol + delta + abort-safety + atomic cross-shard cutover barrier; the live-serve wiring (orchestrator spawn, boot pull-over-socket,
-LOADINGquiesce + serve-flip, acceptor drain) is DEFERRED to its own PR (each touches the live datapath/lifecycle and deserves its own review).Reuse, not reinvent: the repo already has a proven transport-agnostic full-sync + delta engine in
ironcache-repl(drive_full_sync/receive_full_sync,StreamOp/ReplRing/ReplObserver,ReplicaApplier) +snapshot_chunk/insert_objecton the store +crc32in persist. A localized old->new handoff IS a Redis-style full-sync+tail, soupgrade/stream.rsORCHESTRATES those over a unix socket rather than deriving a second protocol.MAGIC "ICHO" | version | kind | len | crc32+ payload; decode is total and fail-closed on bad magic / unknown version / over-cap len / CRC mismatch / truncation / undecodable inner frame (harden: format-version-mismatched snapshots must fail loudly, not boot silently empty #530 posture).snapshot_chunk(the perf: per-slot Arc copy-on-write snapshot isolation, uncontended datapath during a save (#576 core) #588 frozen Arc-COW view = consistent, non-blocking) ->SYNCKV->insert_object, constant-memory borrow discipline.end_offsetis captured BEFORE the scan so any write during transfer lands in the delta (last-write-wins);StreamOps ship asSTREAMPUT/STREAMDEL, applied in-order;Cutover{final_offset}verified equal to the receiver's applied offset before ack.CutoverBarrier, pure): the ONE cross-shard sync point (ADR-0002);Commitonly when EVERY shard committed,Abortsticky the instant any shard fails.Abort-safety (no data lost on ANY failure)
Durable
data_dirsnapshot never touched (always a valid fallback); the sender only READS its store/ring (never mutates) so any sender error leaves the old process serving intact; the receiver builds a FRESH store and adopts it ONLY on the fully CRC-verified, cutover-acked path, dropping it on any error (adopt nothing, sibling exits without serving); the old stops serving a shard only after its cutover is acked, and the all-or-nothing barrier means no key is served by neither/both.Tests (14, over a real AF_UNIX socket)
Multi-shard transfer serving every key after cutover;
writes_during_transfer_are_present_after_cutover(delta correctness); three abort proofs (db_mismatch_aborts_and_leaves_old_serving,delta_overflow_aborts_and_adopts_nothing,receiver_crash_mid_transfer_aborts_and_leaves_old_serving) each leaving the old store intact + adopting nothing; framed-wire CRC/version/magic/malformed fail-closed;barrier_commits_only_when_all_shards_commit+barrier_abort_is_sticky_and_wins_over_commits(atomicity). 602 ironcache + upgrade/persist/turnkey/config suites pass; clippy-D warnings+ io_uring + invariant + prior-art + fmt clean; NO unsafe; no dashes.Deferred to the live-wiring PR (gets a data-loss-focused adversarial review)
Orchestrator spawning the sibling; coordinator boot pulling over the socket + installing the observer/ring on old shards; dispatch
-LOADINGwrite-quiesce + serve-flip; acceptor drain-and-final-accept (socket-activation #389 is the no-RST path). Thehandoff_socket/IRONCACHE_HANDOFF_SOCKETrendezvous knob is added now (default off, byte-unchanged). Streamed path assumes identical old->new shard topology (a shard-count change must fall back to the durable re-shard load).Part of #391. Part of #385.
🤖 Generated with Claude Code