Prune ACTIVATION: durability fence + unclean-recovery rebuild + pre-activation gate hardening [SPEC-342j] - #111
Merged
Conversation
- Add assigned_write_sequence() and flushed_watermark() to the MapDataStore trait (default 0), the store abstraction the tombstone frontier reads through (it holds Arc<dyn MapDataStore>) — no concrete downcast. - Correct the stale RecordStore::soft_flush doc: it returns the last QUEUED (assigned) sequence via MapDataStore::soft_flush, not the last flushed one.
…ecovery rebuild - write_behind: track a prefix-complete flushed watermark from the flush loop — a pending-sequence set resolved on inner-store flush success, coalesce-retire, and terminal discard; expose it via flushed_watermark()/assigned_write_sequence(). - frontier: stamp captures the store's assigned write-sequence as each epoch's byte-durability bound; durable_epoch_watermark is now computed as max E with every epoch <= E flushed (was constant 0). Add effective_low_water_mark clamp and the unclean-recovery E_rec rebuild (max-cursor keyspace scan + live-tombstone restamp into one maximally-lagging epoch). - bin: invoke the rebuild in the pre-listener WAL-recovery window (before set_ready/accept), fail-closed on scan error. Prune + re-admission gate go live together (gate-before-activation).
- AC3c prefix-completeness: an out-of-order resolve never advances the flushed watermark past a still-pending sequence (white-box) + real add/hard_flush advances it only on byte durability. - R12(b): the WAL retains an un-flushed tombstone frame until it is applied (guard against a WAL-GC change that both-loses the bytes). - AC3f: double-crash (WAL-fsync then kill -9; crash again mid-recovery before mark_applied) — the tombstone is recoverable at every step, no resurrection. - AC3d: kill -9 recovery rebuilds all live tombstones into a maximally-lagging E_rec exceeding every persisted cursor epoch; nothing prunable until clients re-confirm; effective_low_water_mark is the durable-backed clamp. - AC3e: activation end-to-end — dark while buffered, real prune (RAM) once the bytes flush (inverts the 342b AC3a dark-mode conjunct).
…h-remove coalesce Two durability-fence correctness fixes surfaced by the review-gate cross-vendor pass and by verifying every write path that mutates the pending-sequence set: - Fold the ordering-sequence bump and the pending-set insert into one assign_tracked_sequence() under the pending_seqs lock. The prior bump-then-track left a window where the counter was already incremented but the set still empty, during which flushed_watermark()'s empty-set branch returns sequence.load() — a value above the just-assigned, still-buffered sequence, violating the documented prefix-complete (no mid-range hole) contract the tombstone prune fences on. - remove_all's batch coalesce retired a possibly-still-buffered predecessor without resolving its sequence, leaking it in the pending set forever and pinning the flushed watermark below it — a permanent prune stall. Resolve the retired sequence on coalesce, symmetric with add()/remove(). Regression test proves the batch-coalesce watermark advances past the retired sequence. Full gate: fmt/clippy -D warnings clean, lib 1598/0, sim 23/0.
… Minor #1) Convert the fresh SPEC-342j R12(c)/(e) leg-ID in bin/topgun_server.rs to a plain WHY-comment per the CLAUDE.md no-spec-refs-in-code convention. The tombstone_frontier_impl.rs leg-IDs are left consistent with the file's established pre-existing 342a/342b convention (36 refs).
…ion regressed bypass) The continuation/push gates key on the not-yet-admitted signal delivered(conn) == 0, but a REUSED connection carries delivered > 0 from an earlier healthy round on the same socket — a regressed replica's sync-init would route to REPLACE while its continuations slipped past the gate and could pull incremental deltas across pruned tombstones. Gated routing now restores the signal (strictly conservative: only defers re-admission until the snapshot lands and a fresh CLIENT_APPLY_ACK arrives). This was the 342c Review-v2 residual queued 'must land before 342j activation' — landed with the activation branch.
Deploying topgun with
|
| Latest commit: |
f318d26
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://581a62a1.topgun-f45.pages.dev |
| Branch Preview URL: | https://feat-sf-342j-prune-durabilit.topgun-f45.pages.dev |
…r untracked devices The op-path gate keyed on the frontier's 'unknown == forgotten': every device that had not yet completed its first ACK round was silently dropped on OR-bearing ops while still receiving OP_ACK — a fresh device flushing its pending oplog on connect would have its writes acked, cleared from the client oplog, and never applied: permanent data loss on both sides once the durability watermark activates. The gate was spec-recorded as non-load-bearing (client-originated tags are regenerated server-side, so the op path cannot re-present a pruned tombstone's tag — resurrection-proof by construction); removing it costs nothing and un-breaks every pre-ACK client. The load-bearing verbatim-tag gate on ORMapPushDiff is untouched. Found by the crdt-ormap integration suite (first run with an ACTIVE watermark end-to-end); helpers/index.ts gains an env-gated server-log passthrough (SERVER_LOG_PASSTHROUGH=1) used to diagnose it. Gate: server lib 1599/0/1, integration 106/106 (17 suites), clippy -D warnings, fmt clean.
…oise-retry) The gauge is a process-global static; exact-equality snapshots raced every unmarked parallel test's OR_REMOVE applies and prune drains (first tripped in CI debug mode once the op-path gate removal let more parallel ops through). serial(tombstone_gauge) serializes the gauge-asserting tests; ambient noise from unmarked tests is absorbed by retrying the scenario on a fresh map — a genuine double-count / dropped-gauge regression is deterministic and fails all five attempts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR activates the OR-Map tombstone prune — the durability half that SPEC-342b deliberately shipped dark (
durable_epoch_watermark ≡ 0). Full SpecFlow lifecycle (audit APPROVED, review APPROVED, /xreview gate — 2 HIGH refuted against the actual WAL-backed production config).What ships
write_behind.rsflush-loop driven — NOTsoft_flush's last-assigned return; trait surface onMapDataStore/record_store), feeding the realdurable_epoch_watermark: prune requires BOTH the fleet-wide-MIN cursor LWM AND byte-durability past the epoch.E_rec(> every persisted cursor + derived watermark bound), inside the pre-listener recovery window;effective_LWMclamp for all consumers.delivered(conn)to 0, restoring the not-yet-admitted signal on a REUSED connection — without it a regressed replica's continuations slipped pastsync_gated_continuationon the stale signal and could pull incremental deltas across pruned tombstones (+ regression test).Activation safety recap
Gate-before-activation held end-to-end: prune only ever runs with the re-admission gate (342c) live and the client-side cross-map ACK min-barrier + injective map-name scheme (SPEC-343, PR #110) already merged. No prune-without-gate window has existed on main at any commit.
Test evidence
-D warnings/ fmt clean; behavioral durability-fence + recovery tests in-branch; sim suite runs in CI.flush_keyresolve-after-durable error-path +reset_lockscope — WAL-backstopped, own test+xreview cycle).Unblocks
SPEC-342f (legacy-tombstone migration needs the ACTIVE prune) → 342g (two-path + fault-injection sim proof) → 342h (soak monitor gate) → the 72h G4b soak.