draft(stage7-p0): P0 integration branch — CI evidence only, merge held for F1#29
Merged
Conversation
added 30 commits
July 8, 2026 15:53
Publish and seal native-era XID authority after seed shutdown, adopt prehistory from per-node anchors, and max-merge startup nextXid before ShmemVariableCache seeding. Spec: spec-6.15b-xid-authority-native-era.md
A second cluster.enabled=off seed pass on a SEALED authority re-opens the native era: clear SEALED at bootstrap so a crash of that pass leaves joiners fail-closed (53RB5, unsealed) instead of silently adopting the previous pass's stale high-water -- false-invisible for every xid the crashed pass consumed. The pass's own clean shutdown re-publishes and re-seals monotonically. Adds cluster_xid_authority_begin_native_run(), the bootstrap enabled=off arm call, unit coverage (seal kept-hw/cleared-flag/idempotent/re-seal), and t/361 N4 (sealed pass-1 -> unsealing pass-2 boot -> immediate-kill -> joiner refuses adoption). Spec: spec-6.15b-xid-authority-native-era.md
…n-gate window spec-6.15b D6 flipped both tests to online_join+xid_striping: formation now passes through the join gate, whose home-block rebuild window transiently answers 'block master is recovering ... retry is safe' -- even at connection time, via the pg_authid read. Retry the first-contact probes and first DDLs (bounded, 60s); every semantic assert stays strict. Spec: spec-6.15b-xid-authority-native-era.md
…I3 recheck, P1-1/P1-2 hard assertions, t/363 rename Review r1 dispositions (0 P0 / 3 P1 / 2 P2, all fixed): - P1: join Phase-1 PREPARE TIMEOUT/non-ACK now drains best-effort and still publishes the staged JOIN_PENDING (the pre-async code ignored PREPARE results; aborting reverted the joiner to DEAD and the next tick re-detected it and re-bumped the epoch once per deadline period — the exact bump storm the staged record forbids). The revert helper is removed; t/363 gains a bounded epoch-advance assertion across the 95s hold window. - P1: clean-leave staged-ACK handoff re-runs the CL-I3 dead_gen-aware version_coherent check before apply (the COMMITTING marker wait now spans ticks; the guarded CAS alone misses a dead_gen-only move). - P1: unit hard assertions — fail-stop fence stage bumps the epoch exactly once while the marker is PENDING and publishes only on ACK; node-remove re-entry while PENDING reports no false contest and no re-bump. - P2: node-remove FENCE_ARMING skips REMOVING-marker/stripe-retire pre-work while the staged fence marker is in flight (new cluster_reconfig_node_removed_fence_stage_pending predicate). - P2: PG-style banners (Author/IDENTIFICATION/NOTES) on the new files. - TAP renamed t/358 -> t/363 (358-362 reserved by parallel lanes); nightly shard split accordingly. Unit fixture gains MyBackendType stub (B_INVALID) and controllable async-marker stubs. Local gates (cassert build): cluster_unit 158 binaries, t/363 all legs, smoke+touched-baseline TAP 13 files/327 tests, cluster_regress 13, PG 219/219, clang-format 0 violations, scn-cmp + no-clog-overlay clean. cppcheck reports one finding in test_cluster_pi_shadow.c:210 — byte-identical to the base commit, local cppcheck 2.20 version drift, not introduced by this branch. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…n unsupported seed, .bak read guard, multi-segment unit leg Review P2-1: the native-era publish+seal now runs only on a TRUE shutdown checkpoint (END_OF_RECOVERY excluded -- sealing mid-run after crash recovery would expose a stale high-water to joiners), and an unsupported native era (MultiXacts / past the prehistory cap) skips the seal with a WARNING instead of FATALing the checkpointer: the refusal still lands fail-closed on every joiner (53RB5, unsealed) but the seed can shut down cleanly instead of wedging in a FATAL loop. Review P3-4: roll_blob_to_bak PANICs on a read() error instead of treating it as EOF (a truncated .bak was already rejected downstream by CRC, but silently swallowing I/O errors broke the module's discipline). Review P3-3: unit multi-segment prehistory round trip (33 pages across SLRU segments 0000/0001, per-page byte pattern, short-clone extension). Spec: spec-6.15b-xid-authority-native-era.md
… hardening F1: begin_native_run/mark_cluster_era install the transitioned image as BOTH primary and .bak (write_header_both) -- rolling the pre-transition image into .bak let a later single-copy corruption fall back to a stale SEALED (or pre-CLUSTER_ERA) authority, handing joiners the previous pass's high-water or re-opening era re-entry. F2: divergent-lineage guard -- before the adopt/skip decision the joiner byte-compares its local pg_xact prefix [0, min(own_next, native_hw)) against the sealed prehistory blob at 2-bit precision; any contradiction is FATAL (a same-sysid clone that ran standalone after cloning is not a pre-seed lineage; neither trusting nor overwriting its bits is sound). Bypassed only when local oldestXid has advanced past the native range. t/361: real scram TCP login leg (hba + listener), node0 shared-heap re-reads after joiner scans (poison proof on the heap, not just CLOG), a 53RB5 SQLSTATE pin (%e before %q -- %q stops expansion for the postmaster), the N5 divergent-lineage negative, and the L9 shared_catalog=off dormant leg. Also: on-disk header layouts locked with StaticAssertDecl; consumers LOG when the authority is served from the .bak fallback; user manual trimmed to user-visible effect. Spec: spec-6.15b-xid-authority-native-era.md
… WAIT_CLUSTER watchdog, event-scoped WAIT_EPOCH escape, dead-node PCM cleanup A fail-stop reconfig could wedge a settled cluster forever on three stacked liveness holes: a BLOCKED HW remaster worker was never relaunched within the same episode (BGW_NEVER_RESTART + launched-latch), WAIT_CLUSTER had no watchdog while its cluster/HW gates were held, and WAIT_EPOCH's strict progress gate wedged permanently when an IC-piggybacked epoch bump landed before the local reconfig event (baseline re-captured as the post-bump value, cur == old forever). - HW remaster: per-dead-node result/attempts/next_attempt shmem state; the FSM relaunches BLOCKED workers with exponential backoff (cap 60s) up to cluster.hw_remaster_retry_max_attempts (SIGHUP-raisable); abnormal worker exit marks BLOCKED via before_shmem_exit; the three silent BLOCKED returns now LOG their cause. wal_threads_dir-unset is detected up front as BLOCKED_STRUCTURAL: never retried, WARNING-once with the config hint, and (level 2) a multi-node shared_catalog=on boot without it now fails fast at startup. - WAIT_CLUSTER: observational watchdog on the rebuild-timeout cadence — WARNING with the missing-survivor list, gate states and per-dead-node retry state, plus a counter; it never unfreezes anything. - WAIT_EPOCH: three proof-carrying layers — abort-to-idle on a new event id, durable observed-epoch adoption, and an event-scoped coordinator witness (REDECLARE_DONE carries the event id; stale-event DONEs are dropped at accounting; the equal-epoch escape additionally requires the coordinator's DONE to have been accounted after the accept snapshot). No timeout-only advance exists; without proof the shards stay frozen. - Dead-node PCM cleanup: the GRD dead-sweep clears the dead node's x_holder/s_holders/pi_holders/master_holder and pending-X residue (LOG summary + pcm/dead_cleanup_entries counter), and a local-master S state with no local residency can now upgrade S->X through the standard invalidate/ACK path instead of failing closed forever — post-kill DDL recovers instead of timing out on cluster locks. - Serve-gate scope alignment: the merged-materialization check in gcs_block phase_for_tag now uses the same thread-recovery scope policy as the unfreeze gate, removing the permanent-RECOVERING mismatch outside the gate's applicable scope (in-scope behavior unchanged). - Observability: hw retry/exhausted + grd cluster_gate_timeout / wait_epoch_escape + pcm dead_cleanup_entries dump keys; S4-reject and HW fail-closed diagnostics. Tests: unit relaunch-decision table (8 rows) + event-scoped DONE rejection/ witness advance + PCM dead-cleanup forms incl. pending-X; t/293 gains the same-episode self-heal and retry-exhaustion legs (unfreeze-extend positive + watchdog WARNING greps); t/337 gains the level-2 fail-fast negative leg; new t/362 runs a 4-node shared_catalog formation, kill -9, and asserts remaster convergence, post-kill DDL, cleanup counters and the 0-wedge invariant end to end. Local gates (cassert): unit 158 binaries, t/293 + t/337 + t/362 + smoke subset, cluster_regress 13/13, PG regress 219/219. Spec: spec-4.6a-grd-recovery-liveness.md
t/358-361 are reserved by parallel lanes (spec-7.2 / S-xid), t/363 by the S-dead lane; occupancy verified against origin branches. The 4-node formation + kill -9 + convergence run gets its own shard for wall-clock isolation. Spec: spec-4.6a-grd-recovery-liveness.md
…adow unit Same inline suppression + reason as the other cluster_unit main() signatures (test_cluster_grd.c precedent); a newer local cppcheck flags it, the CI baseline version does not.
…y baseline + async pre-bump WAIT_EPOCH wedge The ship-gate nightly on the marker-async head surfaced two lane-introduced regressions that the fast-gate matrix does not cover: 1. Dump-category baseline miss: reconfig telemetry added a pg_cluster_state category (55 -> 56), but three cross-node TAPs that assert the total (t/204/205/206) were not updated — only the locally-run L405 subset was. Bump them to 56 with the spec-2.29a note. 2. Async pre-bump WAIT_EPOCH wedge (behavior regression): the async fence / join-Phase-1 / node-remove marker staging bumps the membership epoch at stage-entry but only publishes the reconfig event once the voting-disk marker ACKs — now several LMON ticks later instead of the pre-async single-tick spin. In that window the coordinator's OWN GRD recovery IDLE tick re-captured the already-bumped epoch as its WAIT_EPOCH baseline, so the P0 accept that followed the publish read old == cur and froze the affected shards forever — the spec-4.6a section-0 shape, here triggered by the coordinator on itself with no IC piggyback (reproduced deterministically as t/293 fail-stop remaster + t/326 node-leave shard reopen). Fix: while any pre-bump stage is live, the GRD IDLE tick holds its last stable pre-reconfig baseline instead of re-capturing (cluster_reconfig_has_pending_prebump_stage guard). This restores the pre-async bump->publish atomicity as seen by GRD without touching the epoch bump timing or any user-visible reconfig contract. Adds the cluster_grd_recovery_event_old_epoch accessor + a baseline-hold unit test. Local gates (cassert): unit 158 binaries (incl. baseline-hold test), t/293 + t/325 + t/326 family all green (were red), t/204/205/206 + observability TAPs, PG regress 219/219, clang-format/headers/scn-cmp clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…sion ②b, others-dead) The spec-2.29a marker-async change turned the clean-leave COMMITTING marker into an async, multi-tick wait. In a slow environment (nightly Linux) the leaving node finishes its drain and stops heart-beating inside that wait window, so CSSD marks it SUSPECTED->DEAD and bumps the global dead_generation — and the coherence gate, which compared that SCALAR dead_generation, wrongly read it as a third-party death intruding mid-drain and escalated an otherwise healthy clean leave (t/310 dormant-member legs, t/331 clean_leave x idle). apply_clean_leave_as_coordinator bumps + publishes atomically, so this is NOT the baseline-hold wedge (②a); it is the coherence predicate counting the leaving node's OWN expected DEAD. Fix: compare an others-dead bitmap (the dead set EXCLUDING the leaving node) instead of the scalar dead_generation, at all four coherence sites (CL_COHERENT macro, drive_drain commit-point, staged-ACK re-check, non-staged pre-check). cl_state snapshots the others-dead set at bind; the pure predicate does a bitmap compare (fail-closed on a missing view) and is unit-tested against the reflexive-case matrix. Every third-party incoherence still escalates (epoch move OR a non-leaving node entering the others-dead set); only the leaving node's own expected transition is tolerated. 8.A / CL-I3 protection is unchanged — argued in spec-2.29a §②b with a case matrix. Local gates (cassert): policy unit reflexive-case matrix, cluster_unit 158 binaries, PG regress 219/219, clang-format/headers/scn-cmp clean. NOTE: t/310/331 are timing-sensitive — the false escalation only reproduces in a slow environment, so the fix is verified by nightly, not locally (a local run passes the assertions). A separate pre-existing cassert teardown SIGABRT flake in t/310 (isolated: reproduces without this change) is unrelated. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
…hash) The P6 all-done gate and the WAIT_EPOCH coordinator witness keyed the cross-node DONE on event_id, but event_id folds the sender-local cssd_dead_generation, which drifts with each survivor's private flap observation history and never converges across nodes: any flap asymmetry made the survivors compute different ids for the same episode, drop each other's DONEs, and wedge P6 forever — nondeterministically reintroducing the permanent-freeze shape this branch exists to remove. The quorum-accepted dead SET is what actually converges, so the DONE payload now carries a hash over the dead bitmap alone (same kernel as the event_id hash, riding the same request-id field pair — no envelope change), stamped per episode at P0 accept. Accounting, the P6 gate and the witness compare against the stamp; event_id stays a purely local accept-dedup scope. A late DONE from a previous episode still cannot back a new one: a coordinator re-election changes the dead set (hash mismatch) and a same-node re-death rides a higher epoch through the interposed JOIN bump (epoch conjunct). Pre-accept frames now mismatch the previous stamp and are dropped; senders re-announce every tick, so accounting always lands after the accept snapshot — closing the first-event pre-accept window as a side effect. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R2/R4)
Revert the scope short-circuit that returned NORMAL for a dead static master's blocks wherever online thread recovery cannot run (the default configuration and every >2-node deployment): it skipped both the is_materialized cold-block door and the redo-coverage lost-write door, so a committed write only the dead node saw could be silently read stale from shared storage. No other guard sits on that read path — the GRD freeze ends with the episode, the HW gate covers only extend high-water marks, and the page-SCN checks ride the ship path. Out of scope the posture is now an explicit bounded retryable error (53R9L, hint updated to name the way out: restart the failed node, or enable online thread recovery in a supported scope) for every tag whose static master is dead — including never-written blocks, which the cold-block door cannot distinguish from not-yet-replayed ones — and the door reopens the moment the failed node stops being DEAD. In-scope behavior is byte-identical. The now-orphaned scope helper is removed. t/362 asserts the honest two-outcome contract on fresh connections (success or explicit SQLSTATE — 53R9L or the TT-unknown visibility door — never a hang), rides pre-warmed sessions for convergence observability, and pins the D12 cleanup counter to a positive delta; t/293's post-kill extend leg follows the same posture. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R1/R3)
…grade cleanup - Order the worker's terminal-result store after its backoff-deadline store (pg_memory_barrier) so the relaunch decider never pairs a fresh BLOCKED with a stale deadline and skips one backoff wait. - A structural cause discovered only by the WORKER (SIGHUP race) now leaves the retry deadline unset so the FSM's next tick takes the MARK_STRUCTURAL branch and emits the once-per-episode operator WARNING with the configuration hint. - Wrap the local S->X upgrade in PG_TRY and release the temporary S claim when the ACK wait throws (cancel via CHECK_FOR_INTERRUPTS), not only on the false return; document why the completed upgrade hard-resets the local S refcount (the X grant subsumes all local S declarations). Linker-only exception stubs for the units that link cluster_pcm_lock.o. Spec: spec-4.6a-grd-recovery-liveness.md (Amendment v1.2 R5/R6/R7/R9)
…e-bump shmem bit Two review-r2 items on top of the ②b others-dead coherence fix: P2-1 — the leaving node's barrier-tick own-commit latch (cl_leaving_barrier_tick) inferred 'my leave committed' from epoch>baseline AND the others-dead bitmap unchanged. The bitmap is not monotone: a third-party node that false-fail- stopped (bumping the epoch) then recovered leaves CSSD hysteresis DEAD->ALIVE, so the bitmap rebounds to its bound value while the scalar dead_generation only advances. If the leaver's first epoch>baseline observation lands after that rebound it would mis-latch a leave the survivor coordinator actually refused, suppress the barrier-deadline escalation, and hang forever in BARRIER_WAIT. Restore the scalar conjunct (epoch>baseline AND others_dead==bound AND dead_gen==baseline) via the pure, unit-tested cluster_clean_leave_own_commit_ latched predicate. This does not reintroduce the ②b false positive: the leaver's OWN alive->DEAD never bumps ITS OWN dead_generation (a node does not observe itself dead), so on the leaver side the scalar stays at baseline through its drain. Survivor-side coherence keeps the bitmap-only check. Adds the rebound negative-leg unit (others_dead unchanged + dead_gen advanced -> escalate). t/274 — a backend-context coordinator (the fail-stop inject test) advances the epoch then SYNCHRONOUSLY waits for the fence marker before publishing (MyBackendType != B_LMON path). During that seconds-long wait the epoch is bumped but unpublished, and a concurrent LMON GRD IDLE tick re-captures the post-bump epoch as its WAIT_EPOCH baseline -> old==cur -> the thread replay slot never reaches DONE -> HG#1b/HG#5 done-poll timeout. The ②a baseline-hold guard only reads LMON-process-local staged flags, invisible to a backend's pre-bump window. Publish it in shmem: ReconfigShmem.prebump_sync_active set before the bump, cleared at every return; the guard reads the shmem bit OR the local stage. 8.A (durable-marker-before-publish) unchanged. Reproduced the wedge locally at 87f7930 (HG#1b slot-DONE poll times out forever), then verified the fix: t/274 goes 18/18 green (exit 2 -> 0). This also fixes the long-standing main flake (the wedge is deterministic once LMON is freed from the sync-marker park; it was merely probabilistic before) — the t/274 known-flake / L66 registration should be revised. Local gates (cassert): cluster_unit 158 binaries (incl. own-commit-latched + version-coherent matrices), t/274 18/18, PG regress 219/219, clang-format/ headers/scn-cmp clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
The leaving node's barrier-tick own-commit latch must be backed by
direct evidence (the durable COMMITTED marker confirmation for THIS
leave attempt) and must be immune to third-party transient false-DEAD
flap noise: the scalar dead_generation is monotone, so a flap during
the leave window advances it forever and the r2 P2-1 three-conjunct
inference then refuses a healthy committed leave until the barrier
deadline escalates it (nightly t/331 C1/C4 false-escalation, run
28948167577). The reworked U3b matrix pins:
(a) evidence, no noise -> latch
(d) evidence + flap noise -> still latch (RED on current code)
(c) no evidence, any noise state -> never latch (deadline escalation
stays armed; the r2 P2-1 refused-leave mis-latch wedge stays shut)
Known-red TDD commit: leg (d) fails on the current three-conjunct
predicate; the follow-up commit flips the latch to marker evidence.
Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
A crash between write_header_both's two durable renames leaves the transitioned flag (unseal / CLUSTER_ERA stamp) in the primary copy only, with a stale pre-transition .bak behind it. Nothing on the restart path rewrote the pair -- the bootstrap gates skipped the transition call when the primary already showed the new flag, and the transition functions early-returned on the same observation -- so the stale .bak survived the whole native run. Any later transient primary read failure then fell back to it: a stale SEALED .bak hands joiners the previous pass's high-water (false-invisible + xid reissue), and a stale pre-CLUSTER_ERA .bak re-opens the native-era re-entry guard. Make the two flag transitions idempotent re-asserts: the bootstrap calls them unconditionally on every boot of their respective arm, and they rewrite BOTH copies until both validate with the post-transition flags (a missing or invalid copy counts as unsettled and is restored too). Once both copies are settled the call is a no-write no-op, so the steady-state boot cost is two header reads. All fail-closed semantics (53RB5 refusals, CLUSTER_ERA re-entry FATAL, corrupt-authority PANIC) are unchanged. Red-first unit coverage: torn-window tests for both transitions (complete the transition, re-install the old image as .bak, re-run the boot path, assert the .bak is repaired and that a primary read failure falls back to the repaired image, not the stale one). Spec: spec-6.15b-xid-authority-native-era.md
…osed) The divergent-lineage prefix check broke out of its comparison loop and returned CONSISTENT on the FIRST missing local pg_xact page. That is only sound for a limit that ends where the local tree ends; pg_xact front truncation (SimpleLruTruncate removes whole low segments once every xid they cover is frozen past oldestXid) means a joiner can have segment 0000 legitimately absent while later segments are present AND divergent -- the first-page ENOENT then declared the clone CONSISTENT without ever comparing the surviving pages, and the adopt arm overwrote the joiner's own live-xid outcome pages with the seed's bits (false-visible / false-invisible). Pass the caller's recovery-anchor oldestXid into the check and start the comparison at oldestXid's own 2-bit slot: bits below it are frozen truth that CLOG never consults again and whose on-disk survival is an accident of truncation timing, so they are exempt whether the pages survive or not (neither truncation nor tuple freezing rewrites surviving CLOG bytes, so this can never hide live evidence). Within the comparable range [oldestXid, min(own_next, native_hw)) a missing local page is an anomaly -- pg_xact always covers [oldestXid, nextXid) on a well-formed node -- and now returns UNAVAILABLE (fail-closed 53RB5 at the caller), never CONSISTENT. Sub-byte boundaries at oldestXid are masked at 2-bit precision. Red-first unit coverage: front-truncated divergent clone must DIVERGE (returned CONSISTENT pre-fix), divergence strictly below oldestXid stays CONSISTENT (frozen exemption, mid-byte lead mask), oldestXid at the divergent slot DIVERGES, and a hole inside the comparable range is UNAVAILABLE (the old truth-table leg expecting CONSISTENT for a missing segment encoded exactly the blind spot and now expects UNAVAILABLE). Spec: spec-6.15b-xid-authority-native-era.md
…not inference Nightly t/331 C1 (clean_leave x idle) / C4 (leave_remove x idle) regressed at the r2 P2-1 head (run 28948167577; parent green): the leaver's barrier-tick own-commit latch inferred its commit from epoch>baseline + others-dead bitmap unchanged + scalar dead_generation unchanged. The scalar is monotone, so a third-party transient SUSPECTED->DEAD->ALIVE flap on the leaver's local CSSD view during the leave window advances it forever; the latch then refuses a leave the coordinator ACTUALLY committed and the immediate-escalate arm (or the barrier deadline) aborts it: phase ends ABORTED_ESCALATE -> IDLE, never 'committed', breaking the C1/C4 drains+commits assertions. Reverting to bitmap-only would re-open the r2 P2-1 wedge (rebound mis-latch of a REFUSED leave suppresses escalation forever), so neither inference is usable. Replace inference with direct evidence: latch <=> the durable COMMITTED marker for THIS leave attempt is confirmed. The evidence already reaches the leaver with zero extra IO: the coordinator sends the nonce-bound LEAVE_COMMITTED only after its qvotec ACKs the COMMITTED marker majority-durable (there is no runtime voting-disk read path on the leaver, and none is needed - the LMON tick budget this spec protects is untouched). cl_committed_handler now routes through a pure identity gate (self-addressed + currently leaving + per-attempt nonce + committed epoch past the bound baseline; fail-closed on any mismatch, so a stale confirmation - and through it a stale COMMITTED marker from a previous leave of the same node - can never false-latch) and records the attested committed epoch E before publishing the evidence flag; the barrier tick consumes E instead of re-reading its possibly-stale local epoch view. The latch and the P1-V0.7 exit gate collapse into one step (the evidence IS the durable-truth-source confirmation). Escalation semantics are unchanged: no evidence by the barrier deadline -> the existing cl_escalate path, which is exactly what bounds a refused leave (it never gets a COMMITTED marker). Third-party flaps no longer matter: the coherence observations are now contract inputs the predicate must ignore (pinned by the U3b unit matrix, red-first in the parent commit) and feed only a flap-noise LOG at the latch. Survivor-side coherence sites (drive_drain / staged-ACK / pre-check) are untouched. Unit: U3b matrix flipped green; new U3c pins the evidence identity gate (match / stale-nonce / misrouted / not-leaving / epoch-not-advanced). Local gates: t/331 x4 all green (6/6), t/310 24/24, t/363 18/18, t/274 18/18 (prebump shmem-bit stays green), cluster_unit 158 binaries, cluster_regress 13/13, PG core 219/219, clang-format-18 + comment-headers clean. Spec: spec-2.29a-reconfig-marker-async-lmon-liveness.md
WAIT_BARRIER / WAIT_CLUSTER carried only epoch-coherence guards, so a
survivor whose CSSD detected a coordinator-folded multi-death late kept
announcing REDECLARE_DONE under its stale dead-bitmap hash: peers that
stamped the full set dropped those frames on the composite key and P6 —
which has no timeout by design — wedged permanently (survivor-behind
detection skew). Add a mid-episode guard that aborts to IDLE on a newer
local event whose dead-bitmap hash differs: the next tick re-consumes
the event, re-stamps recovery_event_bitmap_hash, re-runs recovery
against the fuller dead set and re-announces DONE under the new
composite key. Same-set dead_generation drift is absorbed without
episode churn (the event_id keeps its local ABA-scoping role). The
coordinator-behind direction needs no handling: the coordinator's own
later detection bumps the epoch again and the epoch guards abort.
Correct the stamp-site / accounting / hash-helper comments that
overstated the bitmap as quorum-ratified (each node stamps its OWN
accepted event's bitmap; convergence is eventual via the CSSD deadband
plus this guard), and assert a stamped episode hash is never 0 — 0 is
reserved as unstamped in the mark_peer_done drop test, and a JOIN
episode's all-zero bitmap hashes to a fixed nonzero constant.
Unit: survivor-behind growth leg (H({1}) stamped at the folded epoch,
grown {1,2} event arrives -> abort, re-stamp, full-set remaster, DONE
accounting converges) plus a same-set drift no-churn leg; both go red
with the guard disabled.
Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-2, r3-P3a)
…arrier hw_remaster_record_terminal stores the backoff deadline before the terminal result with a release fence between the stores, but the LMON relaunch decider loaded result-then-deadline with no read-side pairing: on a weakly-ordered CPU it could still pair a fresh BLOCKED with a stale zero deadline and skip one backoff wait. Insert pg_read_barrier() between the result and deadline loads (attempts needs no fence — it is written only by the single-writer LMON FSM) and cross-reference the pairing at the write site. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P3b)
The serve-gate unseal (static master no longer CSSD-DEAD -> NORMAL) is heartbeat liveness, not a direct recovery-completion signal. Document the traced proof of why returning NORMAL is nevertheless safe: CSSD — the only heartbeat sender — is spawned by the phase-4 driver at the PM_RUN transition, after the startup process completed the node's crash recovery against shared storage; and even under a stale-ALIVE view a fetch cannot complete against a still-recovering node, because the master-side handler default-denies until the node is an in-quorum MEMBER and only the (equally post-PM_RUN) QVOTEC process can establish the quorum lease after a restart wiped shmem. Deny replies surface as bounded 53R9L, endpoint unavailability as bounded 53R90, and no path falls back to a silent local storage read. Replaces the previous wording that asserted recovery-completion semantics without grounding. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P2-1 prove-safe)
… leg The Amendment v1.2 (R1) serve-gate fix had no test lock: the t/362 two-outcome legs stay green under a reverted gate (their success arm accepts a NORMAL serve of a dead-master page) and no unit links the real phase decision. Add L4h: after the kill, with the dead node never restarted, once the observed survivor's own recovery episode reached IDLE, a FRESH backend on a cold survivor runs a wide pg_class/pg_attribute scan that MUST fail with exactly 53R9L — no success arm, bounded return. An out-of-scope formation never publishes the dead node's materialization proof, so every dead-master page stays RECOVERING while the node is down; the scan spans dozens of catalog pages hashed ~1/4 to the dead node and cold-reads them past the 16MB test pool, and it touches no dead-node-written rows (heap INSERTs only, no DDL), so 53R9L is the only legal outcome. Verified red-first: with the gate reverted the suite fails at exactly this leg while every other leg stays green. The episode-IDLE wait polls the pre-warmed session with the exact query shape it ran pre-kill and compares in Perl: a cast or operator added to the SQL performs fresh syscache lookups that themselves cold-read a dead-master catalog page and trip 53R9L (the very mechanism under test). Also accept the 53R51 write-fence rejection in the pre-existing two-outcome write legs: a transient lease/epoch fence right after the reconfig is an explicit, retryable fail-closed rejection inside the same honest contract. Spec: spec-4.6a-grd-recovery-liveness.md (r3-P1-1 TAP hard leg)
…nt_old_epoch after r3 3-way merge
…hards, 361 into crossnode-b)
sqlrush
force-pushed
the
stage7-p0-integration
branch
from
2acd537 to
bfe00bc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration of PR#26 @ 5b8e8ef + PR#25 @ ea2cdc9 + PR#24 @ d8b1d6d on main e7ef126, for 4-node S1/S2 acceptance testing. NOT for merge (F1 gate closed); draft for fast-gate CI evidence only.
Spec: spec-4.6a-grd-recovery-liveness.md / spec-2.29a-reconfig-marker-async-lmon-liveness.md / spec-6.15b-xid-authority-native-era.md