Skip to content

Add seeded concurrency race harness for lease/reservation stores#148

Draft
bpapillon wants to merge 2 commits into
mainfrom
bpapillon/sch-6651-schematic-node-concurrency-race-harness-for-leasereservation
Draft

Add seeded concurrency race harness for lease/reservation stores#148
bpapillon wants to merge 2 commits into
mainfrom
bpapillon/sch-6651-schematic-node-concurrency-race-harness-for-leasereservation

Conversation

@bpapillon

Copy link
Copy Markdown
Contributor

The conformance suite (#147) pins sequential semantics with deterministic vectors and leaves the truly concurrent interleavings as prose invariants. This adds a seeded randomized race harness (tests/races/) that exercises exactly those gaps, against both store backends (in-memory and Redis-via-fake, like the existing unit suites) and the CreditLeaseManager over a scripted wire server with simulated latency.

Races covered (one scenario each, plus mixed chaos):

  • concurrent acquires (replace) on one slot — keep-first, at most one written
  • extend racing expiry and successor installs — pinned extends never credit a successor; expiry/granted of the surviving lease must equal exactly install + its own pinned extends
  • settle (consume) racing the sweeper — exactly-once claims; exact conservation (granted == remaining + open holds + consumed + crash-leaked) catches double and lost refunds
  • successor replace racing try_reserve — plus a clock-fenced probe proving no debit ever lands on an expired lease
  • manager acquire/extend single-flight over a shared store (two "pods", idempotent scripted server) — the installed lease is never released out from under siblings; local granted never exceeds the server total
  • mixed chaos across slots, including drops and simulated window-1 crashes (bounded-leak accounting)

Each run is deterministic per seed (seeded PRNG drives ops, amounts, virtual-clock advances, and the microtask/setImmediate yields that vary interleaving; ids are counters). A failure prints the seed; RACE_SEED=<n> replays the exact schedule; RACE_ITERATIONS=<n> scales the batch (default 40 per scenario/backend, ~1s total in CI).

Found defect (fixed here, first commit): at seed 1061 (both backends) the harness caught a real race — the idempotent server hands a racing acquire the same active lease back, and if that stale response lands after the local row's expiry (e.g. a concurrent extend pushed the server-side expiry forward), replace rewrote the "expired" slot: localRemainingCredits reset to the full grant, erasing debits whose reservations were still open (seed-1061 trace: install lse_srv_2 → reserve 40 + 7.75 → expiry passes → stale replace(lse_srv_2) resets 452.25 → 500, minting 40 held credits). Fix: a same-id replace over an expired row reconciles like an extend (granted-to-total, expiry only forward, balance untouched) and reports "kept"; the manager's redundant-release logic is unaffected. Sequential regression tests added for both stores and the real-Redis integration suite (Lua verified against redis:7). Note for #147: SPEC.md's replace prose ("only if the slot is empty or the existing lease is expired") should gain the same-id carve-out; no existing vector pins the old behavior.

Local seed batches: 10,000 seeds per scenario/backend (120k runs, ~140s) clean after the fix; the harness was also validated by mutation testing (8 hand-injected store/manager bugs — dropped expiry guards, dropped refund/extend pins, clobbering replace, non-exactly-once consume, non-converging extend, releasing the installed lease — all caught with the failing seed reported).

Verification limits: the Redis store runs against the shared fake-redis (synchronous Lua emulation), so real-Redis-only interleavings (e.g. gaps inside add()'s index writes) are out of scope here, as is server-clock skew — both remain covered only by the opt-in integration suite.

bpapillon added 2 commits July 5, 2026 16:28
…alance

The idempotent server hands a racing acquire the same active lease back. If
that stale response lands after the local row's expiry (e.g. a concurrent
extend pushed the server-side expiry forward), replace() saw an "expired"
slot and rewrote it — resetting localRemainingCredits to the full grant and
erasing debits whose reservations are still open, minting local credits.

Found by the seeded race harness (seed 1061, both backends): replace with
the SAME leaseId over an expired row now reconciles like an extend
(granted-to-total, expiry only forward, balance untouched) and reports
"kept", so the manager's redundant-release logic is unaffected.
…on stores

The conformance suite pins sequential semantics; the truly concurrent
interleavings (single-flight, extend-vs-expiry, release-vs-sweeper,
replace-vs-reserve) were prose invariants only. Per seed, N workers run
randomized op sequences over shared slots with seeded micro-yields varying
the event-loop interleaving, against both the in-memory and (fake) Redis
stores plus the lease manager over a scripted wire server; at quiescence the
prose invariants are asserted (conservation, bounded leak, expiry
monotonicity, pinned refunds/extends, exactly-once claims, no expired-balance
serves via a clock-fenced probe).

Deterministic per seed: RACE_SEED replays a failing schedule exactly;
RACE_ITERATIONS scales the batch (default 40 per scenario/backend).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant