Quarantine directories (<manifest>.lock.stale-<claimed_at_ms>-<nonce>) are never removed. One per crash-eviction, next to the manifest, for the life of the deployment. Found by the openai-auth seat probing three successive evictions against PR #33.
Their persistence is the ABA mechanism, not litter. A delayed racer holding a stale observation renames against that name; the occupied directory is what makes its rename fail. Deleting one immediately after eviction reintroduces the race PR #33 fixes:
- A and B both observe stale owner S0
- A renames lock → stale-S0, verifies, deletes stale-S0, re-mkdirs, publishes A1
- B (delayed) renames lock → stale-S0 and succeeds — moving the live winner's lock into the freed slot
So cleanup must only reclaim directories no in-flight racer can still target.
Safe reclaim rule
A racer can only attempt the rename inside its own claim window. Its read of S0 precedes the quarantine, so its last possible attempt is bounded by quarantine_created + claimDeadline. Reclaim age must therefore be:
max(ttlMs, maxClaimDeadline) + margin
measured from the directory's own creation time (mtime is legitimate here — it is a lower bound on "no racer from that era survives", not a liveness judgement), NOT from the claimed_at_ms in the name, which is older by the owner's lifetime and would reclaim too early.
The margin is load-bearing — write its reason at the reclaim site
claimDeadline bounds when the last rename is issued, not when it lands. A loop that checks the deadline at TTL − 1ms and issues the rename has an in-flight syscall completing at TTL + rename latency. On a loaded host that is tens to hundreds of milliseconds (an event loop on this machine missed a 12s bind timeout at load 3). The margin covers an in-flight rename issued inside the window. It is not a fudge factor; a comment saying only "margin" gets tuned to zero.
Two contract items this depends on, currently undeclared
- Nonce entropy is contract: ≥16 CSPRNG bytes. The collision guarantee (same millisecond AND same nonce) rests on it; a counter, pid+timestamp, or short token turns a vanishing collision into a routine one.
claimDeadline <= ttlMs: measured at ratio 1.00 across three TTLs, but nowhere stated. Loosening the deadline without loosening the reclaim age reintroduces ABA through a knob nobody connected to it.
Both are constraints on the writer, and all three tenants vendor the same module — so the writer is always a future version of this code, never a tenant's. They are local invariants testable in this repo, not a cross-tenant social contract. Worth stating that way in the source.
Also
The directories contain the dead owner's record (tenant, pid, timestamps) — an undeclared crash audit trail in a shared config directory. No credential material, but state nobody asked for, growing without bound.
Tenant pins written against PR #33 assert persistence immediately after eviction with no intervening claim, so a reclaim rule that only touches directories older than the bound above cannot invalidate them.
Quarantine directories (
<manifest>.lock.stale-<claimed_at_ms>-<nonce>) are never removed. One per crash-eviction, next to the manifest, for the life of the deployment. Found by the openai-auth seat probing three successive evictions against PR #33.Their persistence is the ABA mechanism, not litter. A delayed racer holding a stale observation renames against that name; the occupied directory is what makes its rename fail. Deleting one immediately after eviction reintroduces the race PR #33 fixes:
So cleanup must only reclaim directories no in-flight racer can still target.
Safe reclaim rule
A racer can only attempt the rename inside its own claim window. Its read of S0 precedes the quarantine, so its last possible attempt is bounded by
quarantine_created + claimDeadline. Reclaim age must therefore be:measured from the directory's own creation time (mtime is legitimate here — it is a lower bound on "no racer from that era survives", not a liveness judgement), NOT from the
claimed_at_msin the name, which is older by the owner's lifetime and would reclaim too early.The margin is load-bearing — write its reason at the reclaim site
claimDeadlinebounds when the last rename is issued, not when it lands. A loop that checks the deadline atTTL − 1msand issues the rename has an in-flight syscall completing atTTL + rename latency. On a loaded host that is tens to hundreds of milliseconds (an event loop on this machine missed a 12s bind timeout at load 3). The margin covers an in-flight rename issued inside the window. It is not a fudge factor; a comment saying only "margin" gets tuned to zero.Two contract items this depends on, currently undeclared
claimDeadline <= ttlMs: measured at ratio 1.00 across three TTLs, but nowhere stated. Loosening the deadline without loosening the reclaim age reintroduces ABA through a knob nobody connected to it.Both are constraints on the writer, and all three tenants vendor the same module — so the writer is always a future version of this code, never a tenant's. They are local invariants testable in this repo, not a cross-tenant social contract. Worth stating that way in the source.
Also
The directories contain the dead owner's record (tenant, pid, timestamps) — an undeclared crash audit trail in a shared config directory. No credential material, but state nobody asked for, growing without bound.
Tenant pins written against PR #33 assert persistence immediately after eviction with no intervening claim, so a reclaim rule that only touches directories older than the bound above cannot invalidate them.