fix(pm): serve a verify-lock queue whose holds outlast one foreground turn - #12788
Merged
Merged
Conversation
… turn The wait budget cannot be raised: it is spent inside one foreground agent turn, which the harness kills at ~600s, so a larger number buys an unreportable SIGTERM rather than a longer wait. What spans several turns is the slot, and two things stopped it working: - its ageing bound ran from the ORIGINAL arrival, so a caller queued behind two ~19-minute holds lost its place for the offence of the queue being long. The age now runs from when the place was last put down, which is the question the bound was always asking. - it was named only in the exit-99 message, one whole budget after the place it would have kept was lost. A queueing caller is now told at second zero. Exclusion, FIFO order, the cap and the budget are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PfaSTikked61BkcsB5Rn69
This was referenced Aug 27, 2026
os-zhuang
marked this pull request as ready for review
August 27, 2026 19:35
os-zhuang
enabled auto-merge
August 27, 2026 19:35
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #12538
The card's arithmetic is real — a 540s wait budget against holds measured at 1164s — but the budget is not the number that can move, and re-deriving the data changed the shape of the fix. Both of those are stated first, because the PR does something other than what the card's first suggestion asked for.
The route the card led with cannot be taken, and the reason is the cap's stated purpose
OS_VERIFY_LOCK_WAITis clamped down toHARD_CAP_S(effective_budget), so a caller genuinely cannot raise it — the PM assumption holds. What the card missed is why it is clamped: the budget is spent inside one foreground agent turn, and that turn is killed by the harness at ~600s. A budget of 1200s does not buy 1200s of waiting; it buys a SIGTERM at 600s with no VERDICT line, no ledger record and no NOT MEASURED note — the exact silent outcome this file's bash-3.2 block exists to document as the worst failure available here. Raising the cap converts an honestexit 99into an unreportableexit 143.So route 1 is refused, and refused in the file (a comment block on the constant) so the next reader does not re-file it.
The p95 the card asks to raise the budget above does not exist
I read the ledger rather than trusting the card's four hand-correlated anecdotes, as asked.
--reporton this container, 2026-08-27:Zero
queue-timeoutrecords, arrival depth 1 on every row, max hold 344s. Not one contended run of the kind this card is about is in the ledger — because the ledger lives in the container's/tmpand starts empty at every reset, and the card's observations are from the previous day. Filed as #12783.That is load-bearing rather than a footnote: it rules out picking a bigger constant.
HARD_CAP_S * 6would have exactly the standing of the constant it replaced, and would be wrong again the first time three long holds queue instead of two.What actually serves a 19-minute hold, and the two things stopping it
The slot. It already spans foreground turns, and it is the only thing here that can. Two defects kept it from serving the queue the card measured:
SLOT_MAX_AGE_Sis 3 x the cap = 1620s, so a caller queued behind two ~19-minute holds lost its place at 27 minutes — for the offence of the queue ahead of it being long. The bound now runs from when the place was last put down, which is the question it was always asking ("is anybody coming back for this?"). A caller that keeps returning keeps its place however long the queue is; a place nobody returns for expires on the same clock it always did. No constant changed.park_ticketrecords the relinquish instant in the parked record's starttime field — a parked record has no process, so it has no start time to hold, and the label stays last so a label containing spaces still cannot displace a field. A0there (every record written before this) ages from arrival exactly as it did.What the lock still guarantees — nothing here weakens it
Mutual exclusion, FIFO order among entry-point callers,
HARD_CAP_S,DEFAULT_WAIT_S, the ticket queue, the verdict words and the unlocked-mode degradation are all unchanged. No wait was lengthened, shortened or refused: the arrival notice is a disclosure and explicitly does not act on its own prediction, because refusing on an estimate would invert this file's own doctrine (the filter preflight refuses only what is CERTAINLY worthless and fails open in every other direction). A returning slot cannot cut the line — its priority is still exactly its arrival stamp, and the tickets ahead of it can only disappear, so it reaches the head in finite time and is consumed there.The one limit I did not fix is stated in the code rather than papered over: a slot whose owner was SIGKILLed carries no relinquish time and ages from arrival, the shorter of the two windows — the safe direction (a place lost early, never a line-cut that outlives its bound), and transient, since SIGTERM (what the foreground ceiling sends) leaves through the trap.
One repair outside the card's letter, declared
mode_self_testnow unsetsOS_VERIFY_LOCK_SLOT,OS_VERIFY_LOCK_WAITandOS_VERIFY_LOCK_NO_FILTER_CHECK. This is not tidying — I hit it: running--self-testfrom a shell that carriedOS_VERIFY_LOCK_SLOTput every child on the has-a-slot branch, and a new case failed against a script that was behaving correctly. That direction was the lucky one; the same inheritance can silence a case (OS_VERIFY_LOCK_NO_FILTER_CHECK=1skips the check two cases assert). A suite that measures the caller's shell is the instrument family this whole file is about.Verification
Union run and gates on the final commit
7c421832b;git diff HEADempty at the time of both.os-verify-lock.sh --self-test— all 156 cases pass (6 new), quoting its own verdict line under the shared lock:Ablation — the new pins are proved able to fail. Two legs, each with a
traprestore, absolute paths, the mutation confirmed on disk by grep counts of both the injected and the deleted text, and each restore verified byte-identical to the HEAD blob (git hash-objectvsgit rev-parse HEAD:path). No build step is involved — the subject is the script itself, run by path, with nodist/between the edit and the run.✗ a slot put down just now keeps its place however old its arrival is— 155 pass, the expiry case greenannounce_arrivalcall removed✗ and named the slot BEFORE the wait was spent, not only in the verdict+✗ and the arrival line says how many are ahead and what the budget is— 154 passLeg A is the pair that matters: both fixtures have an arrival stamp far past the bound and differ only in the relinquish time, so under the old clock both are dead and the pair cannot pass. The ordering pin in leg B asserts the order (first mention of the slot precedes the VERDICT line), not the presence — asserting presence alone would have passed against the version this fix replaces.
Gates, re-derived for the actual diff with
node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack(9 families, identical to the dispatch list — no additions). Run as one&&chain, socommand-exitcertifies every element:Each gate's own verdict line, not a bare
$?:✓ check-bash32-floor: 22 tracked shell file(s) ... name no bash 4+ construct(load-bearing here — this is a shell file, written to the 3.2 floor:caseguards rather than=~, no arrays, no bash 4 builtins)✓ check-agent-test-spelling: 0 violations — 389 file(s)✓ check-cli-command-ids: 287 command-id literal(s) ... all resolveOK: 20 package(s) read outside themselves, all declared(check:cross-package-test-inputs)✓ check:entry-guard: 170 scripts/ file(s)✓ check:parse-guard: 169 scripts/ file(s)✓ check:pnpm-filter-targets: 140/177 --filter occurrence(s) ... resolveOK: all 109 declared cross-package glob(s) ... are covered(check-ci-filter-parity.mjs)check-nul-bytes: OK (scanned 7118 text file(s) ... no raw ASCII control bytes)OK self-test: 46 live row(s) ... none stale, none missing, none contradicted(bare-root-worklist.mjs --self-test)skip-changeset:scripts/pm/is not a published package, so this PR publishes nothing.Scope
Out of scope: #12528 is the complement of this card on the same file (what the lock does not cover) and is not addressed here — it remains open and untouched.
Filed while working, unassigned, ⛔ not repaired in this PR:
os-verify-lock --reportlabels its arrival-depth column "waiters already ahead" while the recorded value counts the caller itself — an uncontended fleet reads as one-deep #12782 —--reportlabels the arrival-depth column "waiters already ahead" while the recorded value counts the caller itselfos-verify-lockledger lives in the container's /tmp, so--reportcan never answer the question it was built for — "where does hold time go" is scoped to one shift, and says so nowhere #12783 — the ledger lives in the container's/tmp, so--reportcan never answer at fleet scale (this is why the p95 could not be priced)OS_VERIFY_LOCK_SLOTis the only mechanism that serves a queue longer than one foreground turn, and the dispatch prompt never names it — every dev meets it in a failure message #12784 — the dispatch contract never namesOS_VERIFY_LOCK_SLOT; the skills-lane half that triage ruled is not this card's surfaceSession: https://claude.ai/code/session_01PfaSTikked61BkcsB5Rn69
Generated by Claude Code