Skip to content

fix(platform): claim message slots until a deadline, not a fixed count - #3267

Merged
larryro merged 3 commits into
mainfrom
fix/message-slot-claim-deadline
Sep 6, 2026
Merged

fix(platform): claim message slots until a deadline, not a fixed count#3267
larryro merged 3 commits into
mainfrom
fix/message-slot-claim-deadline

Conversation

@larryro

@larryro larryro commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Defect

appendMessageRow (domains/chat/store.ts) and saveMessage (domains/threads/store.ts) claim max("order") + 1 with INSERT … ON CONFLICT (thread_id, "order", step_order) DO NOTHING and re-claim after a lost race — but only MESSAGE_SLOT_ATTEMPTS = 8 times (#3197). Under READ COMMITTED every round has exactly one winner among the appenders racing for a thread, so a burst of N appenders needs up to N rounds for its unluckiest member. Any burst larger than eight therefore failed a legitimate append with message insert failed: no free slot after 8 attempts — automation fan-out or several agents replying on one thread hits that in production.

The integration lane messages: concurrent appends each take their own slot fires twelve concurrent appends and went red on roughly one proof of main in eight (appends=12, failed=1), most recently on c70872f. It had been read as host-load noise.

Fix

One shared helper, claimMessageSlot in domains/threads/store.ts, runs the claim until it lands a row, bounded by a wall-clock deadline (MESSAGE_SLOT_CLAIM_DEADLINE_MS = 10 s) instead of a count — a count is defeated by any burst larger than itself. Rounds pause with a jittered exponential backoff (1 ms → 50 ms cap) so the losers of one round do not re-collide in lockstep. Clock and sleep are injectable (SlotClaimOptions, an optional trailing parameter on both appenders; production callers are untouched). Under SERIALIZABLE the conflict still surfaces as a 40001 and transactSerializable reruns the transaction, as before. Nothing else is written while a claim is pending.

Tests

  • chat/append-message.test.ts, threads/store.test.ts: the first-try and re-claim shapes stay; the exhaustion test now spends the budget through an injected clock (deterministic, names … within 10000 ms (3 attempts)); a new case proves a 40-loss burst still lands (41 inserts) — larger than any fixed count.
  • Integration lane: 32 concurrent appends (deliberately above the old cap), all must land on distinct contiguous slots.

Proof

  • Real-Postgres integration run on this branch: 525/525 checks across 138/138 lanes — the first fully green proof of the campaign (main at c70872f was 524/525, this lane the only red). The lane: appends=32, failed=0, rows=32, distinctOrders=32, contiguous=true.
  • vitest --project server: 589 files / 6504 tests passed.
  • tsc --noEmit, oxlint --type-aware, knip:check, oxfmt clean.
  • Second integration run on the committed tree (3fb9e90, clean): 525/525 across 138/138 lanes again; the lane appends=32, failed=0, rows=32, distinctOrders=32, contiguous=true. (The first run had started before a last cosmetic edit to the backoff base — 2 ms → 1 ms first pause, matching its comment.)

Review follow-ups (adversarial review: approve, advisory only) — e344a02

  • Backoff cap 50 ms → 20 ms: the unluckiest member of a 32-way burst now waits well under a second in total (was ~1.3 s), lockstep still broken.
  • Both suites lock the SERIALIZABLE contract: an error thrown by the claim (a 40001) propagates unchanged after exactly one insert — the claim neither retries nor swallows it, so transactSerializable keeps its rerun.
  • Options parameter renamed claimslot (it holds SlotClaimOptions); the constant's comment now says "at least one winner per round, exactly one in lockstep"; the test header says "while the deadline remains".
  • Not in this PR (reviewer's note, priced): the task-comment lane (tasks/comments.ts via transactSerializable) still carries withRetry's 5-attempt bound for its 40001s; if a human comment burst on one task ever exhausts it, markRetryQueueKey(error, threadId) queues the retries behind the per-thread advisory lock — a separate change with its own proof.
  • Integration proof on this head (e344a02, clean tree): 525/525 across 138/138 lanes; the lane appends=32, failed=0, rows=32, distinctOrders=32, contiguous=true — the third green proof of this branch.

Merged origin/main (4df487e) — e450a1a

Plain merge commit, no conflicts, so the E2E workflow runs under the Bun 1.4.2 pin that #3265 landed (the earlier docs-job reds on this branch were that pin's exit stall, twice, both green on retry). Platform tsc --noEmit and both appender suites (10/10) green on the merged tree.

`appendMessageRow` and `saveMessage` claim `max(order)+1` with `INSERT …
ON CONFLICT DO NOTHING` and re-claim after a lost race, but the loop gave
up after MESSAGE_SLOT_ATTEMPTS = 8 rounds. Under READ COMMITTED every
round has exactly one winner among the appenders racing for a thread, so
a burst of N appenders needs up to N rounds for its unluckiest member: any
burst larger than eight failed a legitimate append with "no free slot
after 8 attempts". The integration lane fires twelve and went red on
roughly one proof in eight — read as host-load noise until now.

One shared claim helper now runs the claim until it lands a row, bounded
by a wall-clock deadline (10 s) instead of a count, with a jittered
exponential pause between rounds so the losers of one round do not
re-collide in lockstep. Clock and sleep are injectable, so the unit tests
exhaust the budget deterministically and prove a 40-loss burst still
lands; the integration lane fires 32 appends, deliberately above the old
cap.
Review follow-ups on the deadline-bounded slot claim: the backoff now caps
at 20 ms instead of 50 ms, so the last member of a 32-way burst waits well
under a second in total while the lockstep is still broken; the options
parameter is named for what it holds (`slot`, a SlotClaimOptions); the
comment states the bound as "at least one winner per round, exactly one
in lockstep"; and both appenders' suites lock the contract the SERIALIZABLE
callers rely on — an error thrown by the claim (a 40001) propagates
unchanged after exactly one insert, never retried or swallowed.
@larryro
larryro merged commit 56ec424 into main Sep 6, 2026
48 checks passed
@larryro
larryro deleted the fix/message-slot-claim-deadline branch September 6, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant