Skip to content

fix(sweeper): Phase 3H — recover from BLPOP-to-claim race#13

Merged
smallchungus merged 3 commits into
mainfrom
phase-3h-stale-queued
Apr 18, 2026
Merged

fix(sweeper): Phase 3H — recover from BLPOP-to-claim race#13
smallchungus merged 3 commits into
mainfrom
phase-3h-stale-queued

Conversation

@smallchungus

Copy link
Copy Markdown
Owner

Closes a real correctness gap.

Before this, a worker that crashed between BLPOP (job ID gone from Redis) and ClaimJob (PG row not yet flipped to running) left the job stuck — status=queued with no last_error, invisible to the existing sweeper paths (reviveOrphans wants status=running; promoteDelayed wants last_error set).

  • store.ListStaleQueuedJobs: SELECT WHERE status=queued AND last_error IS NULL AND created_at < now() - threshold
  • store.PoolForTest exposes the pool for tests that need raw SQL (backdating created_at to simulate stale rows)
  • sweeper.SweepOnce now also calls repushStale, which LPUSHes any stale-queued job IDs back onto their stage queue. Duplicate pushes are safe — ClaimJob is atomic; a worker that picks up an already-claimed job ID gets ErrJobNotClaimable and moves on
  • cmd/sweeper reads STALE_QUEUED_THRESHOLD_SEC env var (default 60)
  • .env.example documents the new var

3 new tests (1 store + 2 sweeper). All existing tests still green. Loadtest still passes (1643 jobs/s).

@smallchungus
smallchungus merged commit bf02295 into main Apr 18, 2026
@smallchungus
smallchungus deleted the phase-3h-stale-queued branch April 18, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant