Skip to content

fix(selfhost): catch unhandled pump() rejections in sqlite-queue/pg-queue #2498

Description

@JSONbored

Parent: #1936

Problem

kickOne()/kickAll() in src/selfhost/sqlite-queue.ts fire pump() via bare void pump() with no .catch(), and pump()'s own try/finally only decrements the active counter — it does not catch, so any exception from claimNext() or reclaimExpiredProcessingJobs() (both run OUTSIDE processOne's own try block, e.g. a raw driver .query() failure) becomes an unhandled promise rejection. src/selfhost/pg-queue.ts has the identical void pump() pattern (more realistic there — Postgres connection drops/lock timeouts are more common than SQLite errors).

src/server.ts only registers process.on('unhandledRejection', ...) when SENTRY_DSN is set. In the (common, since Sentry is opt-in) case where SENTRY_DSN is unset, Node's default behavior for an unhandled rejection is to crash the process. A single transient DB error surfacing from claimNext()/reclaimExpiredProcessingJobs() would silently kill the entire self-host server with no metric bump, no Sentry event, and no /ready degradation warning beforehand — exactly the "quietly stops reviewing PRs" failure mode.

Fix

Wrap the body of pump() in a try/catch (or add .catch(error => { captureError(error, {kind:"queue_pump_crash"}); console.error(...); }) at each void pump() call site) so a claimNext()/reclaim failure is recorded and logged instead of propagating as an unhandled rejection, independent of whether Sentry is configured. Apply the same fix to both sqlite-queue.ts and pg-queue.ts.

Verification

  • Confirm current pump()/kickOne/kickAll implementation against both files (line numbers may have shifted).
  • A test that makes claimNext() throw and confirms the process does NOT crash (no unhandled rejection), and the failure is logged/recorded.
  • Repeat for reclaimExpiredProcessingJobs().

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions