Worker lifecycle: shutdown leaves jobs stuck PROCESSING and /health hides a dead worker

[NathanTarbert]

Summary

Two defects in apps/worker combine into a failure mode that is both likely and invisible: a
worker can wedge with jobs stuck PROCESSING, and its health endpoint keeps reporting ok
so Railway never restarts it. If anyone ever reports "the worker silently stopped processing
but looks healthy", this is the explanation.

Found during a code review of an unrelated PR; verified against source.

1. Graceful shutdown never happens — rows stick in PROCESSING

Worker.start() registers its own SIGTERM/SIGINT handler, which calls stop()
un-awaited and flips running = false first. So by the time apps/worker/src/index.ts's
shutdown() runs await worker.stop(), that call early-returns — and
prisma.$disconnect() plus process.exit(0) execute while jobs are still in flight.
Those job rows are left in PROCESSING, which then permanently blocks the Scheduler's dedupe
for that job type.

Contributing factors in the same function: shutdown() is not idempotent (a second signal
double-disconnects and double-closes), has no try/finally (a rejection becomes an unhandled
rejection and process.exit never runs), has no drain deadline, and does not await
healthServer.close().

2. /health returns 200 even when the worker is dead

The handler builds worker.healthCheck() and then unconditionally writes 200 with
status: 'ok', ignoring health.running. Railway's healthcheck therefore never restarts a
stopped worker. Compounding it, lastPollTime is set before the DB call in the poll loop,
so it stays fresh even when every poll throws — the one field that could reveal a broken loop
looks healthy.

3. Smaller items in the same file

• parseInt(process.env.PORT ?? process.env.HEALTH_PORT ?? '3003') — PORT='' is not
  nullish, so an empty value skips the fallback and yields NaN; listen(NaN) binds a random
  port instead of failing.
• An injected PORT moves the listener while the Dockerfile's HEALTHCHECK/EXPOSE hardcode
  3005 — a healthy worker then crash-loops.
• req.url === '/health' is an exact match, so /health/ or any query string 404s.
• No 'error' listener on the health server (EADDRINUSE crashes opaquely) and no
  unhandledRejection/uncaughtException handlers; the worker starts consuming jobs even if
  the bind failed.
• JobType.TICKET_CLASSIFY has no handler and no worker.on registration, so such jobs are
  never claimed and sit forever without being dead-lettered.
• No defaultTimeoutMs, so GITHUB_REACTION_POLL, JOB_CLEANUP and TRACKER_SYNC silently
  get 30s.
• The file header claims ESCALATION "posts notification"; the handler only writes a DB row.

Acceptance criteria

• One shutdown owner: either Worker handles signals or the entrypoint does, not both;
  in-flight jobs drain (or are released back to PENDING) before $disconnect()
• shutdown() is idempotent, has a drain deadline, and cannot exit 0 on failure
• /health returns non-200 when running is false; lastPollTime updates only after a
  successful poll
• Invalid PORT/HEALTH_PORT fails loudly instead of binding a random port
• TICKET_CLASSIFY is either implemented or removed from JobType
• Tests cover shutdown-with-in-flight-jobs and the unhealthy /health path (apps/worker
  currently has zero test files, masked by passWithNoTests)

[NathanTarbert]

Independently re-verified during the CR of #150, against main at f59ebe4. Both defects in this issue reproduce exactly as described, and two reviewers found them without knowing this issue existed:

• Worker.start() registering its own SIGTERM/SIGINT handler means apps/worker/src/index.ts's await worker.stop() early-returns on !this.running, so process.exit(0) runs with jobs in flight (packages/outpost/queue/src/worker.ts:121, apps/worker/src/index.ts:141).
• /health returns 200 unconditionally (apps/worker/src/index.ts:108), so a stopped worker still reports healthy.

One addition not in the original write-up: stop() can also resolve while a poll is mid-claim, so rows can be left PROCESSING even on an otherwise orderly shutdown.

Scheduling problems found in the same review are filed separately as #153 (per-type claiming serializes types and oversubscribes maxConcurrency) and #154 (unregistered types are never claimed, so the no-handler path is dead and TICKET_CLASSIFY accumulates silently) — those are distinct from this issue's shutdown/health scope.