feat(nuq): RabbitMQ support for job finish events and waiting

[mogery]

Summary by cubic

Adds RabbitMQ-based job finish notifications and waiting to NuQ, with a per-job “listenable” flag and per-pod channels. This fulfills ENG-3630 by letting the API wait for job completion over RabbitMQ instead of Postgres.

• New Features

  • Uses RabbitMQ when NUQ_RABBITMQ_URL is set; falls back to Postgres LISTEN/NOTIFY otherwise.
  • Per-pod listen channels via NUQ_POD_NAME; jobs store listen_channel_id in the DB.
  • addScrapeJob/addScrapeJobs now accept a listenable flag; controllers set it to true for jobs awaited via waitForJob.
  • Worker sends “completed”/“failed” events to RabbitMQ and consumes/acks them.

• Migration

  • Run the SQL change adding listen_channel_id to nuq.queue_scrape.
  • Set NUQ_RABBITMQ_URL to enable RabbitMQ; optionally set NUQ_POD_NAME for unique listen channels.
  • New deps: amqplib and @types/amqplib.

[cubic-dev-ai]

No issues found across 16 files

[mogery]

@cubic-dev-ai Is there any pattern in this that could break existing environments without a NUQ_RABBITMQ_URL configured?

[cubic-dev-ai]

@cubic-dev-ai Is there any pattern in this that could break existing environments without a NUQ_RABBITMQ_URL configured?

@mogery Yes — there are a couple of patterns that can break non-RabbitMQ environments if you don't prepare them:

1. Database schema change (critical)

   • The code now INSERTs and RETURNs listen_channel_id (INSERT ... listen_channel_id and RETURNING listen_channel_id).
   • If the DB migration adding the listen_channel_id column is not applied, those queries will fail (syntax/error on unknown column).
   • Mitigation: run the migration (or apply the schema change) before/when deploying this code.

2. Top-level dependency import

   • amqplib is imported at module load time (import amqp from "amqplib").
   • If the runtime image/container doesn't have the new dependency installed, the process will crash on startup even if NUQ_RABBITMQ_URL is unset.
   • Mitigation: ensure the code is built with the updated package.json (amqplib in deps) and the runtime contains the new deps.

3. Behavioral surface is otherwise backwards-compatible

   • New function parameters (listenable) are appended with defaults so existing callers keep their behavior.
   • nuqWaitMode logic still defaults to previous behavior unless NUQ_RABBITMQ_URL or NUQ_WAIT_MODE are set — so without NUQ_RABBITMQ_URL, wait mode is unchanged.
   • The added listen_channel_id is set to NULL unless a job is created with listenable = true, so DB contents for non-listenable jobs remain similar.

Recommended deployment order for zero risk:

• Apply DB migration (add listen_channel_id).
• Build/publish images with updated deps.
• Roll out the new code.

If you follow that, non-RabbitMQ environments should keep working.