No Lost Jobs, No Duplicates, No Stuck Queues

0.9.0 is a reliability release. We ran the published package as a black box against real Redis, PostgreSQL, MySQL 8.4 and SQLite, with real worker and dispatcher processes, crashes, outages and load, and fixed everything that lost a job, ran one twice, or stopped a queue. Most fixes need nothing from you. A few change behavior you may rely on: read the upgrade steps first.

Upgrade from 0.8

1. On Knex and Kysely, run addDedupColumns() again on each jobs table, once. It replaces the partial dedup index with a unique index on (queue, dedup_id). Without it, sequential dispatches are still deduplicated, but concurrent ones are not. Stop dispatching jobs with .dedup() while it runs.

   // Knex
   await new KnexQueueSchemaService(connection).addDedupColumns('queue_jobs')
   
   // Kysely
   await new KyselyQueueSchemaService(db, { dialect: 'postgres' }).addDedupColumns('queue_jobs')

2. Let your worker process exit. Stopping a Worker does not close the Adapter connections. After await worker.start(), call await QueueManager.destroy(), then close any connection your application owns (an ioredis client, a Knex or Kysely instance):

   await worker.start(['default'])
   await QueueManager.destroy()
   await db.destroy() // only if the application created the connection

   worker.start() now resolves only once every running job is finalized, so this is safe on SIGTERM.

3. Check the behavior changes below. These are the ones most likely to surprise you:

   • maxRetries and retry in defaultJobOptions now apply.
   • QueueManager.init() and QueueManager.destroy() throw while a Worker runs in the same process.
   • The Fake and Sync adapters pass payloads through JSON, like real adapters. Tests comparing payloads to a Date, a class instance or a BigInt need updating.
   • A job class with an invalid timeout makes QueueManager.init() and Locator.register() throw.
   • Retention (removeOnComplete / removeOnFail with count or age) applies to the whole queue, not to each job class. Give job classes with different retention their own queue.

0.9.0 (2026-10-18)

⚠ Breaking changes

Retries from defaultJobOptions apply

maxRetries and retry were typed and accepted in defaultJobOptions, globally and per queue, but ignored. They now apply. Each of maxRetries and backoff comes from the first place that sets it: the job's options, the queue's defaultJobOptions, the queue's retry, the global defaultJobOptions, then the global retry. Remove these settings to keep the previous behavior.

QueueManager.init() and destroy() refuse to run under a Worker

Called while a Worker ran in the same process, QueueManager.init() and QueueManager.destroy() closed the Worker's Adapter: the Worker failed every cycle with Connection is closed and never ran another job. From worker.start() until the Worker has stopped, they now throw E_INIT_WHILE_WORKER_RUNNING and E_DESTROY_WHILE_WORKER_RUNNING, and change nothing. The Worker initializes QueueManager with its own config, so a worker process can dispatch jobs without calling init().

Fake and Sync adapters use JSON payloads

Both adapters kept the payload by reference. A test could pass and the code fail in production: a Date stayed a Date, a class instance kept its class, a BigInt dispatched fine, and a payload mutated after dispatch changed the recorded job. On Sync, a handler's mutations also reached the caller, and a retry received the payload as the previous attempt left it. Both now store a JSON copy, as Redis and SQL do, and each Sync attempt gets a fresh copy. A payload JSON cannot encode makes the dispatch reject, and a dispatchMany() batch holding one runs none of its jobs.

Invalid job timeouts fail at boot

A job class whose own timeout is invalid (negative, fractional, or above 2147483647 ms, about 24.8 days, such as '30d') started fine, then every dispatched job failed with E_INVALID_TIMEOUT without running, without calling failed(), and logged only at debug. Registering it now throws E_INVALID_TIMEOUT naming the job (Invalid timeout "30d" of job "SlowJob"), both from Locator.register() and when QueueManager.init() loads locations, as init() already did for worker.timeout and defaultJobOptions. With hotReload, jobs are re-imported on each run, so the check still happens when the job runs. 0 still means no timeout.

Batches and dedup in your own transaction

On an adapter created on an application transaction, dispatchMany() and .dedup() dispatches now run directly in that transaction, and a failure propagates the driver error unchanged. A missing dedup column still raises the addDedupColumns() hint, with the driver error as its cause. On Kysely, .dedup() used to throw there; it now works. On Knex, it ran in a savepoint and retried a conflict; when two dispatches race for the same new dedup id, the loser now gets the unique violation and your application retries its transaction.

Roll the transaction back after such a failure: PostgreSQL refuses to commit it anyway, while on MySQL and SQLite a commit keeps the rows written before the failure. Outside your own transaction, a batch is still all or nothing and a dedup conflict is still retried for you.

Bug fixes

Jobs no longer run twice after a shutdown

await worker.start() resolved when the first running job finished after SIGTERM, SIGINT or worker.stop(), while other jobs were still running. Cleanup after it closed the connections under those jobs: they stayed active and ran again once recovered as stalled, on every deploy. start() now resolves once every job is finalized.

Failed finalizations are retried and logged

When completeJob(), failJob() or retryJob() rejected, for example on a MySQL deadlock with another worker's lease renewal, nothing was logged and the job ran again later. The Worker now retries a finalization up to 3 times on a deadlock (MySQL ER_LOCK_DEADLOCK, PostgreSQL 40P01 / 40001), and logs at the error level a finalization that still fails, an error thrown by a failed() hook, and a job that fails to initialize. A failed lease renewal is logged at warn.

MySQL: retention no longer causes duplicates

With count or age retention, every finalization pruned its queue with a DELETE that range-locked the status index. Under load, InnoDB aborted the update that completed a job, which then ran again: dozens of duplicates per 20,000 jobs. Pruning now reads the jobs to remove, then deletes them by primary key. In our load test, duplicates went to zero and throughput with retention doubled. A pruning that still fails is retried on a deadlock, otherwise logged, and no longer fails the finalization. On your own transaction, it rejects so you know the transaction failed.

MySQL: no more pauses on delayed job promotion

Two workers promoting delayed jobs at once could deadlock: popFrom() rejected and the worker paused 5 seconds. A deadlock there now just postpones the promotion to the next poll.

SQL: a dedup id belongs to a single job

A .dedup() dispatch racing with a worker acquiring the job that owned the id could enqueue a second job, or return the id of a job that was never stored. A unique index on (queue, dedup_id) now lets a single job own a dedup id whatever its status, on PostgreSQL, MySQL and SQLite. Run addDedupColumns() to create it (upgrade step 1).

SQLite: several processes work

Claims, stalled recovery, schedule claims and .dedup() dispatches started with a deferred BEGIN, so a write from another process made them fail at once with SQLITE_BUSY instead of waiting for the busy timeout. A worker next to a dispatcher process paused 5 seconds at each collision, and concurrent .dedup() dispatches rejected. These transactions now take the write lock first. In our tests, the p50 pickup latency of a worker next to a dispatcher went from about 10 seconds to about 20 ms. The README documents how to run SQLite with several processes.

SQL: large batches

dispatchMany() failed and inserted nothing above 500 jobs on Knex with SQLite, 6,553 jobs on SQLite, and 13,107 jobs on PostgreSQL, with the whole SQL statement and its payloads in the error. Batches are now inserted in chunks of 500 within one transaction.

Redis: one payload can no longer stop a queue

Lua's JSON decoder rejects some JSON that JSON.parse accepts, such as a lone surrogate ('\ud800', for example from a string truncated in the middle of an emoji) or more than 1000 levels of nesting. One such payload made every popFrom() of its queue fail, and a retried one ended in no state. Scripts now fall back to defaults for such a job (priority 5) and read everything before their first write.

Redis: cron schedules survive a crashed claim

If a process died between claiming a cron schedule and writing its next run, the schedule left the due index and never ran again until it was defined again. An unfinalized claim now stays indexed and is finalized by the next claimer after a 10 second lease.

Cron expressions that never match fail at creation

An expression that parses but never matches, such as 0 0 31 4,6 * (April and June have no 31st), made run() reject with the parser's Invalid expression, loop limit exceeded instead of E_INVALID_CRON_EXPRESSION. It now throws E_INVALID_CRON_EXPRESSION.

Logs go to your logger

• Errors of the Redis connection the adapter creates go to the configured logger, instead of ioredis' Unhandled error event on stderr. A connection you pass in keeps your own listeners.
• A job file that fails to import is logged at error with its path, and no longer followed by a misleading No jobs found warning.

Smaller fixes

• A job whose class the worker does not know, put back in the queue, is traced as a retry (status: 'retrying', OK span with a messaging.retry event), not as a failure.
• The Sync adapter cancels pending delayed jobs on destroy(), so they no longer run after it or keep the process alive.
• cron-parser 5.10: next runs are correct across daylight saving jumps of more than one hour, a day-of-week step no longer adds Sunday, and an occurrence exactly on the to date is no longer skipped.

Compatibility

• ioredis 6 is supported (peer range ^5.0.0 || ^6.0.0). The test suite now runs against ioredis 6.

New exports

• AdapterFactoryContext (@boringnode/queue/types): adapter factories receive an optional { logger }.
• RegisterFromGlobOptions (@boringnode/queue/types): the options of Locator.registerFromGlob(), which accepts a logger.
• logger option on new KnexAdapter() and new KyselyAdapter(); knex() and kysely() pass the QueueManager logger.
• Errors E_INIT_WHILE_WORKER_RUNNING and E_DESTROY_WHILE_WORKER_RUNNING.

Documentation

• Retention prunes per queue and status, not per job class.
• Shutdown: close the connections your application owns after QueueManager.destroy().
• Kysely with pg: attach an error listener to each pool client, or an outage crashes the worker.
• exponentialBackoff has ±25% jitter by default; every() counts from the claim time.