Skip to content

Jobs That Stay Put, Schedules That Survive Deploys

Choose a tag to compare

View all tags
@github-actions github-actions released this 05 Oct 09:24
· 44 commits to main since this release
v0.8.0
bb88316

0.8.0 was long overdue. It fixes a lot of small issues, and it breaks a few things while we still can, before 1.0. Once you upgrade, jobs stop getting lost or running twice, and schedules survive your deploys. Read the upgrade steps first.

Upgrade from 0.7

0.8 changes how every adapter stores schedules. Do not run 0.7 and 0.8 processes at the same time.

  1. Stop every 0.7 process. That includes the workers and any process that dispatches jobs or defines schedules.

  2. On Redis, move the schedules to their new keys, once:

    await QueueManager.init(config)
await QueueManager.use('redis').migrate()

    Until it runs, 0.8 does not see the 0.7 schedules, and defining one of them throws an error that points to migrate().

  3. On Knex and Kysely, convert the schedule dates to epoch milliseconds, once per schedules table. Do it even if you never defined a schedule. Until the table is migrated, a worker processes no job and logs an error that points to migrateScheduleDates() at each attempt.

    // Knex
await new KnexQueueSchemaService(connection).migrateScheduleDates('queue_schedules', {
  timezone: 'Europe/Paris',
})

// Kysely
await new KyselyQueueSchemaService(db, { dialect: 'postgres' }).migrateScheduleDates(
  'queue_schedules',
  { timezone: 'Europe/Paris' }
)

    timezone is the time zone the 0.7 processes wrote dates in. It applies to the dates stored without a time zone: Kysely on PostgreSQL and MySQL, and Knex on MySQL. Get it right, because the migration drops the old columns and a wrong value shifts every date for good. It is the time zone of the process, unless you configured the driver otherwise. With the mysql2 timezone: 'Z' option, pass 'UTC'. On MySQL, if the 0.7 connections used another session time zone, for example by setting time_zone, also pass it as databaseTimeZone, such as '+02:00'. See the README for the details.

  4. On MySQL, convert the text columns if you store payloads larger than 64 KB. MySQL blocks writes to each table while it copies it.

    await schemaService.migrateTextColumns({
  jobsTable: 'queue_jobs',
  schedulesTable: 'queue_schedules',
})

  5. Update your imports. QueueSchemaService is now KnexQueueSchemaService, exported from @boringnode/queue/drivers/knex_adapter. Import QueueInstrumentationConfig from @boringnode/queue/otel. Under drivers/ and contracts/, you can only import the adapter modules and contracts/adapter.

  6. Check the behavior changes in the list below. These are the ones most likely to surprise you:

    • timeout and failOnTimeout in defaultJobOptions now apply, and timeout: 0 means no timeout.
    • Defining a schedule again no longer resumes it, and keeps its next run. A run missed while the application was stopped now runs once when a worker starts, if the schedule is active and has not reached its run limit or end date. 0.7 skipped it.
    • A job whose class the worker does not know goes back to the queue. Set worker.unknownJobRetries: 0 to fail it at once, as before.
    • A job that stalls too many times now fails through failed().
    • Run one Worker per process.

  7. If you wrote a custom adapter, implement the new contract: migrate(), lease tokens, recoverStalledJobs() with maxExceeded, and upsertSchedule() with nextRunAt.

0.8.0 (2026-10-05)

⚠ Breaking changes

Schedules are stored differently

Every adapter changes how it stores schedules, so existing schedules need a one-time migration. On Redis, stop every 0.7 process and run QueueManager.use('redis').migrate() before you start the workers. On Knex and Kysely, run migrateScheduleDates() on each schedules table, even if you never defined a schedule. The upgrade steps above show both. You cannot downgrade to 0.7 after the migration. If you type your Kysely database with QueueScheduleTable, its date columns are now numbers instead of Date.

Defining a schedule again keeps its state

Your application defines its schedules at boot, so every deploy defines them again. This no longer resets them. A paused schedule stays paused, and only schedule.resume() resumes it. The schedule also keeps its run count and its next run, unless you change its timing.

Timeouts

timeout and failOnTimeout in defaultJobOptions now apply, and they take precedence over worker.timeout. timeout: 0 means no timeout. A timeout outside the range a timer accepts makes QueueManager.init() throw. A timed out handler keeps its worker slot until it returns, so worker.stop() waits for it. A handler that ignores this.signal and never returns blocks the stop.

Jobs with an unknown class go back to the queue

During a rolling deploy, a worker can receive a job whose class only the new version knows. Instead of failing the job at once, the worker puts it back in the queue, up to worker.unknownJobRetries times (10 by default). Set it to 0 to keep the old behavior. An invalid value makes new Worker() throw E_CONFIGURATION_ERROR.

One Worker per process

A second worker.start() while another Worker runs in the same process rejects with E_WORKER_ALREADY_RUNNING. Run each Worker in its own process.

Imports and exports

QueueSchemaService is now KnexQueueSchemaService, and you import it from @boringnode/queue/drivers/knex_adapter instead of the package root. Import QueueInstrumentationConfig from @boringnode/queue/otel instead of @boringnode/queue/types. Under ./drivers and ./contracts, only the adapter modules and ./contracts/adapter are exported. Other paths fail with ERR_PACKAGE_PATH_NOT_EXPORTED. errors.E_NO_JOBS_FOUND no longer exists.

Custom adapters

If you wrote your own adapter, it must implement the new contract:

  • migrate(), which does nothing when the adapter has nothing to migrate.
  • Lease tokens. popFrom() and recoverStalledJobs() return a lease token with each job. completeJob(), failJob(), retryJob(), and renewJobs() take JobLease objects and return booleans, so only the execution that holds a job can finalize it.
  • recoverStalledJobs() takes a fourth maxExceeded argument and returns { recovered, exceeded }. It keeps the jobs that stalled too many times active, reassigns up to maxExceeded of them to the calling worker, and returns them, so the worker fails them through failed().
  • upsertSchedule() receives the next run in config.nextRunAt. For an existing schedule, it keeps the status, run count, last run, and creation date, and keeps the next run unless the timing changes.

Features

  • export JobDispatcher and missing public types (014f934)

Performance Improvements

  • redis: claim due schedules through a sorted set index, in O(log N) instead of a scan of every schedule (#14) (a404106)

Bug Fixes

  • accept every OpenTelemetry SDK 2.x instrumentation release (fd254ba)
  • apply timeout and failOnTimeout from defaultJobOptions (1062c7f)
  • fail stalled jobs instead of deleting them (bd6a06f), closes #30
  • keep the status and next run of a schedule defined again (9b140a9)
  • keep the worker slot of a timed out handler until it returns (3782cc5)
  • knex: scope KnexQueueSchemaService to the driver module (#27)
  • make timeout 0 disable the timeout and reject out-of-range timeouts (3cf3647)
  • only export the adapter modules under ./drivers (08d35a6)
  • only the current lease holder can finalize a job (9c902f5)
  • put jobs with an unknown class back in the queue (0022f0e)
  • redis: remove dedup keys when the connection has a key prefix (9116c92)
  • redis: store schedule hashes under schedules::data (d560a4a), references #14
  • reject a second Worker started in the same process (2a99ed1)
  • store an undefined schedule payload as an empty object (13ae803), references #14
  • store payloads and errors as LONGTEXT on MySQL (5437b71)
  • store SQL schedule dates as epoch milliseconds (18b1b1b)
  • tracing: drop the channel name type argument (#28) (6008ed6)
  • types: drop the OpenTelemetry re-export from the types barrel (#29)
  • worker: interrupt completion delay on stop (2ebc06e)
  • worker: log failed cycles through the logger (cfaeb44)
  • worker: queue job completions (0cc98f2)
  • worker: reduce idle acquisition amplification (96c0a89)

Code Refactoring

  • remove the unused E_NO_JOBS_FOUND error (2562f52)
Assets 2
Loading