Skip to content

Storage migrations, graceful shutdown, and hot reload

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 18 Oct 10:51
Immutable release. Only release title and notes can be modified.
v0.7.0
83b272c

Upgrade notes

@adonisjs/queue now uses @boringnode/queue@0.9.0, coming from 0.6. These releases change how schedules and deduplication ids are stored, and how some job options behave. Read their notes before upgrading:

1. Stop every process running the previous version

Workers, and any process that dispatches jobs or defines schedules. Do not run the old and new versions at the same time.

2. Migrate the storage

Run it before starting the new workers. Until then, a worker on SQL tables of the previous version processes no job, and Redis schedules of the previous version stay invisible.

node ace queue:migrate

The command upgrades the storage of every configured adapter:

  • Redis: moves the schedules to their new keys.
  • Database (Lucid) and Kysely: converts the schedule dates to epoch milliseconds, converts the payload columns to LONGTEXT on MySQL, and replaces the deduplication index with a unique (queue, dedup_id) index.

Every step detects whether it is needed, so the command does nothing on storage that is already up to date, such as the tables of a fresh install. It is safe to run more than once.

Schedule dates of the previous version are stored without a time zone with Lucid on MySQL, and with Kysely on PostgreSQL and MySQL. Pass the time zone the previous version wrote them in. A wrong value shifts every date for good, so the command refuses to guess:

node ace queue:migrate --timezone=Europe/Paris

The command upgrades SQL tables of adapters created with drivers.database() or drivers.kysely(). For an adapter created with knex() or kysely() of @boringnode/queue, it only checks the schedule dates and warns: run migrateScheduleDates(), migrateTextColumns() and addDedupColumns() of its schema service yourself.

Use --adapter to migrate a single adapter, --database-timezone when the MySQL connections of the previous version used another session time zone, and --force to skip the confirmation prompt in production.

3. Update your imports

QueueSchemaService no longer exists. Import the schema services from the driver modules:

import { KnexQueueSchemaService } from '@adonisjs/queue/drivers/knex_adapter'
import { KyselyQueueSchemaService } from '@adonisjs/queue/drivers/kysely_adapter'

4. Check the behavior changes

The ones most likely to surprise you:

  • maxRetries, retry, timeout and failOnTimeout in defaultJobOptions now apply.
  • The sync driver passes payloads through JSON, like the Redis and database drivers. A Date becomes a string, and a class instance a plain object.
  • A job class with an invalid timeout makes the application fail at boot.
  • Defining a schedule again keeps its state: a paused schedule stays paused.
  • A job whose class the worker does not know goes back to the queue, up to worker.unknownJobRetries times.
  • Retention (removeOnComplete / removeOnFail with count or age) applies to the whole queue, not to each job class.

Graceful shutdown

On SIGTERM, AdonisJS terminated the application while the worker was still running jobs. The queue connections closed under these jobs, which stayed active and ran again once recovered as stalled.

queue:work now stops the worker first, waits until every running job is finalized, then terminates the application and closes its Redis and database connections. The process exits on its own.

Hot reload

Jobs now reload during development without restarting the worker.

node ace queue:work --hmr

A changed job runs its new version on its next execution. A change that cannot be hot reloaded, such as a config file, restarts the worker once its running jobs are finalized. hot-hook must be installed, as in the starter kits.

hotReload is enabled automatically when the process runs under hot-hook, including node ace serve --hmr, unless you set it in config/queue.ts.

New flags

  • queue:work --all-queues processes the default queue and every queue declared in config/queue.ts, so a new queue cannot be forgotten on the command line.
  • queue:work --adapter=<name> processes the jobs and schedules of one adapter. Run one worker per adapter that owns queues.
  • queue:scheduler:list, queue:scheduler:remove and queue:scheduler:clear accept --adapter to manage schedules stored on another adapter than the default one.

Configure

  • node ace configure @adonisjs/queue registers the #jobs/* import alias and creates the app/jobs directory. @adonisjs/core 7.6 or later is required.

Bug fixes

  • queue:work exits with status 1 when it cannot start, such as with an unknown --adapter, instead of exiting with status 0 or staying alive.

  • queue:work rejects a --concurrency below 1, which started a worker that never processed a job.

  • Commands that warm up the application, such as node ace codegen, no longer open the queue connections and keep the process alive. (thanks Davi de Carvalho)

  • The migration published by node ace configure with the database driver imported QueueSchemaService, which no longer exists.

  • The config published for the database driver used a connection named primary. It now uses the primary connection of Lucid, whatever its name.

  • @adonisjs/redis 11 is accepted as peer dependency.

Assets 3
Loading