que-scheduler is an extension to Que that adds support for scheduling items using a cron style configuration file. It works by running as a que job itself, determining what needs to be run, enqueueing those jobs, then enqueueing itself to check again later.
To install, add the gem to your Gemfile:
Specify a schedule in a yml file (see below). The default location that que-scheduler will look for it is
config/que_schedule.yml. They are essentially the same as resque-scheduler files, but with additional features.
Add a migration to start the job scheduler and prepare the audit table.
class CreateQueSchedulerSchema < ActiveRecord::Migration def change Que::Scheduler::Migrations.migrate!(version: 4) end end
The schedule file is a list of que job classes with arguments and a schedule frequency (in crontab syntax). The format is similar to the resque-scheduler format, though priorities must be supplied as integers, and job classes must be migrated from Resque to Que. Cron syntax can be anything understood by fugit.
It has one additional feature,
schedule_type: every_event. This is set on a job that must be run for every
single matching cron time that goes by, even if the system is offline over more than one match.
To better process these
every_event jobs, they are always enqueued with the first
argument being the time that they were supposed to be processed.
CancelAbandonedOrders: cron: "*/5 * * * *" # Specify the job_class, using any name for the key queue_documents_for_indexing: cron: "0 0 * * *" class: QueueDocuments # Specify the job queue ReportOrders: cron: "0 0 * * *" queue: reporting # Specify the job priority using Que's number system BatchOrders: cron: "0 0 * * *" priority: 25 # Specify job arguments SendOrders: cron: "0 0 * * *" args: ['open'] # Use simpler cron syntax SendBilling: cron: "@daily" # Use timezone cron syntax SendCoupons: cron: "0 7 * * * America/Los_Angeles" # Altogether now all_options_job: cron: "0 0 * * *" class: QueueDocuments queue: reporting priority: 25 args: ['open'] # Ensure you never miss a job, even after downtime DailyBatchReport: cron: "0 3 * * *" # This job will be run every day. If workers are offline for several days, then the backlog # will all be scheduled when they are restored, each with that event's timestamp as the first arg. schedule_type: every_event
A job can have a
schedule_type assigned to it. Valid values are:
default- This job will be scheduled in a manner closer to resque-scheduler. If multiple cron times go by during an extended period of downtime (eg a long maintenance window) then only one job will be enqueued when the system starts back up. Multiple missed events are coalesced. This mimics the way resque-scheduler would perform if it were taken down for some time.
every_event- Every cron match will result in a job being scheduled. If multiple cron times go by during an extended period of downtime, then a job will be scheduled for every one missed on startup. This
schedule_typeshould be used for daily batch jobs that need to know which day they are running a batch for. The job will always be scheduled with an ISO8601 string of the cron that matched as the first argument.
This feature ensures that jobs which must run for a certain cron match will always eventually execute, even after a total system crash, or even a DB backup restore.
You can configure some aspects of the gem with an initializer. The default is given below.
Que::Scheduler.configure do |config| # The location of the schedule yaml file. config.schedule_location = ENV.fetch('QUE_SCHEDULER_CONFIG_LOCATION', 'config/que_schedule.yml') # Specify a transaction block adapter. By default, que-scheduler uses the one supplied by que. # However, if, for example you rely on listeners to ActiveRecord's exact `transaction` method, or # Sequel's DB.after_commit helper, then you can supply it here. config.transaction_adapter = ::Que.method(:transaction) end
An audit table que_scheduler_audit is written to by the scheduler to keep a history of what jobs were enqueued when. It is created by the included migration tasks.
HA Redundancy and DB restores
Because of the way que-scheduler works, it requires no additional processes. It is, itself, a Que job. As long as there are Que workers functioning, then jobs will continue to be scheduled correctly. There are no HA concerns to worry about and no namespace collisions between different databases.
Additionally, like Que, when your database is backed up, your scheduling state is stored too. If your workers are down for an extended period, or a DB restore is performed, the scheduler will always be in a coherent state with the rest of your database.
Concurrent scheduler detection
No matter how many tasks you have defined in your schedule, you will only ever need one que-scheduler job enqueued. que-scheduler knows this, and it will check before performing any operations that there is only one of itself present.
It also follows que job design best practices, using ACID guarantees, to ensure that it will never run multiple times. If the scheduler crashes for any reason, it will rollback correctly and try again. It won't schedule jobs twice for a cron match.
How it works
que-scheduler is a job that reads a schedule file, enqueues any jobs it determines that need to be run, then reschedules itself. The flow is as follows:
- The que-scheduler job runs for the very first time.
- que-scheduler loads the schedule file. It will not schedule any other jobs, except itself, as it has never run before.
- Some time later it runs again. It knows what jobs it should be monitoring, and notices that some have are due. It enqueues those jobs and then itself. Repeat.
- After a deploy that changes the schedule, the job notices any new jobs to schedule, and knows which ones to forget. It does not need to be re-enqueued or restarted.
When there is a major version (breaking) change, a migration should be run in. The version of the migration proceeds at a faster rate than the version of the gem. To run in all the migrations required up to a number, just migrate to that number with one line, and it will perform all the intermediary steps.
Que::Scheduler::Migrations.migrate!(version: 4) will perform all migrations necessary to
reach migration version
As of migration
4, two elements are added to the DB for que-scheduler to run.
- The first is the scheduler job itself, which runs forever, re-enqueuing itself to performs its duties.
- The second part comprises the audit table
que_scheduler_auditand the "enqueued" table
que_scheduler_audit_enqueued. The first tracks when the scheduler calculated what was necessary to run (if anything). The second then logs every job that the scheduler enqueues.
You can add tests to validate your configuration during the spec phase. This will perform a variety of sanity checks and ensure that:
- The yml is present and valid
- The job classes exist and are descendants of Que::Job
- The cron fields are present and valid
- The queues (if present) are strings
- The priorities (if present) are integers
- The schedule_types are known
describe 'check que_schedule.yml' do it 'loads the schedule from the default location' do # Will raise an error if any config is invalid expect(Que::Scheduler.schedule).not_to be nil end end
If there is an error during scheduling, que-scheduler will report it using the standard que error notifier. The scheduler will then continue to retry indefinitely.
que-scheduler uses semantic versioning, so major version changes will usually require additional actions to be taken upgrading from one major version to another.
Major feature changes are listed below. The full CHANGELOG can be found in the root of the project.
- Addition of a config initializer.
- Addition of numerous extra columns to the audit table.
- Drop support for ruby 2.1 and rails 3.x
- Required cumulative migration:
- Introduction of the audit table.
- Support for older versions of postgres
- Required cumulative migration:
- Sequel support
- Config specified Timezone support
- Required migration adding the initial job:
- The first public release.
- Required migration adding the initial job:
Your postgres database must be at least version 9.4.0.
This gem was inspired by the makers of the excellent Que job scheduler gem.