Ent Periodic Jobs

Mike Perham edited this page Sep 12, 2016 · 21 revisions

Sidekiq Enterprise supports periodic jobs, aka cron or recurring jobs. You register your periodic jobs with a schedule upon startup and Sidekiq will fire off corresponding jobs on that schedule. Sidekiq will ensure that only a single process creates jobs so if you are running 1 or 100 processes, you don't need to worry about duplicate job creation.

See periodic jobs in action here:

Periodic Jobs

Definition

Periodic Jobs should be defined in your Sidekiq initializer, like so:

Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    # see any crontab reference for the first argument
    # e.g. http://www.adminschoice.com/crontab-quick-reference
    mgr.register('0 * * * *', SomeHourlyWorkerClass)
    mgr.register('* * * * *', SomeWorkerClass, retry: 2, queue: 'foo')
    mgr.register(cron_expression, worker_class, job_options={})
  end
end

For example, with the schedule above Sidekiq will create a SomeWorkerClass job every minute in the foo queue to be processed like any normal job. If the job raises an error, Sidekiq will retry it like any normal job. Periodic job options will override any options set with sidekiq_options within the Worker class.

Time Zones

The cron expressions above implicitly use the Ruby process's Time Zone, which defaults to the system time zone. You can set the TZ manually with something like TZ=America/Los_Angeles bundle exec .... There's no way to manually set the TZ in your cron expression. If you have a machine set to Pacific time and want to run a job at 4am Eastern, you can set the cron expression to run at 1am since 1am Pacific == 4am Eastern.

Dynamic Jobs

Sidekiq Enterprise does not support end-user-managed cron jobs out of the box. Sidekiq never touches your database and anything user-managed should be controlled by your database. Your application should have a user interface to manage dynamic jobs like any other Rails resource, allowing the user to create, list, delete, etc their jobs as normal.

You'd then have a Periodic Job running every minute which looks for database records representing jobs which need to run now. The static job would enqueue each dynamic job and update the database record with the next timestamp.

create_table :dynamic_jobs do |t|
  # these are totally optional, might want to track them for multi-tenancy or security purposes
  t.references :account_id, null: false, index: true
  t.references :user_id, null: false, index: true

  t.string :klass, null: false
  t.string :cron_expression, null: false
  t.timestamp :next_run_at, null: false, index: true
end

class DynamicJobWorker
  include Sidekiq::Worker

  def perform
    DynamicJob.find_each("next_run_at <= ?", Time.now) do |job|
      # Multi-tenant apps can use a server-side middleware to set DB connection based on account_id and/or user_id.
      Sidekiq::Client.push(:class => job.klass.constantize, :args => [],
                           :account_id => job.account_id, :user_id => job.user_id)
      x = Sidekiq::CronParser.new(job.cron_expression)
      job.update_attribute!(:next_run_at, x.next.to_time)
    end
  end
end

Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    mgr.register("* * * * * *", DynamicJobWorker)
  end
end

Extra credit: support a job argument by adding it to the model and passing it in the push method.

Limitations

The most frequently a job can be scheduled with the crontab format is every minute. If you want a job to run every 15 seconds, you'll need to build it yourself (perhaps creating a custom thread plus the Leader Election feature) or use a meta-job to schedule jobs to run in 0,15,30,45 seconds. Note that Sidekiq's scheduler is not very precise out of the box, you can adjust the precision as explained here.

This implementation does not support backfill. If Sidekiq is shutdown, it will not create jobs for the times missed on restart. I advise people to make their cron jobs resilient to timing. Instead of having an hourly job which processes only the last hour of data, make it process the last N hours. 99.9% of the time, it will only need to process one hour's worth of data anyways but if a job is missed, the next hour's job will process it.

API

The Periodic API allows you to list registered periodic jobs and see enqueue history:

loops = Sidekiq::Periodic::LoopSet.new
loops.each do |lop|
  p [lop.schedule, lop.klass, lop.lid, lop.options, lop.history]
end

Web UI

The Web UI exposes a "Cron" tab with any registered jobs. You can see an overview of all registered jobs and see job execution history. Make sure you require 'sidekiq-ent/web' in your config/routes.rb.

screenshot

Notes

  • The perform method for periodic workers must take NO parameters (or have default values for all parameters).