Scheduled Jobs

For jobs to be performed at a desired time, Goose provides 2 options:

• Absolute: perform-at
• Relative: perform-in-sec

Adjusting Polling interval

• RabbitMQ has native support for scheduled message delivery & doesn't need a polling interval
  • Delayed Message Exchange Plugin must be enabled for Scheduled Jobs to work in RabbitMQ
• Redis needs to poll at regular intervals
  • The interval can be defined using scheduler-polling-interval-sec parameter of redis/new-consumer
  • 5 seconds of polling interval is a good default. Can be increased or decreased depending on load of Redis & latency of Scheduled Jobs
  • Goose coordinates between multiple processes to only poll scheduled jobs once in an interval irrespective of the count of workers. All workers must be configured with same value for this to take effect

Nuances

• When a job is due for execution, both Redis/RabbitMQ schedulers enqueue a job, instead of executing it
  • A worker configured with job's queue should be running in order to execute the job
• Goose prioritises scheduled jobs above enqueued jobs
  • In case of a huge backlog on execution queue, Scheduled jobs will have better SLAs
  • This implies newly due jobs will be executed before if older jobs are still in the queue
• Scheduling a job in past will result in priority enqueue, followed by execution
• For Redis, moving a job from sorted set to execution queue is done within a transaction & is reliable

Limitations of Scheduled Jobs in RabbitMQ

• Maximum delay supported by plugin is 2^32 ms(~49 days 17 hours)
• Delayed messages are stored on a single-node. Losing that node will result in data loss
• Does not scale beyond 1,000,000+ delayed messages
• Refer to Delayed Message Plugin's README for more details
• Delayed Message Plugin was chosen because it serves 90% use-cases. If there's a compelling case against it, we are open to implement scheduling using NServiceBus. It is scalable, has lesser limitations; but more complicated to implement

Usage

(ns scheduled-jobs
  (:require [goose.client :as c]))

(let [instant (java.time.Instant/parse "2022-10-31T18:46:09.00Z")]
     (c/perform-at client-opts instant `my-fn :foo))

(c/perform-in-sec client-opts 300 `my-fn :foo)

---

Previous: Priority Queues        Next: Batch Jobs