Implementation of a sharded First-in First-out queue using Redis and Resque.
This gem unables you to guarantee in-order job processing based on a shard key. Useful for business requirements that are race-condition prone or needs something processed in a streaming manner (jobs that require preservation of chronological order).
Sharding is automatically managed depending on the number of workers available. Durability is guaranteed with failover resharding using a consitent hash. Built on the reliability of resque and is pure ruby which simplifies deployment if you are already using resque with ruby on rails.
Add this line to your application's Gemfile:
And then execute:
Or install it yourself as:
$ gem install resque-fifo-queue
This adds a new task, to run fifo queues:
rake resque:fifo-worker rake resque:fifo-workers
Available options are similar to rake resque:work
Workers will assign their own queue names which is automatically managed by the sharding algorithm.
Supports the same parameters as resque:work but creates additional queues behind the scenes to support fifo queues.
fifo workers can also double duty as standard resque worker in order to simplify resource sharing:
QUEUE=high rake resque:fifo-worker
Aside from being a worker that processes jobs from the fifo queue it will process jobs from the high queue as well.
To start a job using a fifo strategy:
class SampleJob def self.perform(*args) # run your resque job here end end shard_key = "user_00001" # These async jobs will be guaranteed to run one after another in a single worker Resque::Plugins::Fifo::Queue::Manager.enqueue_to(shard_key, SampleJob, "hello") Resque::Plugins::Fifo::Queue::Manager.enqueue_to(shard_key, SampleJob, "hello1")
Resque web extensions
This gem adds a FIFO_queue tab under resque web where you can see information about workers and queues used. This can be accessed via:
Ensuring workers are updated
Since only one worker is assigned to a particular shard, it is important to make sure a worker is running otherwise jobs will pile up for that shard. The shard table is updated everytime a worker is started or stopped properly (using kill -S QUITE). However in exceptional cases (workers are forced killed for example), a shard may be left without a worker. In cases like this a scheduled task is necessary to make sure the worker list is constantly updated:
# resque_schedule.yml auto_refresh_fifo_queues: cron: '*/5 * * * * UTC' class: Resque::Plugins::Fifo::Queue::DrainWorker queue: fifo_refresh description: 'Check if fifo workers are still valid and update worker table'
Do make sure to add the resque-scheduler as well https://github.com/resque/resque-scheduler
If you need something more performant and robust, Apache Kafka is still the way to go. Though you need a couple more libraries to set it up with rails.
After checking out the repo, run
bin/setup to install dependencies. Then, run
rake spec to run the tests. You can also run
bin/console for an interactive prompt that will allow you to experiment.
To install this gem onto your local machine, run
bundle exec rake install. To release a new version, update the version number in
version.rb, and then run
bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the
.gem file to rubygems.org.
Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/resque-fifo-queue. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
The gem is available as open source under the terms of the MIT License.