Job queue system to run job with Docker
Barbeque is under development but already used on production at Cookpad.
Documentation and open-sourcing plugins are work in progress.
Barbeque is a job queue system that consists of:
- Web console to manage jobs
- Web API to queue a job
- Worker to execute a job
A job for Barbeque is a command you configured on web console. A message serialized by JSON and a job name are given to the command when performed. In Barbeque worker, they are done on Docker container.
- You can achieve job-level auto scaling using tools like Amazon ECS and EC2 Auto Scaling group
- It requires a plugin to run job with ECS, but it's not open-sourced for now
- You don't have to manage infrastructure for each application like Resque or Sidekiq
For details, see Scalable Job Queue System Built with Docker // Speaker Deck.
Install barbeque.gem to an empty Rails app and mount Barbeque::Engine.
And deploy it as you like.
You also need to prepare MySQL, Amazon SQS and Amazon S3.
$ rake barbeque:worker BARBEQUE_QUEUE=defaultWeb API documentation is available at doc/toc.md.
barbeque_client.gem has API client and ActiveJob integration.
The gem is available as open source under the terms of the MIT License.