Job queue system to run job with Docker
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app Make delay_seconds optional in MessageEnqueueingService Aug 23, 2018
bin rails plugin new --mountable -T barbeque Jul 26, 2016
config Change job execution URL from `/job_executions/:id` to `/job_executio… Sep 15, 2017
db/migrate
doc Add delay_seconds parameter to support SQS's delay_seconds Aug 23, 2018
lib Version 2.5.0 Aug 24, 2018
spec Make delay_seconds optional in MessageEnqueueingService Aug 23, 2018
tools Use modularized aws-sdk Aug 31, 2017
vendor/assets/javascripts Update plotly.js to v1.29.3 Aug 17, 2017
.gitignore Ignore Gemfile.lock Sep 1, 2016
.rspec rails g rspec:install Aug 1, 2016
.travis.yml
CHANGELOG.md Version 2.5.0 Aug 24, 2018
Gemfile
MIT-LICENSE
README.md Add flag to enable verification in enqueueing step Apr 27, 2018
Rakefile
barbeque.gemspec Upgrade Rails to 5.2 Apr 12, 2018

README.md

Barbeque Build Status

Job queue system to run job with Docker

Project Status

Barbeque is used on production at Cookpad.

What's Barbeque?

Barbeque is a job queue system that consists of:

  • Web console to manage jobs
  • Web API to queue a job
  • Worker to execute a job

A job for Barbeque is a command you configured on web console. A message serialized by JSON and a job name are given to the command when performed. In Barbeque worker, they are done on Docker container.

Why Barbeque?

  • You can achieve job-level auto scaling using tools like Amazon ECS EC2 Auto Scaling group
    • For Amazon ECS, Barbeque has Hako executor
  • You don't have to manage infrastructure for each application like Resque or Sidekiq

For details, see Scalable Job Queue System Built with Docker // Speaker Deck.

Deployment

Web API & console

Install barbeque.gem to an empty Rails app and mount Barbeque::Engine. And deploy it as you like.

You also need to prepare MySQL, Amazon SQS and Amazon S3.

For sandbox environment

Barbeque's enqueue API tries to be independent of MySQL by design. Although that design policy, verifying the enqueued job is useful in some environment (such as sandboxed environment). Passing BARBEQUE_VERIFY_ENQUEUED_JOBS=1 to the Web API server enables the feature that verifies the enqueued job by accessing MySQL.

Worker

$ rake barbeque:worker BARBEQUE_QUEUE=default

The rake task launches four worker processes.

  • Two runners
    • receives message from SQS queue, starts job execution and stores its identifier to the database
  • One execution poller
    • gets execution status and reflect it to the database
  • One retry poller
    • gets retried execution status and reflect it to the database

Usage

Web API documentation is available at doc/toc.md.

Ruby

barbeque_client.gem has API client and ActiveJob integration.

Executor

Barbeque executor can be customized in config/barbeque.yml. Executor is responsible for starting executions and getting status of executions.

Barbeque has currently two executors.

Docker (default)

Barbeque::Executor::Docker starts execution by docker run --detach and gets status by docker inspect.

Hako

Barbeque::Executor::Hako starts execution by hako oneshot --no-wait and gets status from S3 task notification.

Requirement

You must configure CloudWatch Events for putting S3 task notification. See Hako's documentation for detail. https://github.com/eagletmt/hako/blob/master/docs/ecs-task-notification.md

License

The gem is available as open source under the terms of the MIT License.