Process long running jobs asynchronously

Overview

To keep responses quick and help scale a web application's needs, its necessary to process certain tasks asynchronously from the web server process.

A few examples include:

• Opening objects from slow-clients
• Scaling images upon creation (e.g. multiple styles using Paperclip)
• Sending emails
• Importing/exporting data from/to external locations

The following will set you up to use Sidekiq, a powerful, concurrent asynchronous tasks library.

After the following steps, you will:

• Be able to create and execute asynchronous tasks
• Have visibility into the task queues at status.lvh.me:3000/sidekiq
• Be able to scale your worker capacity directly with the heroku scale command

Tutorial

Assumptions

1. You have redis installed locally and running on default port 6379

• install with brew install redis
• run brew info redis for info about auto-start on login with launchd

Steps

Include gems

• Uncomment the following gems in the gemfile

gem "httparty",                 "~> 0.12"     # make requests to slow clients in async workers
gem "sidekiq",                  "~> 2.17"
gem "sidekiq-failures",         "~> 0.2"
gem "sinatra",                  "~> 1.4.4"

• Bundle with bundle install

Handle the non-worker scenario

• Add the following to development.rb:

# run async workers synchronously in development
# so that a simple `rails s` still 'just works'
unless ENV['WORKERS_PRESENT'] == 'true'
  require 'sidekiq/testing/inline'
end

Set up the worker process

• Create queue configuration file at config/worker.yml with the following content:

# A queue with a weight of 2 will be checked twice 
# as often as a queue with a weight of 1
---
:verbose: true
:concurrency:  10
:queues:
  - [mailer, 3]
  - [paperclip, 2]
  - [some_custom_queue_name, 2]
  - [default]

production:
  :verbose: false

• Create sidekiq configuration file at config/initializers/sidekiq.rb with the following:

# Normally with a web server, you set the client size to 1.
# The Rails process is the client, feeding jobs into Redis.
#
# The Sidekiq process is the server. It will have approx
# the same number of database connections as concurrency,
# which defaults to 25.

Rails.logger.info "*** Configuring Sidekiq..."

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

unless Rails.application.config.consider_all_requests_local
  Sidekiq::Web.use Rack::Auth::Basic do |username, password|
    username == ENV['BASIC_AUTH_USER'] && password == ENV['BASIC_AUTH_PASSWORD']
  end
end

# By default, Sidekiq assumes Redis is located at localhost:6379
#
# Also by default, it will use ENV vars configured in the following way:
# RedisToGo can be used with the REDISTOGO_URL env var directly.
# All others can use the REDIS_PROVIDER env var.
#
##  Ex: REDIS_PROVIDER=REDISGREEN_URL will configure Sidekiq to use
##  the REDISGREEN_URL value for it's redis connection.
#
# Finally, the generic REDIS_URL var may be used.
#
# Note that heroku (addon) redis providers create/use these ENV vars by default.

# To point to an external Redis server, configure server and client:
#
# Sidekiq.configure_server do |config|
#   config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
# end
#
# # When in Unicorn, this block needs to go in unicorn's `after_fork` callback:
# Sidekiq.configure_client do |config|
#   config.redis = { :url => 'redis://redis.example.com:7372/12', :namespace => 'mynamespace' }
# end

• Define the worker process by adding the following to your Procfile:

worker: bundle exec sidekiq -C ./config/worker.yml -e $RACK_ENV

(Optional) Set up the worker process for development with foreman

• Define the worker process if Procfile.dev:

worker: bundle exec sidekiq -C ./config/worker.yml -e development

• Process jobs asynchronously in development by adding the following to .env

WORKERS_PRESENT=true

Create a route to the sidekiq dashboard

• Add the following as low as you can in your routes block in config/routes.rb:

  constraints subdomain: /\Astatus/ do
    mount Sidekiq::Web => '/sidekiq'
  end

Configure worker persistence layer on staging/production

• Add redis to staging (and production) heroku addons:add rediscloud

• Add config variable to tell sidekiq how to connect to the worker persistence layer. heroku config:add REDIS_PROVIDER=REDISCLOUD_URL

• Start a worker process heroku scale worker=1

Next Steps

1. Read the Sidekiq reference for information on using workers
2. Use the delayed_paperclip gem to automatically process paperclip image processing tasks