Skip to content

Monitoring

Jump to bottom
Josh Adams edited this page Jun 3, 2016 · 161 revisions

I recommend using a tool to monitor your Sidekiq processes in production to ensure they are always up and aren't using too much memory or CPU. I built Inspeqtor because I didn't like the existing tools that were available (e.g. monit, God and bluepill). My recommendations:

  1. Use Upstart or Systemd to start/stop Sidekiq. This ensures if the Ruby VM crashes, the process will respawn immediately.
  2. Use Inspeqtor to monitor the CPU and memory usage and restart Sidekiq if necessary.

Web UI

Sidekiq comes with a Sinatra application that can display the current state of a Sidekiq installation.

Rails

Add sinatra to your Gemfile:

# if you require 'sinatra' you get the DSL extended to Object
gem 'sinatra', :require => nil

Add the following to your config/routes.rb:

require 'sidekiq/web'
mount Sidekiq::Web => '/sidekiq'

Forbidden

If you receive a Forbidden error when trying to submit a form, you do not have a valid session configured. A valid session is required to prevent CSRF attacks. You must configure Sinatra to share the same session with Rails. Try putting this in your routes.rb after the require:

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_token]
Sidekiq::Web.set :sessions, Rails.application.config.session_options

Authentication

In a production application you'll likely want to protect access to this information. You can use the constraints feature of routing (in the config/routes.rb file) to accomplish this:

Devise

Allow any authenticated User

# config/routes.rb
authenticate :user do
  mount Sidekiq::Web => '/sidekiq'
end

Same as above but also ensures that User#admin? returns true

# config/routes.rb
authenticate :user, lambda { |u| u.admin? } do
  mount Sidekiq::Web => '/sidekiq'
end

Clearance

Clearance provides routing constraints to restrict access to routes.

Blog::Application.routes.draw do
  # Restricts access to all authenticated users
  constraints Clearance::Constraints::SignedIn.new do
    mount Sidekiq::Web, at: '/sidekiq'
  end

  # Restricts access to all authenticated admins
  constraints Clearance::Constraints::SignedIn.new { |user| user.admin? } do
    mount Sidekiq::Web, at: '/sidekiq'
  end
end

Authlogic

# lib/admin_constraint.rb
class AdminConstraint
  def matches?(request)
    return false unless request.cookie_jar['user_credentials'].present?
    user = User.find_by_persistence_token(request.cookie_jar['user_credentials'].split(':')[0])
    user && user.admin?
  end
end

# config/routes.rb
require "admin_constraint"
mount Sidekiq::Web => '/sidekiq', :constraints => AdminConstraint.new

Restful Authentication or Sorcery

Checks a User model instance that responds to admin?

# lib/admin_constraint.rb
class AdminConstraint
  def matches?(request)
    return false unless request.session[:user_id]
    user = User.find request.session[:user_id]
    user && user.admin?
  end
end

# config/routes.rb
require 'sidekiq/web'
require 'admin_constraint'
mount Sidekiq::Web => '/sidekiq', :constraints => AdminConstraint.new

Custom External Authentication

class AuthConstraint
  def self.admin?(request)
    return false unless (cookie = request.cookie_jar['auth'])

    Rails.cache.fetch(cookie['user'], :expires_in => 1.minute) do
      auth_data = JSON.parse(Base64.decode64(cookie['data']))
      response = HTTParty.post(Auth.validate_url, :query => auth_data)

      response.code == 200 && JSON.parse(response.body)['roles'].to_a.include?('Admin')
    end
  end
end

# config/routes.rb
constraints lambda {|request| AuthConstraint.admin?(request) } do
  mount Sidekiq::Web => '/admin/sidekiq'
end

Rails with Google authentication

@jonhyman breaks down how Appboy uses Google to protect access to Sidekiq.

Rails HTTP Basic Auth from Routes

# config/routes.rb
require "sidekiq/web"
Sidekiq::Web.use Rack::Auth::Basic do |username, password|
  username == ENV["SIDEKIQ_USERNAME"] && password == ENV["SIDEKIQ_PASSWORD"]
end if Rails.env.production?
mount Sidekiq::Web, at: "/sidekiq"

If you get an ActionDispatch::Request::Session error, you've hit an incompatibility between Rails and Rack. See this comment for a workaround.

Standalone

Here's an example config.ru for booting Sidekiq::Web in your choice of Rack server:

require 'sidekiq'

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

require 'sidekiq/web'
run Sidekiq::Web

You can mount sidekiq to existing Rack (Sinatra) application as well:

require 'your_app'

require 'sidekiq/web'
run Rack::URLMap.new('/' => Sinatra::Application, '/sidekiq' => Sidekiq::Web)

Or if you want to use Sidekiq::Web with Rakefiles:

task :monitor do
  # optional: Process.daemon (and take care of Process.pid to kill process later on)
  require 'sidekiq/web'
  app = Sidekiq::Web
  app.set :environment, :production
  app.set :bind, '0.0.0.0'
  app.set :port, 9494
  app.run!
end

Rack session and protection against web attacks

Note that Sidekiq::Web requires a valid Rack session to work. If you see a Forbidden error when clicking a button in the Web UI, it's because the Rack session is not configured correctly. Sidekiq cannot configure a session for you. If you do not know how to set up a valid session in your system, your best option is to search StackOverflow or post a question there with the code you are using to run the Web UI.

Sidekiq::Web uses Rack::Protection to protect your application against typical web attacks (such as CSRF, XSS, etc). Rack::Protection would invalidate your session and raise Forbidden error if it finds that your request doesn't satisfy security requirements. One of the possible situations is having your application working behind a reverse proxy and not passing important headers to it (X-Forwarded-For, X-Forwarded-Proto). Such situation and solution could be found in this article and issue #2560.

If you have wildcard domains with your Rails app and want to access the Web UI from all of them, see issue #2730.

If you do everything right, you should see this in your browser:

Web UI

Standalone with GitHub OAuth

This configuration will allow access only to members of your GitHub organization. Start by creating a new OAuth application. You will also need to create a secure session cookie:

openssl rand -base64 48
require 'sinatra_auth_github'

module Sidekiq
  class Web
    set :session_secret, ENV['RACK_SESSION_COOKIE']

    set :github_options, {
      :scopes    => "user",
      :client_id => ENV['GITHUB_KEY'],
      :secret    => ENV['GITHUB_SECRET']
    }

    register Sinatra::Auth::Github

    before do
      authenticate!
      github_organization_authenticate!(ENV['GITHUB_ORG'])
    end

    get '/logout' do
      logout!
    end
  end
end

Standalone with Basic Auth

# this code goes in your config.ru
require 'sidekiq'

Sidekiq.configure_client do |config|
  config.redis = { :size => 1 }
end

require 'sidekiq/web'
map '/sidekiq' do
  use Rack::Auth::Basic, "Protected Area" do |username, password|
    username == 'sidekiq' && password == 'sidekiq'
  end

  run Sidekiq::Web
end

Nagios

Below is a collection of nagios checks that includes check_sidekiq_queue script, which validates that a given queue depth is within a particular range. It's a simple shell script that uses redis-cli command line tool, and does not have any dependency on ruby.

https://github.com/wanelo/nagios-checks

Scout

scout sidekiq plugin

The Sidekiq Monitor plugin for Scout, a hosted server monitoring service, reports key metrics like enqueued, processed, and failed jobs. Triggers can be configured to alert when the metrics reach specified thresholds. A Scout account is required to use the plugin.

Monitoring Queue Backlog

You can use a simple HTTP endpoint with Pingdom to check the size of your Sidekiq 'default' queue backlog. Put this in config/routes.rb:

require 'sidekiq/api'
match "queue-status" => proc { [200, {"Content-Type" => "text/plain"}, [Sidekiq::Queue.new.size < 100 ? "OK" : "UHOH" ]] }

Now when you hit http://example.com/queue-status, the body of the response will be either 'OK' or 'UHOH'. We have a Pingdom check every minute which fires off an email if the response == 'UHOH'.

Monitoring Queue Latency

Using a custom end-point

If you throw a lot of jobs into the queue, you can get false positives when monitoring the queue backlog. Instead, monitor the queue latency. Queue latency is the difference between when the oldest job was pushed onto the queue versus the current time. This code will check that jobs don't spend more than 30 seconds enqueued. Put this in config/routes.rb:

require 'sidekiq/api'
match "queue-latency" => proc { [200, {"Content-Type" => "text/plain"}, [Sidekiq::Queue.new.latency < 30 ? "OK" : "UHOH" ]] }

Now when you hit http://example.com/queue-latency, the body of the response will be either 'OK' or 'UHOH'.

Using the built-in dashboard

Sidekiq provides a JSON formatted dashboard at /dashboard/stats. You get this :

{
  "sidekiq": {
    "processed": 12345,
    "failed": 56,
    "busy": 25,
    "enqueued": 178,
    "scheduled": 0,
    "retries": 0,
    "default_latency": 12
  },
  "redis": {
    "connected_clients": "120",
    "uptime_in_days": "35",
    "used_memory_human": "602.31M",
    "used_memory_peak_human": "1.01G"
  }
}

Troubleshooting

The /sidekiq route is not there even though I've added it to config/routes.rb

If you have added the route to your application but run into a 'no route error' when trying to access it, it is likely that you have the following in your Gemfile:

gem 'sinatra'

Change it to:

gem 'sinatra', require: false

This is because if you allow sinatra to be required, it makes the sinatra/base api, which Sidekiq uses, inaccessible.

Previous: Deployment Next: Internals

Home | The Basics | Best Practices | Using Redis | Error Handling | Advanced Options | Problems?

Clone this wiki locally