Deploying to Ubuntu

Ryan McGeary edited this page Dec 31, 2016 · 7 revisions

The most common deployment scenario is using Capistrano to deploy Sidekiq to an Ubuntu server. This page explains how I would do this. Note these directions will 99% work for CentOS 6.x too.

NOTE: If you come across a typo, feel free to click Edit and fix it. This is community documentation.

Using Upstart

Upstart is Ubuntu's init system which controls all critical system services. When the machine reboots, Upstart will restart these services. Services are configured in /etc/init/*.conf and controlled via initctl <verb> <service>. Assuming you want to start a single Sidekiq process, you'll use this script.

The most complex part of that script is juggling whichever Ruby manager you want to use. Edit the user, directory, etc as necessary and save the script to /etc/init/workers.conf. If you need multiple processes, you can create multiple copies with some naming scheme of your choice. Run initctl start workers index=0 and verify the process starts up and works. Log output will go to /var/log/upstart/workers.log; check it for error messages. initctl status workers will give you the current PID.

Set up sudo

Make sure your deploy user has sudo access to initctl. Upstart is poorly designed in that you must have root credentials to do anything with it so we'll need sudo. Add deploy to the sudo group:

usermod -a -G sudo deploy

Using Capistrano

Now that we have Sidekiq set up and running as a system service, we'll create some simple Capistrano hooks to control the process as part of the deploy process. We want to do two things:

  1. Send the USR1 signal at the very start of the deploy to tell Sidekiq to stop processing new jobs. This allows the process to "quiet" down so it can terminated cleanly.
  2. Use Upstart to restart the process at the very end of the deploy so it picks up the new source code.
# For capistrano 3
namespace :sidekiq do
  task :quiet do
    on roles(:app) do
      puts capture("pgrep -f 'sidekiq' | xargs kill -USR1") 
    end
  end
  task :restart do
    on roles(:app) do
      execute :sudo, :initctl, :restart, :workers
    end
  end
end

after 'deploy:starting', 'sidekiq:quiet'
after 'deploy:reverted', 'sidekiq:restart'
after 'deploy:published', 'sidekiq:restart'

# If you wish to use Inspeqtor to monitor Sidekiq
# https://github.com/mperham/inspeqtor/wiki/Deployments
namespace :inspeqtor do
  task :start do
    on roles(:app) do
      execute :inspeqtorctl, :start, :deploy
    end
  end
  task :finish do
    on roles(:app) do
      execute :inspeqtorctl, :finish, :deploy
    end
  end
end

before 'deploy:starting', 'inspeqtor:start'
after 'deploy:finished', 'inspeqtor:finish'

Note we're not using capistrano-sidekiq gem here. capistrano-sidekiq is more full featured but also more complex and doesn't defer process control to Upstart, which I think is a bad idea.

Extra Credit

One problem with Upstart is that if Sidekiq crashes it doesn't notify you so you know there was a problem. You can use Inspeqtor to monitor your Sidekiq service to ensure it is stable, isn't leaking memory or consuming too much CPU, etc. Here's an example script to monitor Sidekiq with Inspeqtor:

# /etc/inspeqtor/services.d/workers.inq
check service workers
  if memory:rss > 2g then alert, restart
  if cpu:user > 95% for 2 cycles then alert

Add your deploy user to the adm group so it can control Inspeqtor without sudo.

usermod -a -G adm deploy

That's it!

Now you should be able to run cap production deploy to update your site and have Sidekiq restarted as part of the normal deploy flow.