Ent Multi Process

Sidekiq Enterprise 1.2.0+ has the ability to start and manage multiple Sidekiq worker processes, a la Unicorn or Puma. With multi-process mode, you get several advantages:

1. significant memory savings by sharing memory between processes
2. running multiple processes on a single Heroku dyno, allowing you to minimize your dyno costs.
3. easy to create a single service in Upstart or Systemd which scales to all machine cores

The term for running multi-process is swarm. A swarm has a master process and N worker processes.

Starting a Swarm

Sidekiq Enterprise provides a sidekiqswarm binary. This binary is designed to run under Upstart, Systemd or Foreman as a service. It does not allow old-style options like --daemonize, --logfile or --pidfile.

sidekiqswarm [options]

Start and supervise a swarm of Sidekiq processes.
All arguments are passed to each Sidekiq instance.
Do not provide `-i` as sidekiqswarm will automatically generate it.

You may not use the `-d`, `-L` or `-P` options.

Use the COUNT and INDEX environment variables to control sidekiqswarm.

COUNT	Number of Sidekiq processes to start, defaults to number of cores
INDEX	Starting index for Sidekiq Pro's reliable fetch, defaults to 0

Example:
COUNT=5 bundle exec sidekiqswarm -r ./myworker.rb

Running via init

systemd

Sidekiq has a sample systemd unit file here. Starting sidekiqswarm instead is almost identical, just update the ExecStart line and configure the environment as necessary:

# if you want to override the default number of processes
Environment=COUNT=2
ExecStart=/usr/local/bin/bundle exec sidekiqswarm -e production

upstart

Sidekiq has a sample upstart conf file here. Starting sidekiqswarm instead is almost identical, just update the exec line within the script block and configure the environment as necessary:

# if you want to override the default number of processes
env COUNT=2

exec bundle exec sidekiqswarm -e production

Signals and Controlling a Swarm

Use the standard upstart and systemd tools to manage the service for your swarm, e.g. systemctl restart sidekiq.

You can send the TERM and USR1 signals to the master process and it will pass those signals to the underlying children. Once the master process has received USR1 or TERM, it will not spawn any more children; it must be restarted. The master process does not handle the TTIN signal.

Limitations

Sidekiq forks the worker processes after running Bundler.require(:default) but before booting the application so the workers can share the memory consumed by loading the gems. Your Gemfile should eager load gems where possible; using gem 'something', require: false in your Gemfile will limit any memory savings.

TODO

The master should watch the children and restart them if they get above a certain RSS memory usage, minimizing the pain of leaky applications.