-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Ent Multi Process
This feature is still under development
Sidekiq Enterprise has the ability to start and manage multiple Sidekiq worker processes, like Unicorn or Puma. With multi-process mode, you get two advantages: significant memory savings by sharing memory between processes and running multiple processes on a single Heroku dyno, allowing you to minimize your dyno costs.
The term for running multi-process is swarm. A swarm has a master process and N worker processes.
Sidekiq Enterprise provides a sidekiqswarm binary. This binary is designed to run under Upstart, Systemd or Foreman as a service. It does not allow old-style options like --daemonize, --logfile or --pidfile.
sidekiqswarm [options]
Start and supervise a swarm of Sidekiq processes.
All arguments are passed to each Sidekiq instance.
Do not provide `-i` as sidekiqswarm will automatically generate it.
You may not use the `-d`, `-L` or `-P` options.
Use the COUNT and INDEX environment variables to control sidekiqswarm.
COUNT Number of Sidekiq processes to start, defaults to number of cores
INDEX Starting index for Sidekiq Pro's reliable fetch, defaults to 0
Example:
COUNT=5 bundle exec sidekiqswarm -r ./myworker.rb
Sidekiq has a sample systemd unit file here. Starting sidekiqswarm instead is almost identical, just update the ExecStart line and configure the environment as necessary:
# if you want to override the default number of processes
Environment=COUNT=2
ExecStart=/usr/local/bin/bundle exec sidekiqswarm -e production
Sidekiq has a sample upstart conf file here. Starting sidekiqswarm instead is almost identical, just update the exec line within the script block and configure the environment as necessary:
# if you want to override the default number of processes
env COUNT=2
exec bundle exec sidekiqswarm -e production
Use the standard upstart and systemd tools to manage the service for your swarm, e.g. systemctl restart sidekiq. You can send the TERM and USR1 signals to the master process and it will pass those signals to the underlying children.
The master could watch the children and restart them if they get above a certain RSS memory usage, helping minimize the pain of leaky applications.