Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Systemd notify and watchdog support for sidekiqswarm #4511

Open
krasnoukhov opened this issue Mar 26, 2020 · 4 comments
Open

Systemd notify and watchdog support for sidekiqswarm #4511

krasnoukhov opened this issue Mar 26, 2020 · 4 comments

Comments

@krasnoukhov
Copy link
Contributor

@krasnoukhov krasnoukhov commented Mar 26, 2020

Changes that were made for #4488 don't work for sidekiqswarm. I'm sure lots of enterprise customers will benefit from having the same neat integration when running multiple processes.

@mperham
Copy link
Owner

@mperham mperham commented Mar 26, 2020

Unfortunately there are some serious impl issues. The notify functionality doesn't map well to a parent process with N child processes.

  • systemd allows a watchdog timer which will restart a process which does not ping the watchdog within N seconds. This watchdog makes little sense in a multi-process scenario; you want to monitor the individual children and restart any that lock up.
  • what does it mean for the parent swarm process to be ready? when all children have been forked? N seconds after boot where no children have died?

I'll do some googling and see if anyone has thought about or worked through this issue.

@mperham
Copy link
Owner

@mperham mperham commented Mar 26, 2020

I suspect because the process model is completely different I will need to rework the systemd integration to be completely different for the sidekiq and sidekiqswarm binaries. No easy solution that I can see.

mperham added a commit that referenced this issue Mar 26, 2020
…binaries, like sidekiqswarm or a custom Sidekiq launcher, #4511
@misdoro
Copy link
Contributor

@misdoro misdoro commented Mar 27, 2020

Sidekiq swarm may be replaced by systemd template services, letting systemd run many sidekiq instances on a single host, with
sidekiq@.service template
including something like

ExecStartPost=/bin/sh -c "/bin/echo $MAINPID > /var/run/sidekiq%I.pid"
ExecStopPost=/bin/sh -c "rm /var/run/sidekiq%I.pid"

if you need pid files.

Then you may start as many sidekiq instances as you want e.g. with a simple bash script:

/bin/bash
for i in $(seq 1 ${SIDEKIQ_WORKERS}); do sudo systemctl enable --now sidekiq@$i.service; done

Also a custom systemd target may be created:
add a section in
sidekiq@.service:

[Install]
WantedBy=sidekiq.target

and add
sidekiq.target:

[Install]
WantedBy=multi-user.target

That will enable stopping or starting all sidekiq processes with a single command
sudo systemctl stop/start sidekiq.target

@gingerlime
Copy link

@gingerlime gingerlime commented Apr 20, 2020

@misdoro +1 That's pretty much what we're using, but instead of just using sequential ids for each worker, you can even assign a worker per queue (although this might not be the most flexible solution, depending on your needs)

[Unit]
Description=sidekiq %I
# start us only once the network and logging subsystems are available,
# consider adding redis-server.service if Redis is local and systemd-managed.
After=syslog.target network.target
PartOf=sidekiq.target
ReloadPropagatedFrom=sidekiq.target

# See these pages for lots of options:
#
#   https://www.freedesktop.org/software/systemd/man/systemd.service.html
#   https://www.freedesktop.org/software/systemd/man/systemd.exec.html
#
# THOSE PAGES ARE CRITICAL FOR ANY LINUX DEVOPS WORK; read them multiple
# times! systemd is a critical tool for all developers to know and understand.
#
[Service]
Type=simple
WorkingDirectory=/var/local/app
RuntimeDirectory=sidekiq

# NOTE: %I will be replaced by the queue name used when initializing, e.g.
#       systemctl start sidekiq@batch.service
ExecStart=/usr/bin/bundle exec sidekiq -e production -q %I

# writes the main process id to a pid file
ExecStartPost=/bin/sh -c "echo $MAINPID > /run/sidekiq/sidekiq.%I.pid"

# use `systemctl reload sidekiq` to send the quiet signal to Sidekiq
# at the start of your deploy process.
ExecReload=/bin/kill -TSTP $MAINPID

User=www-data
Group=www-data
UMask=0002

# Greatly reduce Ruby memory fragmentation and heap usage
# https://www.mikeperham.com/2018/04/25/taming-rails-memory-bloat/
Environment=MALLOC_ARENA_MAX=2

# if we crash, restart
RestartSec=1
Restart=on-failure

# output goes to /var/log/syslog (does not seem to affect logging, controlled by Semantic Logger)
StandardOutput=syslog
StandardError=syslog

# This will default to "bundler" if we don't specify it
SyslogIdentifier=sidekiq

[Install]
WantedBy=sidekiq.target
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.