AWS Beanstalk and sidekiq 6.0 +

[silva96]

The new 6.0 versions removes daemonization of sidekiq, but Elastic Beanstalk script uses pidfile and logfiles

Can you provide a guide to be able to use sidekiq 6 in beanstalk?

# Sidekiq interaction and startup script
# taken from: https://gist.github.com/joshtab/8666546c3cec4b221603d55b02bf24c5
commands:
  create_post_dir:
    command: "mkdir -p /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars

      EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
      SIDEKIQ_CONFIG=$EB_APP_DEPLOY_DIR/config/sidekiq.yml
      SIDEKIQ_LOG=$EB_APP_DEPLOY_DIR/log/sidekiq.log

      cd $EB_APP_DEPLOY_DIR

      if [ -f $SIDEKIQ_PID ]
      then
        echo "terminating existing sidekiq"
        su -s /bin/bash -c "kill -TERM `cat $SIDEKIQ_PID`" $EB_APP_USER
        su -s /bin/bash -c "rm -rf $SIDEKIQ_PID" $EB_APP_USER
      fi

      . /opt/elasticbeanstalk/support/envvars.d/sysenv

      sleep 10

      echo "starting sidekiq"
      su -s /bin/bash -c "bundle exec sidekiq \
        -e $RACK_ENV \
        -P $SIDEKIQ_PID \
        -C $SIDEKIQ_CONFIG \
        -L $SIDEKIQ_LOG \
        -d" $EB_APP_USER

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars

      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
      if [ -f $SIDEKIQ_PID ]
      then
        echo "TSTP/quieting sidekiq"
        su -s /bin/bash -c "kill -TSTP `cat $SIDEKIQ_PID`" $EB_APP_USER
      fi

I see there is some ways to enable services and commands

https://stackoverflow.com/questions/21682443/how-to-install-and-enable-a-service-in-amazon-elastic-beanstalk

[mperham]

You should use a script which uses supervisord to watch and restart your Sidekiq processes. Here's one example:

https://stackoverflow.com/a/22533800/1494519

[ctrlaltdylan]

Would be great if there was an updated example for Elasticbeanstalk specifically documented for v6. The SO answer references a django application, not ruby.

[ctrlaltdylan]

Finally found a way without dealing with supervisord:

https://gist.github.com/ctrlaltdylan/f75b2e38bbbf725acb6d48283fc2f174

[mperham]

Sidekiq 6 + EB:

https://medium.com/kite-srm/setting-up-sidekiq-6-0-on-aws-b4f2e01f451c

[mperham]

The real crime is EB forcing you to use Upstart 0.6.5 which is something like 10 years old.

[NemyaNation]

EB running Ruby on Amazon Linux 2 has removed support for Upstart. Any other configuration we can use to run Sidekiq on EB running Ruby 2.7 on Amazon Linux 2.

Update: This worked for me: https://wiki.lyrasis.org/display/~elr37/Set+up+Sidekiq+for+Rails+app+on+AWS+Elastic+Beanstalk
Update2: This configuration needs to be applied on Amazon Linux 2
https://forums.aws.amazon.com/thread.jspa?threadID=330819

[ashbarot]

Sidekiq ~ 6
Ruby ~ 2.7
Amazon Linux 2:

Finally a step by step guide on how to deploy:
https://www.barot.us/running-sidekiq-on-amazon-linux-2/

I spent over 160 hours figuring it out. Hope it helps you save time.

[mperham]

@fromwolf You spent 160 hours building scripts to recreate the old, crappy deploy rather than using capistrano-sidekiq with AL2/systemd as I have recommended for years.

Stop creating PID file and logfile hacks manually. Let the operating system manage sidekiq and other services for you. That's literally the job of an operating system.

[kei178]

Just sharing the example setup to run Sidekiq with systemd on AWS ElasticBeanstalk (Amazon Linux 2) in case it would be useful for someone: https://medium.com/@kei178/rails-how-to-run-sidekiq-with-systemd-on-aws-elasticbeanstalk-b28efa105d04

[bowtiesolutions]

For anyone coming from a google search, Amazon Linux 2 supports procfiles to determine the application(s) startup

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html

Sidekiq can be easily loaded with lifecycle management by adding a single line to the Procfile in the root of your application

web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb
sidekiq: bundle exec sidekiq

[matiasmasca]

For anyone coming from a google search, Amazon Linux 2 supports procfiles to determine the application(s) startup

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html

Sidekiq can be easily loaded with lifecycle management by adding a single line to the Procfile in the root of your application

web: bundle exec puma -C /opt/elasticbeanstalk/config/private/pumaconf.rb
sidekiq: bundle exec sidekiq

To create a different Procfile depending on with type of tier it is, I used an ENV variable and I created this script: .platform/hooks/prebuild/01_create_procfile.sh

#!/bin/sh

echo "Create a Procfile by ENV_TYPE `$ENV_TYPE`" &>> /tmp/deploy_echo_container_commands.log

if [ $ENV_TYPE == "web" ]; then
    echo "web: puma -C /var/app/current/config/puma.rb" > /var/app/staging/Procfile
else
    echo "worker: bundle exec sidekiq" > /var/app/staging/Procfile
fi

echo "FINISH: Procfile created at: `date`" >> /tmp/deploy_echo_container_commands.log