Kubernetes

Sidekiq is a cluster of worker processes "around" a Redis server. This page provides information on how to operate Sidekiq using Kubernetes.

Basic Setup

Running and connecting to Redis...

Safe Shutdown

How to configure k8s to get a clean shutdown with TSTP and TERM...

During the termination lifecycle, k8s will send a SIGTERM to the root process on each pod:

In practice, this means your application needs to handle the SIGTERM message and begin shutting down when it receives it. This means saving all data that needs to be saved, closing down network connections, finishing any work that is left, and other similar tasks.

Sidekiq uses the TERM signal to start shutting down within the timeout/-t seconds.

k8s also has its own concept of a grace period, indicated by the terminationGracePeriodSeconds in the pod's configuration. k8s will wait this amount of time before sending a SIGKILL to the container, forcefully terminating all processes.

It's recommend to have the Sidekiq timeout set to a value less than the k8s terminationGracePeriodSeconds. If using the default Sidekiq timeout of 25 seconds, setting terminationGracePeriodSeconds in k8s to 30 seconds is recommended (the k8s default).

Health Checks

Each time k8s brings up a new pod, it needs to assert the pod is "healthy" before considering the action complete. "Healthy" for something serving web requests might mean that that pod responds with 200 OK to a health check endpoint.

For Sidekiq processes, we need a different approach. Instead of using a web request, you can use a combination of Sidekiq lifecycle hooks and a file-based readinessProbe. Here's how that looks:

# config/initializers/sidekiq.rb
SIDEKIQ_WILL_PROCESSES_JOBS_FILE = Rails.root.join('tmp/sidekiq_process_has_started_and_will_begin_processing_jobs').freeze

Sidekiq.configure_server do |config|
  # We touch and destroy files in the Sidekiq lifecycle to provide a
  # signal to Kubernetes that we are ready to process jobs or not.
  #
  # Doing this gives us a better sense of when a process is actually
  # alive and healthy, rather than just beginning the boot process.
  config.on(:startup) do
    FileUtils.touch(SIDEKIQ_WILL_PROCESSES_JOBS_FILE)
  end

  config.on(:shutdown) do
    FileUtils.rm_f(SIDEKIQ_WILL_PROCESSES_JOBS_FILE)
  end
end

And then later in your pod's YAML, assuming your Rails application is deployed to /var/www within the pod and takes about 10 seconds to start:

readinessProbe:
  failureThreshold: 10
  exec:
    command:
    - cat
    - /var/www/tmp/sidekiq_process_has_started_and_will_begin_processing_jobs
  initialDelaySeconds: 10
  periodSeconds: 2
  successThreshold: 2
  timeoutSeconds: 1

Autoscaling

Scaling via queue latency...

Cloud Providers

Notes about using AWS, GCP, DO and other KaaS providers...?