idle-timeout not shutting down when workers are set

[davidalejandroaguilar]

Expected

Puma to shut down after 5 seconds of inactivity.

# puma.rb
#
# ... rest of file
# 
# workers ENV.fetch("WEB_CONCURRENCY") { 2 }

bundle exec puma hello.ru --idle-timeout=5

➜  git:(main) ✗ bundle exec puma hello.ru --idle-timeout=5
Puma starting in single mode...
* Puma version: 6.4.0 (ruby 3.2.0-p0) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 5
*  Environment: development
*          PID: 23471
* Listening on http://0.0.0.0:3000
Use Ctrl-C to stop
- Gracefully stopping, waiting for requests to finish
=== puma shutdown: 2023-09-21 13:14:34 -0700 ===
- Goodbye!

Actual

Puma restarts workers infinitely.

# puma.rb
#
# ... rest of file
# 
workers ENV.fetch("WEB_CONCURRENCY") { 2 }

bundle exec puma hello.ru --idle-timeout=5

➜  git:(main) ✗ bundle exec puma hello.ru --idle-timeout=5
[23279] Puma starting in cluster mode...
[23279] * Puma version: 6.4.0 (ruby 3.2.0-p0) ("The Eagle of Durango")
[23279] *  Min threads: 5
[23279] *  Max threads: 5
[23279] *  Environment: development
[23279] *   Master PID: 23279
[23279] *      Workers: 2
[23279] *     Restarts: (✔) hot (✔) phased
[23279] * Listening on http://0.0.0.0:3000
[23279] Use Ctrl-C to stop
[23279] - Worker 0 (PID: 23405) booted in 0.0s, phase: 0
[23279] - Worker 1 (PID: 23406) booted in 0.0s, phase: 0
[23279] - Worker 0 (PID: 23412) booted in 0.0s, phase: 0
[23279] - Worker 1 (PID: 23411) booted in 0.01s, phase: 0
[23279] - Worker 0 (PID: 23417) booted in 0.01s, phase: 0
[23279] - Worker 1 (PID: 23418) booted in 0.0s, phase: 0

Puma config:

Default Puma config when initializing a Rails app.

# Puma can serve each request in a thread from an internal thread pool.
# The `threads` method setting takes two numbers: a minimum and maximum.
# Any libraries that use thread pools should be configured to match
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies the `worker_timeout` threshold that Puma will use to wait before
# terminating a worker in development environments.
#
worker_timeout 3600 if ENV.fetch("RAILS_ENV", "development") == "development"

# Specifies the `port` that Puma will listen on to receive requests; default is 3000.
#
port ENV.fetch("PORT") { 3000 }

# Specifies the `environment` that Puma will run in.
#
environment ENV.fetch("RAILS_ENV") { "development" }

# Specifies the `pidfile` that Puma will use.
pidfile ENV.fetch("PIDFILE") { "tmp/pids/server.pid" }

# Specifies the number of `workers` to boot in clustered mode.
# Workers are forked web server processes. If using threads and workers together
# the concurrency of the application would be max `threads` * `workers`.
# Workers do not work on JRuby or Windows (both of which do not support
# processes).
#
workers ENV.fetch("WEB_CONCURRENCY") { 2 }

# Use the `preload_app!` method when specifying a `workers` number.
# This directive tells Puma to first boot the application and load code
# before forking the application. This takes advantage of Copy On Write
# process behavior so workers use less memory.
#
# preload_app!

# Allow puma to be restarted by `bin/rails restart` command.
plugin :tmp_restart

To Reproduce

hello.ru file:

run lambda { |env| [200, {"Content-Type" => "text/plain"}, ["Hello World"]] }

Run it with:

bundle exec puma hello.ru --idle-timeout=5

Notes

I initially thought this was because of Scout APM's Puma worker loop, because I was seeing this when also enabling preload_app and using scout_apm: (on a new Rails app, which you can clone here):

➜  git:(main) ✗ RAILS_ENV=production SCOUT_DEV_TRACE=true bundle exec puma --idle-timeout=5
[24120] Puma starting in cluster mode...
[24120] * Puma version: 6.4.0 (ruby 3.2.0-p0) ("The Eagle of Durango")
[24120] *  Min threads: 5
[24120] *  Max threads: 5
[24120] *  Environment: production
[24120] *   Master PID: 24120
[24120] *      Workers: 2
[24120] *     Restarts: (✔) hot (✖) phased
[24120] * Preloading application
[24120] * Listening on http://0.0.0.0:3000
[24120] Use Ctrl-C to stop
I, [2023-09-21T13:20:03.516036 #24245]  INFO -- : Installing Puma worker loop.
[24120] - Worker 0 (PID: 24245) booted in 0.0s, phase: 0
I, [2023-09-21T13:20:03.517641 #24246]  INFO -- : Installing Puma worker loop.
[24120] - Worker 1 (PID: 24246) booted in 0.0s, phase: 0
I, [2023-09-21T13:20:08.541142 #24249]  INFO -- : Installing Puma worker loop.
[24120] - Worker 1 (PID: 24249) booted in 0.0s, phase: 0
I, [2023-09-21T13:20:08.541951 #24250]  INFO -- : Installing Puma worker loop.
[24120] - Worker 0 (PID: 24250) booted in 0.0s, phase: 0

However, looks like Scout APM will detect when shutting down, so the real culprit is the workers option not allowing Puma to shut down.

Versions

Desktop (please complete the following information):

• OS: MacOS Monterrey 12.6
• Puma Version 6.4.0

[davidalejandroaguilar]

Tagging related PR for idle-timeout (thanks for this!) #3209

[nateberkopec]

😓 That totally makes sense.

What's the desired behavior here? Let's say you have a cluster mode puma with 4 workers. After 5 seconds of inactivity, should the master kill everything and shut itself down?

[davidalejandroaguilar]

Well, I expected Puma to completely shut itself down when using this feature.

Though perhaps if combined with some new flag such as min-workers, it'd remain alive with only n workers, and scale up when a request comes. Effectively scaling up and down.

[joshuay03]

My bad, I didn't consider cluster mode. I've got a failing test to pass with a very rough but not ideal solution. I'll get a PR up as soon as I can. There's already the implementation to gracefully shut down all workers and finally the master process, which we can use if a single worker times out. The tricky part is sending a signal to the master process from the server, which is where the timeout was implemented.

Update: Got this up with a better solution than what I started with.