Pumactl restart not working

[deathbob]

#409

I don't think this problem every really went away for me, and I've spent today trying to fix it, with no luck.
It is present for me in 2.7.1, 2.9.0, and master branch.

Again, the fact that restart says "Command restart sent success" but the actual Puma process is down is pretty much horrible. It might be true that the command was sent successfully, but I don't care about that, I care if the #@?! Puma process has in fact restarted. The definition of success when you ask something to restart is if it comes back up. This is something Apache and Nginx understand, and they do report a failure if attempts to restart them fail.

Puma really has to block on this, and report failure when it happens.

For now I overrode the built in capistrano restart task to simply call stop and start. Not really a great solution.

[deathbob]

Puma 2.9.0, ruby 2.0.0p195
Also tried with Puma 2.7.1, and master.

environment 'production'
daemonize true

app_path = '/home/deploy/toro/current'

directory app_path
stdout_redirect "#{app_path}/log/puma.stdout.log", "#{app_path}/log/puma.stderr.log"
workers 8
threads 1,2
bind "unix://#{app_path}/tmp/sockets/puma.sock"
state_path  "#{app_path}/tmp/pids/puma.state"
pidfile     "#{app_path}/tmp/pids/puma.pid"

activate_control_app "unix://#{app_path}/tmp/sockets/pumactl.sock", { no_token: true }

[SaimonL]

I am not using any capistrano at the moment and the way I restart puma is to run this bash command from the application directory. Works every time, however if you did bundle update or any Gemfile changes and then puma instance may not restart successfully (rare).

#!/bin/bash
kill -s SIGUSR2 $(cat tmp/pids/puma.pid)

I see you are doing

workers 8

Which I think it means you are doing cluster server which I have no experience in. You will have to wait for some one from Puma to reply.

[vlmonk]

same problem with puma 2.9.0, ruby 2.0.0p481
puma complete ignore pumactl restart and kill -USR2 <pumapid>

my config:

directory "/home/deploy/app/current"
environment "production"
daemonize true
pidfile  "/home/deploy/app/shared/puma/puma.pid"
state_path "/home/deploy/app/shared/puma/puma.state"
threads 0, 8
bind "unix:///home/deploy/app/shared/puma/puma.sock"
workers 0

[rhinon]

I'm having a similar issue with puma 2.9.0 and jruby-1.7.13
Pretty much exactly the same as what is described in #436

[jmagoon]

It looks like pumactl isn't checking for a response from the Process.kill('SIGUSR2',pid). I don't know enough about the Process library, but I'm not sure if it can. For me, Process.kill('SIGUSR2',pid) returns 1 whether or not puma actually does a restart. Doing a system call %x(kill -s SIGUSR2 <pid>) always returns an exit status of 0, whether puma restarts or crashes.

Success:

irb(main):010:0> Process.kill('SIGUSR2',3922)
=> 1
[3922] - Gracefully shutting down workers...
[3922] * Restarting...
[3922] Puma starting in cluster mode...
[3922] * Version 2.8.2 (ruby 2.1.2-p95), codename: Sir Edmund Percival Hillary

Failure:

irb(main):012:0> Process.kill('SIGUSR2',3922)
=> 1
[3922] - Gracefully shutting down workers...
[3922] * Restarting...
/Users/jmagoon/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/puma-2.8.2/lib/puma/cli.rb:424:in `fileno': closed stream (IOError)
    from /Users/jmagoon/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/puma-2.8.2/lib/puma/cli.rb:424:in `block in restart!'
    from /Users/jmagoon/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/puma-2.8.2/lib/puma/cli.rb:423:in `each'

My current hacky workaround is to try to restart, check if the PID is running, and if not, try a cold start. If that fails, then I can handle the error (neither Process nor a system call return any puma errors).

[pencilcheck]

This is the command I used, and it works all the time kill -s USR2 <pid>
But USR1 always starts new process with the old path and thus it is essentially not working at all.
I'm running puma 2.9.0 and ruby 2.1.1

[SaimonL]

kill -s USR2 

Used to take down the server but would not start it back up. But now it works for me, puma gracefully restarts the rails server by killing old threads one at a a time and then starting new ones. I tested it in CentOS 7 (64-Bit) and in Debain 7.7 (64-Bit) and both works.
Using puma 2.9.2

[urkle]

this is not working at all under RHEL 6 64-bit..
puma 2.12.1 on ruby 2.2.1.
any time I do a pumactl restart, phased-restart, or use the kill -USR2 / kill -USR1, puma simply exits. This is rather disappointing that a core piece functionality is severely broken in puma.

When I start puma (non-daemonized) in one terminal (pumactl start) and restart it from another terminal (pumactl phased-restart) here is the output I get.

[2856] Puma starting in cluster mode...
[2856] * Version 2.12.1 (ruby 2.2.1-p85), codename: Plutonian Photo Shoot
[2856] * Min threads: 5, max threads: 5
[2856] * Environment: production
[2856] * Process workers: 2
[2856] * Preloading application
[2856] * Listening on tcp://0.0.0.0:3000
[2856] Use Ctrl-C to stop
[2856] - Worker 0 (pid: 2863) booted, phase: 0
[2856] - Worker 1 (pid: 2868) booted, phase: 0
[2856] * phased-restart called but not available, restarting normally.
[2856] - Gracefully shutting down workers...
[2856] * Restarting...
Available commands: halt, restart, phased-restart, start, stats, status, stop, reload-worker-directory

why would it spit out "available commands" like that ???
.. time passes..
Well, it seems I found the issue.. if I start the server with puma instead of pumactl it restarts correctly!! So it seems somewhere in the code the restart_command is being incorrectly calculated when using pumactl to start the app

[evanphx]

I'm sorry about that, I should just remove the start functionality from pumctl. It provides an additional unnecessary vector for confusion.

[evanphx]

And another vector for bugs. In fact, I should just reimplement it in terms of exec'ing normal puma.

[evanphx]

@urkle what commands were you running? You didn't paste them in.

[urkle]

I was running
pumactl start and pumactl phased-restart
to start the server and now am changing it to puma

It would be better to reimplement it in terms of exec'ing puma, as right now (from others on this bug) running pumactl restart will start puma if it's not running (though presumably via the wrong command)
As for my configuration

workers Integer(ENV['WEB_CONCURRENCY'] || 2)
threads_count = Integer(ENV['MAX_THREADS'] || 5)
threads threads_count, threads_count

preload_app!
daemonize

rackup      DefaultRackup
port        ENV['PORT'] || 3000
environment ENV['RAILS_ENV'] || 'development'
state_path  './tmp/puma.state'
pidfile     './tmp/pids/puma.pid'
activate_control_app

[urkle]

Any progress in getting this properly fixed?

[isaiah]

@urkle In the readme it says:

Note that preload_app can’t be used with phased restart, since phased restart kills and restarts workers one-by-one, and preload_app is all about copying the code of master into the workers.

@evanphx Maybe you should put that in a big banner :)

[urkle]

Regardless of that, the issue still remains pumactl start sets the wrong process name so restarting does not function.

[tonytonyjan]

I encountered the same problem with the config below (no preload_app!):

threads 8, 32
workers 4
bind 'tcp://0.0.0.0:3000'
daemonize

ruby 2.2.3p173
puma version 2.14.0

[localhostdotdev]

For future Googlers like me, now it's:

#!/bin/bash
kill -s SIGUSR2 $(cat tmp/pids/server.pid)

(credit to @SaimonL)

[omenking]

It's 2021 and I still can't get pumactl stop to work as expected.

I guess I'll just kill as @localhostdotdev suggests.

[cjlarose]

A bunch of stuff has changed in puma since this issue was first opened. @omenking if you're experiencing a similar problem in Puma 5, please feel free to open a new issue.