USR2 restart kills master / worker processes with prune_bundler option #550

ghost · 2014-07-01T17:55:22Z

Going to see if I can maybe track the issue down myself, but figure I'd open this issue up in the meantime.

In the first example, with the prune_bundler option in the config, the process dies after the USR2 signal, giving the error that's listed below. Once this option is removed from config, it restarts properly with this signal (see the second example).

$ cat config/puma.rb
threads 0, 4
workers 2
prune_bundler

$ #  bundle exec puma -C config/puma.rb
$ ps -eo pid,cmd | grep '[p]uma 2.8.2'
19478 puma 2.8.2 (tcp://0.0.0.0:9292)

$ kill -USR2 19478

[19478] - Gracefully shutting down workers...
[19478] * Restarting...
/home/foo/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/dependency.rb:298:in `to_specs': Could not find '-C' (>= 0) among 492 total gem(s) (Gem::LoadError)

#---------------------------------
$ cat config/puma.rb
threads 0, 4
workers 2

$ #  bundle exec puma -C config/puma.rb
$ ps -eo pid,cmd | grep '[p]uma 2.8.2'
19594 puma 2.8.2 (tcp://0.0.0.0:9292)

$ kill -USR2 19594  # successful

[19594] Puma starting in cluster mode...
[19594] * Version 2.8.2 (ruby 2.1.2-p95), codename: Sir Edmund Percival Hillary
[19594] * Min threads: 0, max threads: 4

fred · 2014-07-04T11:31:08Z

Im also having this issue in production. Looks like it's a serious issue.

When I running pumactl command like this

bundle exec pumactl -S /app/shared/tmp/pids/puma.state restart

==> puma.stderr.log <==
/home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/dependency.rb:298:in `to_specs': Could not find '-C' (>= 0) among 339 total gem(s) (Gem::LoadError)
    from /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/dependency.rb:309:in `to_spec'
    from /home/ubuntu/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:53:in `gem'
    from /home/ubuntu/www/hipflat/shared/bundle/ruby/2.1.0/gems/puma-2.8.2/bin/puma-wild:10:in `block in <main>'
    from /home/ubuntu/www/hipflat/shared/bundle/ruby/2.1.0/gems/puma-2.8.2/bin/puma-wild:8:in `each'
    from /home/ubuntu/www/hipflat/shared/bundle/ruby/2.1.0/gems/puma-2.8.2/bin/puma-wild:8:in `<main>'

Then puma fails to load.

fred · 2014-07-04T11:38:35Z

But when using phased-restart I dont see the Bundle error.

bundle exec pumactl -S /app/shared/tmp/pids/puma.state phased-restart

tbuehlmann · 2014-07-10T14:19:02Z

Joining the party with a similar problem. With and without prune_bundler, executing bundle exec pumactl -S /home/deploy/application/alpha/shared/puma.state restart will kill Puma. When redirecting errors, this is what I get:

/home/deploy/application/alpha/shared/bundle/ruby/2.1.0/gems/puma-2.8.2/bin/puma-wild:8:in

': undefined method split' for nil:NilClass (NoMethodError)

Using phased-restart seems to work.

Edit: This was using Puma 2.8.2. When using Puma from master I get the same error message as @fred.

evanphx · 2014-07-13T01:10:00Z

This should be fixed in 2.9.0. If not, please reopen.

ghost · 2014-07-14T00:31:45Z

Great, thanks @evanphx . Not seeing this issue with 2.9.0

fred · 2014-07-14T05:32:41Z

thanks for the fix, working great now.
cheers.

tbuehlmann · 2014-07-14T08:40:09Z

I would like to reopen this as I still get an error with prune_bundler. This is my puma.rb:

env = 'alpha'
user = 'user'
project = 'project'

environment env
daemonize
pidfile "/home/#{user}/applications/#{project}/#{env}/shared/puma.pid"
state_path "/home/#{user}/applications/#{project}/#{env}/shared/puma.state"
workers 4
threads 2,4
bind "unix:///home/#{user}/applications/#{project}/#{env}/shared/puma.sock"
prune_bundler
stdout_redirect "/home/#{user}/applications/#{project}/#{env}/shared/log/puma_stdout.log", "/home/#{user}/applications/#{project}/#{env}/shared/log/puma_stderr.log", true

When I issue a restart command using pumactl the process dies because of this:

/opt/rubies/ruby-2.1.2/lib/ruby/2.1.0/rubygems/dependency.rb:298:in `to_specs': Could not find '-C' (>= 0) among 98 total gem(s) (Gem::LoadError)
  from /opt/rubies/ruby-2.1.2/lib/ruby/2.1.0/rubygems/dependency.rb:309:in `to_spec'
  from /opt/rubies/ruby-2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_gem.rb:53:in `gem'
  from /home/user/applications/project/alpha/shared/bundle/ruby/2.1.0/gems/puma-2.9.0/bin/puma-wild:10:in `block in <main>'
  from /home/user/applications/project/alpha/shared/bundle/ruby/2.1.0/gems/puma-2.9.0/bin/puma-wild:8:in `each'
  from /home/user/applications/project/alpha/shared/bundle/ruby/2.1.0/gems/puma-2.9.0/bin/puma-wild:8:in `<main>'

This might not get a reopen, will open a fresh issue. :)

bmorton · 2014-10-29T23:26:34Z

We're seeing the same exact thing on 2.9.1 still. If we remove prune_bundler, things work fine. Tested on both Ruby 2.0.0-p481 and 2.1.4 with the same result as above.

@tbuehlmann did you ever get a resolution on this?

evanphx · 2014-10-29T23:32:22Z

I guess the prune_bundler restart is still having problems...

evanphx · 2014-10-29T23:34:13Z

@bmorton How are you starting puma?

bmorton · 2014-10-30T00:18:19Z

We're using upstart to bring it up/down and sending USR2 to hot restart. Here's the config for that: https://gist.github.com/bmorton/adb2488cca9a62db61c9

evanphx · 2014-10-30T00:19:56Z

@bmorton I found the bug and just released 2.9.2 with the fix. Could you try it out and see if everything looks ok?

bmorton · 2014-10-30T00:27:23Z

Oh sweet, thanks for the quick turnaround! I'll give it a go and report back.

bmorton · 2014-10-30T00:34:36Z

Looks like that fixes it... thanks again! 👍

tbuehlmann · 2014-10-30T07:26:48Z

Looks okay for me, too. 💛

xiaoronglv · 2015-07-24T03:27:45Z

We're seeing the same exact thing on 2.9.2 still. If we remove prune_bundler, things work fine.

environment

ruby 2.2.0
puma 2.9.2

that's my project

https://github.com/xiaoronglv/searchapp

cd /home/webapp/www/searchapp/current && ( RACK_ENV=production ~/.rvm/bin/rvm 2.2.0 do bundle exec puma -C /home/webapp/www/searchapp/shared/puma.rb --daemon )
* Pruning Bundler environment
/home/webapp/.rvm/rubies/ruby-2.2.0/lib/ruby/site_ruby/2.2.0/rubygems/dependency.rb:315:in `to_specs': Could not find 'rack' (= 1.6.4) among 15 total gem(s) (Gem::LoadError)
Checked in 'GEM_PATH=/home/webapp/.rvm/gems/ruby-2.2.0:/home/webapp/.rvm/gems/ruby-2.2.0@global', execute `gem env` for more information
    from /home/webapp/.rvm/rubies/ruby-2.2.0/lib/ruby/site_ruby/2.2.0/rubygems/dependency.rb:324:in `to_spec'
    from /home/webapp/.rvm/rubies/ruby-2.2.0/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:64:in `gem'
    from /home/webapp/www/searchapp/shared/bundle/ruby/2.2.0/gems/puma-2.9.2/bin/puma-wild:20:in `block in <main>'
    from /home/webapp/www/searchapp/shared/bundle/ruby/2.2.0/gems/puma-2.9.2/bin/puma-wild:18:in `each'
    from /home/webapp/www/searchapp/shared/bundle/ruby/2.2.0/gems/puma-2.9.2/bin/puma-wild:18:in `<main>'

#!/usr/bin/env puma

directory '/home/webapp/www/searchapp/current'
rackup "/home/webapp/www/searchapp/current/config.ru"
environment 'production'

pidfile "/home/webapp/www/searchapp/shared/tmp/pids/puma.pid"
state_path "/home/webapp/www/searchapp/shared/tmp/pids/puma.state"
stdout_redirect '/home/webapp/www/searchapp/shared/log/puma_access.log', '/home/webapp/www/searchapp/shared/log/puma_error.log', true


threads 4,8

bind 'unix:///home/webapp/www/searchapp/shared/tmp/sockets/puma.sock'

workers 1



prune_bundler


on_restart do
  puts 'Refreshing Gemfile'
  ENV["BUNDLE_GEMFILE"] = "/home/webapp/www/searchapp/current/Gemfile"
end

richardrails · 2015-08-13T11:06:14Z

update it to latest 2.12.3 and it works

evanphx closed this as completed Jul 13, 2014

evanphx reopened this Oct 29, 2014

evanphx closed this as completed in e44ced1 Oct 30, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

USR2 restart kills master / worker processes with prune_bundler option #550

USR2 restart kills master / worker processes with prune_bundler option #550

ghost commented Jul 1, 2014

fred commented Jul 4, 2014

fred commented Jul 4, 2014

tbuehlmann commented Jul 10, 2014

evanphx commented Jul 13, 2014

ghost commented Jul 14, 2014

fred commented Jul 14, 2014

tbuehlmann commented Jul 14, 2014

bmorton commented Oct 29, 2014

evanphx commented Oct 29, 2014

evanphx commented Oct 29, 2014

bmorton commented Oct 30, 2014

evanphx commented Oct 30, 2014

bmorton commented Oct 30, 2014

bmorton commented Oct 30, 2014

tbuehlmann commented Oct 30, 2014

xiaoronglv commented Jul 24, 2015

richardrails commented Aug 13, 2015

USR2 restart kills master / worker processes with prune_bundler option #550

USR2 restart kills master / worker processes with prune_bundler option #550

Comments

ghost commented Jul 1, 2014

fred commented Jul 4, 2014

fred commented Jul 4, 2014

tbuehlmann commented Jul 10, 2014

evanphx commented Jul 13, 2014

ghost commented Jul 14, 2014

fred commented Jul 14, 2014

tbuehlmann commented Jul 14, 2014

bmorton commented Oct 29, 2014

evanphx commented Oct 29, 2014

evanphx commented Oct 29, 2014

bmorton commented Oct 30, 2014

evanphx commented Oct 30, 2014

bmorton commented Oct 30, 2014

bmorton commented Oct 30, 2014

tbuehlmann commented Oct 30, 2014

xiaoronglv commented Jul 24, 2015

richardrails commented Aug 13, 2015