Support Passenger 5 restarting

[Kriechi]

Passenger 5 introduces a new way to restart apps: passenger-config restart-app
More information over at their blog:
https://blog.phusion.nl/2015/03/04/whats-new-in-passenger-5-part-2-better-logging-better-restarting-better-websockets-and-more/

Would be nice to see this implemented in a nifty way.
(auto detecting Passenger 4 / 5?)

[betesh]

#4 was merged

[JamesChevalier]

This does not work for me, so I'm not sure if there's still a lingering issue or if something else in my environment is the cause... I'm going to try & give as much info as I can, but I've been working at this for at least an hour so it's likely that I've missed something - let me know if you need more information.

Software

ruby 2.1.5
rails 4.1.10
rvm 1.26.10
capistrano 3.4.0
capistrano-bundler 1.1.4
capistrano-ext 1.2.1
capistrano-passenger 0.0.3
capistrano-rails 1.1.2
capistrano-rvm 0.1.2
capistrano-sidekiq 0.5.2

config/deploy/production.rb

set :passenger_restart_command, 'passenger-config restart-app'
set :passenger_restart_options, -> { "#{deploy_to} --ignore-app-not-running" }

Output at end of deploy:

INFO [63733f58] Running /usr/bin/env passenger-config restart-app /home/user_name/app_name --ignore-app-not-running as user_name@host_name
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as user_name@host_name: passenger-config restart-app exit status: 1
passenger-config restart-app stdout: Nothing written
passenger-config restart-app stderr: Please pass either an app path prefix or an app group name. See --help for more information.

SSHKit::Command::Failed: passenger-config restart-app exit status: 1
passenger-config restart-app stdout: Nothing written
passenger-config restart-app stderr: Please pass either an app path prefix or an app group name. See --help for more information.

Tasks: TOP => passenger:restart
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as user_name@host_name: passenger-config restart-app exit status: 1
passenger-config restart-app stdout: Nothing written
passenger-config restart-app stderr: Please pass either an app path prefix or an app group name. See --help for more information.

It seems to take the config, since this is the command it's running to restart the app:

/usr/bin/env passenger-config restart-app /home/user_name/app_name --ignore-app-not-running

The error seems to indicate that it's not getting the /home/user_name/app_name part of the command.
If I log into the server as the user_name and try to run that manually, I get:

*** ERROR: You are not authorized to query the status for this

If I add that user to the sudoers group, and change my config to set :passenger_restart_command, 'rvmsudo passenger-config restart-app' it fails because it can't find rvmsudo.

If I try using /usr/local/rvm/bin/rvmsudo instead, that works if I run the whole /usr/bin/env ... command manually but through capistrano that just spits out this mess:

INFO [3fb8db9d] Running /usr/bin/env /usr/local/rvm/bin/rvmsudo passenger-config restart-app /home/user_name/app_name --ignore-app-not-running as user_name@host_name
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as user_name@host_name: /usr/local/rvm/bin/rvmsudo passenger-config restart-app exit status: 1
/usr/local/rvm/bin/rvmsudo passenger-config restart-app stdout: Nothing written
/usr/local/rvm/bin/rvmsudo passenger-config restart-app stderr: Warning: can not check /etc/sudoers for secure_path, falling back to call via /usr/bin/env, this breaks rules from /etc/sudoers. Run:

>export rvmsudo_secure_path=1

to avoid the warning, put it in shell initialization file to make it persistent.

In case there is no secure_path in /etc/sudoers. Run:

>export rvmsudo_secure_path=0

to avoid the warning, put it in shell initialization file to make it persistent.
Please pass either an app path prefix or an app group name. See --help for more information.

SSHKit::Command::Failed: /usr/local/rvm/bin/rvmsudo passenger-config restart-app exit status: 1
/usr/local/rvm/bin/rvmsudo passenger-config restart-app stdout: Nothing written
/usr/local/rvm/bin/rvmsudo passenger-config restart-app stderr: Warning: can not check /etc/sudoers for secure_path, falling back to call via /usr/bin/env, this breaks rules from /etc/sudoers. Run:

>export rvmsudo_secure_path=1

to avoid the warning, put it in shell initialization file to make it persistent.

In case there is no secure_path in /etc/sudoers. Run:

>export rvmsudo_secure_path=0

to avoid the warning, put it in shell initialization file to make it persistent.
Please pass either an app path prefix or an app group name. See --help for more information.

Tasks: TOP => passenger:restart
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as user_name@host_name: /usr/local/rvm/bin/rvmsudo passenger-config restart-app exit status: 1
/usr/local/rvm/bin/rvmsudo passenger-config restart-app stdout: Nothing written
/usr/local/rvm/bin/rvmsudo passenger-config restart-app stderr: Warning: can not check /etc/sudoers for secure_path, falling back to call via /usr/bin/env, this breaks rules from /etc/sudoers. Run:

>export rvmsudo_secure_path=1

to avoid the warning, put it in shell initialization file to make it persistent.

In case there is no secure_path in /etc/sudoers. Run:

>export rvmsudo_secure_path=0

to avoid the warning, put it in shell initialization file to make it persistent.
Please pass either an app path prefix or an app group name. See --help for more information.

If I could get this working via rvmsudo, I'd be happy - does anything here suggest that I'm pretty close to getting that working?

Edit

I've just rolled back to capistrano-passenger version 0.0.2, and the entire deploy went flawlessly. Obviously, still just restarting the app with /usr/bin/env touch /home/user_name/app_name/releases/20150326021332/tmp/restart.txt (if that event works any more).

One thing I noticed after going back to version 0.0.2 is that version 0.0.3 had stopped working with sidekiq entirely (wasn't stopping anything, wasn't starting anything), and it may have also been skipping the part where it gets rid of old releases (I say 'may have' because I don't recall its earlier behavior, but my 0.0.2 deploy removed a dozen old releases).

[betesh]

@JamesChevalier What version of passenger are you running? You say that capistrano-passenger 0.0.2 works for you, which would suggest you are using a version of passenger < 5, but capistrano-passenger is detecting version 5.

Is rvm installed at the system level or the user level? i.e. What's the output of cap rvm:hook?

Is passenger installed via rvm? Run which passenger on the server to see where it's installed.

Also, can you set log_level to debug so that we can see the exact command that the passenger:restart task sends to your app server? Sometimes at the debug level the command isn't exactly what it's reported to be at the info level.

As for sidekiq, I have no experience with it, but a quick look at the sourcecode makes me wonder if there's some weird interaction involving rvm_map_bins. Try adding this to your config/deploy.rb:

puts fetch(:rvm_map_bins).inspect

and see what the output is. It should contain passenger and sidekiq among others.

[VEkh]

I'm having the same exact problem and went through just about all the steps that @JamesChevalier did.

Reverting to capistrano-passenger (0.0.2) has also solved the problem for the time being.

[betesh]

@VEkh Can you address the same questions I asked @JamesChevalier so that we can figure this one out?

[kule]

Same issue for me - here's the answers to your questions:

What version of passenger are you running? 5.0.5
Is rvm installed at the system level or the user level? user level

i.e. What's the output of cap rvm:hook?

cap production rvm:hook
DEBUG [7a4b69f0] Running /usr/bin/env [ -d ~/.rvm ] as foo@bar.com
DEBUG [7a4b69f0] Command: [ -d ~/.rvm ]
DEBUG [7a4b69f0] Finished in 0.442 seconds with exit status 0 (successful).
DEBUG [7d1445c9] Running ~/.rvm/bin/rvm version as foo@bar.com
DEBUG [7d1445c9] Command: ~/.rvm/bin/rvm version
DEBUG [7d1445c9]  rvm 1.26.10 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
DEBUG [7d1445c9] Finished in 0.412 seconds with exit status 0 (successful).
rvm 1.26.10 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
DEBUG [9726298b] Running ~/.rvm/bin/rvm current as foo@bar.com
DEBUG [9726298b] Command: ~/.rvm/bin/rvm current
DEBUG [9726298b]  ruby-2.0.0-p598
DEBUG [9726298b] Finished in 0.389 seconds with exit status 0 (successful).
ruby-2.0.0-p598
DEBUG [0e1f48ab] Running ~/.rvm/bin/rvm default do ruby --version as foo@bar.com
DEBUG [0e1f48ab] Command: ~/.rvm/bin/rvm default do ruby --version
DEBUG [0e1f48ab]  ruby 2.0.0p598 (2014-11-13 revision 48408) [i686-linux]
DEBUG [0e1f48ab] Finished in 0.606 seconds with exit status 0 (successful).
ruby 2.0.0p598 (2014-11-13 revision 48408) [i686-linux]

Is passenger installed via rvm? Nope, using Ubuntu apt packages provided by Phusion
Run which passenger on the server to see where it's installed: /usr/bin/passenger

Also, can you set log_level to debug so that we can see the exact command that the passenger:restart task sends to your app server? Sometimes at the debug level the command isn't exactly what it's reported to be at the info level.

cap production deploy:restart
DEBUG [7024a687] Running /usr/bin/env [ -d ~/.rvm ] as foo@bar.com
DEBUG [7024a687] Command: [ -d ~/.rvm ]
DEBUG [7024a687] Finished in 0.515 seconds with exit status 0 (successful).
DEBUG [edcd411e] Running ~/.rvm/bin/rvm version as foo@bar.com
DEBUG [edcd411e] Command: ~/.rvm/bin/rvm version
DEBUG [edcd411e]  rvm 1.26.10 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
DEBUG [edcd411e] Finished in 0.356 seconds with exit status 0 (successful).
rvm 1.26.10 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
DEBUG [fcedb7ab] Running ~/.rvm/bin/rvm current as foo@bar.com
DEBUG [fcedb7ab] Command: ~/.rvm/bin/rvm current
DEBUG [fcedb7ab]  ruby-2.0.0-p598
DEBUG [fcedb7ab] Finished in 0.493 seconds with exit status 0 (successful).
ruby-2.0.0-p598
DEBUG [a34c7c94] Running ~/.rvm/bin/rvm default do ruby --version as foo@bar.com
DEBUG [a34c7c94] Command: ~/.rvm/bin/rvm default do ruby --version
DEBUG [a34c7c94]  ruby 2.0.0p598 (2014-11-13 revision 48408) [i686-linux]
DEBUG [a34c7c94] Finished in 0.642 seconds with exit status 0 (successful).
ruby 2.0.0p598 (2014-11-13 revision 48408) [i686-linux]
DEBUG [51429ba2] Running ~/.rvm/bin/rvm default do passenger -v as foo@bar.com
DEBUG [51429ba2] Command: ~/.rvm/bin/rvm default do passenger -v
DEBUG [51429ba2]  Phusion Passenger version 5.0.5
DEBUG [51429ba2]
DEBUG [51429ba2]  "Phusion Passenger" is a trademark of Hongli Lai & Ninh Bui.
DEBUG [51429ba2] Finished in 0.807 seconds with exit status 0 (successful).
INFO [bdaab3df] Running /usr/bin/env passenger-config restart-app /home/foo/bar.com --ignore-app-not-running as foo@bar.com
DEBUG [bdaab3df] Command: passenger-config restart-app
DEBUG [bdaab3df]  Please pass either an app path prefix or an app group name. See --help for more information.

[kule]

Ok think I've got this one sorted, it was just a permission issue so you need to update /etc/sudoers to allow restarting of passenger without a password.

Use visudo to edit the file (so if you mess up you have the chance to re-edit the file), at the end of the file (*) add the following:

deployuser    ALL=(root) NOPASSWD: /usr/bin/passenger-config

Then in your deploy.rb add

set :passenger_restart_with_sudo, true

(*) it needs to be the end of the file or after %sudo ALL=(ALL) ALL to take precedence. Alternatively you can uncomment #includedir /etc/sudoers.d and put the above line in a new file e.g. in /etc/sudoers.d/cap-passenger-restarts

[JamesChevalier]

Thanks for helping out, @betesh !
I'm now using capistrano-passenger version 0.0.4 (with the same error results).

What version of passenger are you running?

Passenger is installed through apt, and is at version 5.0.5-1~wheezy1

What's the output of cap rvm:hook?

DEBUG [f4ee54bd] Running /usr/bin/env [ -d ~/.rvm ] as user_name@host_name
DEBUG [f4ee54bd] Command: [ -d ~/.rvm ]
DEBUG [f4ee54bd] Finished in 0.716 seconds with exit status 1 (failed).
DEBUG [244acd26] Running /usr/bin/env [ -d /usr/local/rvm ] as user_name@host_name
DEBUG [244acd26] Command: [ -d /usr/local/rvm ]
DEBUG [244acd26] Finished in 0.059 seconds with exit status 0 (successful).
DEBUG [ed92f84d] Running /usr/local/rvm/bin/rvm version as user_name@host_name
DEBUG [ed92f84d] Command: /usr/local/rvm/bin/rvm version
DEBUG [ed92f84d]  rvm 1.26.10 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
DEBUG [ed92f84d] Finished in 0.468 seconds with exit status 0 (successful).
rvm 1.26.10 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
DEBUG [defe16fe] Running /usr/local/rvm/bin/rvm current as user_name@host_name
DEBUG [defe16fe] Command: /usr/local/rvm/bin/rvm current
DEBUG [defe16fe]  ruby-2.1.5
DEBUG [defe16fe] Finished in 0.608 seconds with exit status 0 (successful).
ruby-2.1.5
DEBUG [cfd12633] Running /usr/local/rvm/bin/rvm default do ruby --version as user_name@host_name
DEBUG [cfd12633] Command: /usr/local/rvm/bin/rvm default do ruby --version
DEBUG [cfd12633]  ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]
DEBUG [cfd12633] Finished in 1.026 seconds with exit status 0 (successful).
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-linux]

Is passenger installed via rvm? Run which passenger on the server to see where it's installed.

/usr/bin/passenger

Also, can you set log_level to debug so that we can see the exact command that the passenger:restart task sends to your app server?

INFO [71692a0a] Running /usr/bin/env mv /home/user_name/app_name/releases/current /home/user_name/app_name as user_name@host_name
DEBUG [71692a0a] Command: /usr/bin/env mv /home/user_name/app_name/releases/current /home/user_name/app_name
INFO [71692a0a] Finished in 0.062 seconds with exit status 0 (successful).
DEBUG [6059ba5f] Running /usr/local/rvm/bin/rvm default do passenger -v as user_name@host_name
DEBUG [6059ba5f] Command: /usr/local/rvm/bin/rvm default do passenger -v
DEBUG [6059ba5f]  Phusion Passenger version 5.0.5
DEBUG [6059ba5f]
DEBUG [6059ba5f]  "Phusion Passenger" is a trademark of Hongli Lai & Ninh Bui.
DEBUG [6059ba5f] Finished in 0.714 seconds with exit status 0 (successful).
INFO [2ad6c54d] Running /usr/bin/env passenger-config restart-app /home/user_name/app_name --ignore-app-not-running as user_name@host_name
DEBUG [2ad6c54d] Command: passenger-config restart-app
DEBUG [2ad6c54d]  Please pass either an app path prefix or an app group name. See --help for more information.
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as user_name@host_name: passenger-config restart-app exit status: 1
passenger-config restart-app stdout: Nothing written
passenger-config restart-app stderr: Please pass either an app path prefix or an app group name. See --help for more information.

SSHKit::Command::Failed: passenger-config restart-app exit status: 1
passenger-config restart-app stdout: Nothing written
passenger-config restart-app stderr: Please pass either an app path prefix or an app group name. See --help for more information.

Tasks: TOP => passenger:restart
(See full trace by running task with --trace)
The deploy has failed with an error: Exception while executing as user_name@host_name: passenger-config restart-app exit status: 1
passenger-config restart-app stdout: Nothing written
passenger-config restart-app stderr: Please pass either an app path prefix or an app group name. See --help for more information.

[zjx20]

strange failure with capistrano-passenger (0.0.4):

INFO [fe538cd4] Running /usr/bin/env passenger-config restart-app /hd01/webapp/AFSSite/ --ignore-app-not-running as joe@10.0.1.91
DEBUG [fe538cd4] Command: passenger-config restart-app
DEBUG [fe538cd4]        Please pass either an app path prefix or an app group name. See --help for more information.

but the command is fine when I ran it on the server manually.

[betesh]

@zjx20 I think you're the first to spot why this is happening--@JamesChevalier's output contained the same thing:

INFO [2ad6c54d] Running /usr/bin/env passenger-config restart-app /home/user_name/app_name --ignore-app-not-running as user_name@host_name
DEBUG [2ad6c54d] Command: passenger-config restart-app

And so did @kule's. It looks like we're not really passing the full command. Note that this error message is only triggered when your console session is not a TTY.

I need to do some more digging on this to figure out why it's outputting the correct command at INFO level but not sending the full command to the server.

[betesh]

I'm putting this issue on hold for now, because of phusion/passenger#1392
As soon as Passenger 5 supports a way to restart the app without sudo, this will get much easier. In the mean time, it would be difficult to describe in the README how to add your deployment user to visudo and try to cover all cases--RVM, Rbenv, etc., and there's no way to make this work Out-Of-The-Box.

[kule]

@betesh Incase it's helpful; when I've set this up using sudo the command runs correctly i.e.

INFO [8f29dabf] Running /usr/bin/env sudo passenger-config restart-app /home/foo/bar.com --ignore-app-not-running as foo@bar.com
DEBUG [8f29dabf] Command: /usr/bin/env sudo passenger-config restart-app /home/foo/bar.com --ignore-app-not-running

Not sure exactly where execute is run (I'm presuming SSHKit?) but perhaps it's something to do with here: passenger.cap does execute expect a Symbol as the first argument (rather than the string "passenger-config restart-app") to use the rest of the args?

[betesh]

@kule You're on the right track. When the first arg to execute is a Symbol, sshkit looks up that Symbol in the command map and supplies it with whatever prefixes have been configured.

That's how we're able to configure rvm_map_bins and then safely call any ruby or rake command and rely on sshkit to take care of running the command in the right RVM context.

However, there is currently no way to restart an app using passenger 5 without sudo, which is an obstacle that RVM usually mitigates but not in this case. I tried making passenger-config and restart-app 2 Symbols instead of a String, so that it runs in the RVM environment, and the result is an error from passenger-config something like: You are not authorized to query the status for this Phusion Passenger instance. Trying with sudo or with rvmsudo involves making changes to visudo that I don't want to start recommending to everyone, and documenting all of that will hopefully be unnecessary soon, if phusion address phusion/passenger#1392 promptly. At that point, we will have to change passenger-config to a Symbol, but the solution proposed at phusion/passenger#1392 will require a little more work than that, so I prefer to wait and see how they address it before trying to integrate with it.

[kule]

@betesh makes sense thanks for the explanation :)

[betesh]

@JamesChevalier @zjx20 @kule I think #8 fixed your issue. Still waiting for phusion/passenger#1392 before addressing the sudo issue, but this should get you a step closer, so that @kule's solution of editing visudo will probably work for you now.

[FooBarWidget]

I've just finished implementing a change in Passenger which would allow passenger-config restart-app to work without sudo. It is expected that this change lands in 5.0.10. See phusion/passenger#1392 (comment)

[FooBarWidget]

Aaand it's live: https://blog.phusion.nl/2015/06/11/phusion-passenger-5-0-10/

[FooBarWidget]

There were a number of bugs in the sudo-less restart mechanism in 5.0.10. They will be addressed in 5.0.11, which will be released soon. Please check the changelog for details: https://github.com/phusion/passenger/blob/stable-5.0/CHANGELOG