New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infinite loop when run from cron job with not-yet-trusted rvmrc? #791

Closed
simonrussell opened this Issue Feb 28, 2012 · 17 comments

Comments

Projects
None yet
4 participants
@simonrussell
Contributor

simonrussell commented Feb 28, 2012

Running RVM 1.10.2 on Ubuntu 10.04.4.

I've got a cron job setup which doesn't mention RVM; there is an .rvmrc in the project. When cron runs, the system ends up falling over because pretty much an infinite number of my cron job gets spawned. This only seems to happen if the .rvmrc hasn't ever been trusted -- if you trust then reset the trust, the problem doesn't occur. It's only if it's never been trusted.

Not sure what's causing the infinite respawn -- cron, or rvm -- but it's quite alarming. Any ideas? It may well not be RVM, but I'm not really sure where to start looking, and I'd be unhappy if this caused pain for anyone else.

(yes, I know I should have trusted the rvmrc -- I forgot; I don't think this should be the outcome however.)

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Feb 28, 2012

Member

@simonrussell can you gist your script and the way you run it with cron ?

Member

mpapis commented Feb 28, 2012

@simonrussell can you gist your script and the way you run it with cron ?

@ghost ghost assigned mpapis Feb 28, 2012

@simonrussell

This comment has been minimized.

Show comment
Hide comment
@simonrussell

simonrussell Feb 28, 2012

Contributor

Cron line is:

0,15,30,45 * * * * /bin/bash -l -c 'cd /home/users/username/appname/releases/20120228064002 && RAILS_ENV=production bundle exec script/grabber -s >> /home/users/username/appname/releases/20120228064002/log/cron.log 2>&1'

If I run that exact line, I get prompted to trust .rvmrc. So it's something to do with how cron runs the script. (The script never actually runs, so I've left it out. Shebang in script is:

#!/usr/bin/env ruby

I've replicated this on a VM by copying the relevant bits from our server onto a pretty much stock Ubuntu 10.04.4 Server install. I haven't tried a fresh install of RVM + deploy to see what happens there.

Contributor

simonrussell commented Feb 28, 2012

Cron line is:

0,15,30,45 * * * * /bin/bash -l -c 'cd /home/users/username/appname/releases/20120228064002 && RAILS_ENV=production bundle exec script/grabber -s >> /home/users/username/appname/releases/20120228064002/log/cron.log 2>&1'

If I run that exact line, I get prompted to trust .rvmrc. So it's something to do with how cron runs the script. (The script never actually runs, so I've left it out. Shebang in script is:

#!/usr/bin/env ruby

I've replicated this on a VM by copying the relevant bits from our server onto a pretty much stock Ubuntu 10.04.4 Server install. I haven't tried a fresh install of RVM + deploy to see what happens there.

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Feb 28, 2012

Member

@simonrussell trust db (file) is stored per user do you run it from the users cron that trusted it or via root ?

also there are now available few options in head rvm that do not require trusting: https://gist.github.com/1912050#gistcomment-86575

Member

mpapis commented Feb 28, 2012

@simonrussell trust db (file) is stored per user do you run it from the users cron that trusted it or via root ?

also there are now available few options in head rvm that do not require trusting: https://gist.github.com/1912050#gistcomment-86575

@simonrussell

This comment has been minimized.

Show comment
Hide comment
@simonrussell

simonrussell Feb 28, 2012

Contributor

Sorry, yeah, that's a user cron job.

My worry isn't so much that it needs trusting (which is fine), it's more how it fails -- it spawned so many processes the server died. I was sort of expecting an error message.

Contributor

simonrussell commented Feb 28, 2012

Sorry, yeah, that's a user cron job.

My worry isn't so much that it needs trusting (which is fine), it's more how it fails -- it spawned so many processes the server died. I was sort of expecting an error message.

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Feb 28, 2012

Member

you should not use bash -l in cron job, it's a bad idea, check other possible ways to run script from cron: https://rvm.beginrescueend.com/integration/cron/

Member

mpapis commented Feb 28, 2012

you should not use bash -l in cron job, it's a bad idea, check other possible ways to run script from cron: https://rvm.beginrescueend.com/integration/cron/

@simonrussell

This comment has been minimized.

Show comment
Hide comment
@simonrussell

simonrussell Feb 28, 2012

Contributor

Hmm, that's what the 'whenever' gem is giving me. I can probably hack it to do something else. Given we're getting kicked off the server (for killing it), I'm probably just going to go to a system-wide manual install though. Perhaps the RVM magics shouldn't be on there.

Do you think there actually something in RVM that is causing this loop though?

Contributor

simonrussell commented Feb 28, 2012

Hmm, that's what the 'whenever' gem is giving me. I can probably hack it to do something else. Given we're getting kicked off the server (for killing it), I'm probably just going to go to a system-wide manual install though. Perhaps the RVM magics shouldn't be on there.

Do you think there actually something in RVM that is causing this loop though?

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Feb 28, 2012

Member

no i think it's mix of the shell and using bash -l it should never be used in cron (https://rvm.beginrescueend.com/support/faq/#shell_login) you should be fine with using any of the mentioned ways to run cron jobs listed here: https://rvm.beginrescueend.com/integration/cron/

Member

mpapis commented Feb 28, 2012

no i think it's mix of the shell and using bash -l it should never be used in cron (https://rvm.beginrescueend.com/support/faq/#shell_login) you should be fine with using any of the mentioned ways to run cron jobs listed here: https://rvm.beginrescueend.com/integration/cron/

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Feb 28, 2012

Member

I have reported this as an issue in whenever javan/whenever#216

Member

mpapis commented Feb 28, 2012

I have reported this as an issue in whenever javan/whenever#216

@simonrussell

This comment has been minimized.

Show comment
Hide comment
@simonrussell

simonrussell Feb 28, 2012

Contributor

I should mention that I've been running a cron job a similar way with a trusted rvmrc with success on a different app. The fact that the bug only appears for a not-yet-trusted rvmrc would sort of imply it's something to do with RVM. The bash -l thing mightn't be ideal, but it surely shouldn't cause this to occur.

Anyway, I'm not really helping much; I should just find the source of the bug and do a pull request. It might be a while though...

It might be a good idea to add a fat warning to the cron page.

Contributor

simonrussell commented Feb 28, 2012

I should mention that I've been running a cron job a similar way with a trusted rvmrc with success on a different app. The fact that the bug only appears for a not-yet-trusted rvmrc would sort of imply it's something to do with RVM. The bash -l thing mightn't be ideal, but it surely shouldn't cause this to occur.

Anyway, I'm not really helping much; I should just find the source of the bug and do a pull request. It might be a while though...

It might be a good idea to add a fat warning to the cron page.

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Feb 28, 2012

Member

adding the warning should be easy - https://github.com/wayneeseguin/rvm-site/blob/master/content/integration/cron.haml and "Edit this page"

the bug is not inside of rvm - it's caused by using bash -l inside of cron.

As for your issue I really encourage you to try one of the new project files described here: https://gist.github.com/1912050#gistcomment-86575 - they do not require trusting

Member

mpapis commented Feb 28, 2012

adding the warning should be easy - https://github.com/wayneeseguin/rvm-site/blob/master/content/integration/cron.haml and "Edit this page"

the bug is not inside of rvm - it's caused by using bash -l inside of cron.

As for your issue I really encourage you to try one of the new project files described here: https://gist.github.com/1912050#gistcomment-86575 - they do not require trusting

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Mar 6, 2012

Member

it looks you can disable -l with whenever javan/whenever#216 (comment)

Member

mpapis commented Mar 6, 2012

it looks you can disable -l with whenever javan/whenever#216 (comment)

@mpapis mpapis closed this Mar 6, 2012

@simonrussell

This comment has been minimized.

Show comment
Hide comment
@simonrussell

simonrussell Mar 6, 2012

Contributor

Thanks for your help. I'm not actually running RVM in production anymore, which is probably the best solution to this problem. Given the way the problem manifests, it would seem likely to me that this is actually a bug in RVM that could be fixed, but given I don't have time to track it down and you're not so worried about it, I guess this bug report can be left as documentation for future unfortunate souls.

Contributor

simonrussell commented Mar 6, 2012

Thanks for your help. I'm not actually running RVM in production anymore, which is probably the best solution to this problem. Given the way the problem manifests, it would seem likely to me that this is actually a bug in RVM that could be fixed, but given I don't have time to track it down and you're not so worried about it, I guess this bug report can be left as documentation for future unfortunate souls.

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Mar 7, 2012

Member

the issue happens because of shell is started with sourcing initialization files again and again, and while sourcing it loads rvm again and again, it's hard to reproduce it because it never happens in normal shell (I wrote code to protect it from it), but somehow when it's used in scripting/cron it happens to overcome my protections, as this is not reproducible in standard running (really) interactive shell and I see no way that should actually happening - my only suspicion is wrong use of shell - just saying.

Member

mpapis commented Mar 7, 2012

the issue happens because of shell is started with sourcing initialization files again and again, and while sourcing it loads rvm again and again, it's hard to reproduce it because it never happens in normal shell (I wrote code to protect it from it), but somehow when it's used in scripting/cron it happens to overcome my protections, as this is not reproducible in standard running (really) interactive shell and I see no way that should actually happening - my only suspicion is wrong use of shell - just saying.

@serpentinefire

This comment has been minimized.

Show comment
Hide comment
@serpentinefire

serpentinefire Jan 13, 2013

I'm having the same problem with rvm 1.17.7, Ruby 1.9.3-p326 and Rails 3.1.10 on FreeBSD. A cron job that calls a rake task spawns infinite mutants.

With a previous version of rvm, Ruby 1.9.3-p0 and Rails 3.1.3, this problem did not occur.

Here's the crontab that causes infinite spawning:
0 0 * * * source ~/.bash_profile; cd /path/to/app/current; RAILS_ENV=production bundle exec rake app:activate

The first process looks normal:
user 24991 0.0 0.0 8940 1260 ? Ss 23:33 0:00 /bin/sh -c source ~/.bash_profile; cd /path/to/app/current; RAILS_ENV=production bundle exec rake app:activate

The mutant spawned processes are not using the version of the bundle command loaded by the bash_profile:
user 25022 0.0 0.0 9408 1456 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate
user 25025 0.0 0.0 9408 1456 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate
user 25028 0.0 0.0 9408 1452 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate
user 25031 0.0 0.0 9408 1456 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate

serpentinefire commented Jan 13, 2013

I'm having the same problem with rvm 1.17.7, Ruby 1.9.3-p326 and Rails 3.1.10 on FreeBSD. A cron job that calls a rake task spawns infinite mutants.

With a previous version of rvm, Ruby 1.9.3-p0 and Rails 3.1.3, this problem did not occur.

Here's the crontab that causes infinite spawning:
0 0 * * * source ~/.bash_profile; cd /path/to/app/current; RAILS_ENV=production bundle exec rake app:activate

The first process looks normal:
user 24991 0.0 0.0 8940 1260 ? Ss 23:33 0:00 /bin/sh -c source ~/.bash_profile; cd /path/to/app/current; RAILS_ENV=production bundle exec rake app:activate

The mutant spawned processes are not using the version of the bundle command loaded by the bash_profile:
user 25022 0.0 0.0 9408 1456 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate
user 25025 0.0 0.0 9408 1456 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate
user 25028 0.0 0.0 9408 1452 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate
user 25031 0.0 0.0 9408 1456 ? S 23:33 0:00 bash /web/sites/user/.rvm/bin/bundle exec rake app:activate

@mpapis

This comment has been minimized.

Show comment
Hide comment
@mpapis

mpapis Jan 13, 2013

Member

@serpentinefire as I already explained before the problem is in sourcing rvm in cron, change your task to:

0 0 * * * /path/to/rvm/bin/rvm in /path/to/app/current do env RAILS_ENV=production bundle exec rake app:activate
Member

mpapis commented Jan 13, 2013

@serpentinefire as I already explained before the problem is in sourcing rvm in cron, change your task to:

0 0 * * * /path/to/rvm/bin/rvm in /path/to/app/current do env RAILS_ENV=production bundle exec rake app:activate
@serpentinefire

This comment has been minimized.

Show comment
Hide comment
@serpentinefire

serpentinefire Jan 13, 2013

Thanks. Following the example here:
https://rvm.io/integration/cron/

I made this change, which works:
0 0 * * * cd /path/to/app/current; RAILS_ENV=production /path/to/user/.rvm/bin/rake-ruby-1.9.3-p362@global app:activate

serpentinefire commented Jan 13, 2013

Thanks. Following the example here:
https://rvm.io/integration/cron/

I made this change, which works:
0 0 * * * cd /path/to/app/current; RAILS_ENV=production /path/to/user/.rvm/bin/rake-ruby-1.9.3-p362@global app:activate

@hangmiao

This comment has been minimized.

Show comment
Hide comment
@hangmiao

hangmiao Jul 19, 2017

Super late for the party but thought I'd share b/c it took us quite a while to get this resolved.
In our case, it's the extra BASH_ENV souring specified in cron_tab that causes the issue:

# In cron file
SHELL=/bin/bash
BASH_ENV=~/.bashrc <------------------- this causes endless spawning
...

Taking that out works. Hope this helps.

hangmiao commented Jul 19, 2017

Super late for the party but thought I'd share b/c it took us quite a while to get this resolved.
In our case, it's the extra BASH_ENV souring specified in cron_tab that causes the issue:

# In cron file
SHELL=/bin/bash
BASH_ENV=~/.bashrc <------------------- this causes endless spawning
...

Taking that out works. Hope this helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment