restart.txt not reloading #84

Closed
ryanb opened this Issue Aug 29, 2012 · 14 comments

Comments

Projects
None yet
3 participants
@ryanb

ryanb commented Aug 29, 2012

I am having difficulty getting touch tmp/restart.txt to reload changes to the application. No error is being reported, it is just serving the old application. I am using:

Ubuntu 12.04
JRuby 1.7.0-preview2
Java 7 (installed through openjdk-7-jdk package)
Trinidad 1.4.1
Rails 3.2.8
Capistrano 2.13.3

There is no trinidad.yml file so all configuration is at its default (running with -e production). Threadsafe mode is not enabled in the Rails app. The trinidad.log file shows that it recognized restart.txt and reloaded the application, however the changes do not show up until I issue a full restart of Trinidad.

It looks like others have experienced this issue on the mailing list here and here but no solution was given.

This may have something to do with using Capistrano and it not properly detecting the updated current symlink? However the restart.txt is in the new release so it is detecting that properly. Also the application does not respond for several sections after running the command so it appears to be restarting the Rails app.

@kares

This comment has been minimized.

Show comment Hide comment
@kares

kares Aug 29, 2012

Member

Hey Ryan, I've tried reproducing with a simple Rails 3.2.8 application with the same version of Trinidad as you did.
I changed a controller file and it got correctly updated after the reloading was done. Tested with following JRubies :

jruby 1.7.0.preview2.dev (1.9.3p203) 2012-08-03 6f4a2a2 on Java HotSpot(TM) Server VM 1.6.0_32-b05 [linux-i386]
jruby 1.7.0.preview2.dev (1.9.3p203) 2012-08-03 6f4a2a2 on OpenJDK Server VM 1.7.0_147-icedtea-b147 [linux-i386]

for the record here's my Gemfile/Gemfile.lock https://gist.github.com/d3450ec829238e845225

It probably won't be capistrano related (might be a variant of a pending reloading issue #75), but would be great if we can rule this out. Thus could you try if it behaves the same if you change a say a controller file (or whatever was changed and not picked up when you last tried) and touch tmp/restart.txt manually (without doing a cap deploy) ? Thanks

Member

kares commented Aug 29, 2012

Hey Ryan, I've tried reproducing with a simple Rails 3.2.8 application with the same version of Trinidad as you did.
I changed a controller file and it got correctly updated after the reloading was done. Tested with following JRubies :

jruby 1.7.0.preview2.dev (1.9.3p203) 2012-08-03 6f4a2a2 on Java HotSpot(TM) Server VM 1.6.0_32-b05 [linux-i386]
jruby 1.7.0.preview2.dev (1.9.3p203) 2012-08-03 6f4a2a2 on OpenJDK Server VM 1.7.0_147-icedtea-b147 [linux-i386]

for the record here's my Gemfile/Gemfile.lock https://gist.github.com/d3450ec829238e845225

It probably won't be capistrano related (might be a variant of a pending reloading issue #75), but would be great if we can rule this out. Thus could you try if it behaves the same if you change a say a controller file (or whatever was changed and not picked up when you last tried) and touch tmp/restart.txt manually (without doing a cap deploy) ? Thanks

@ryanb

This comment has been minimized.

Show comment Hide comment
@ryanb

ryanb Aug 29, 2012

I tried manually changing the code on the server and running touch tmp/restart.txt and it does not load in the changes. This rules out Capistrano. There must be something else specific to my setup which is causing this problem. Any other ideas? Thanks for looking into this.

ryanb commented Aug 29, 2012

I tried manually changing the code on the server and running touch tmp/restart.txt and it does not load in the changes. This rules out Capistrano. There must be something else specific to my setup which is causing this problem. Any other ideas? Thanks for looking into this.

@ryanb

This comment has been minimized.

Show comment Hide comment
@ryanb

ryanb Aug 29, 2012

I did a bit more testing and found out this problem only happens when Trinidad starts up through the trinidad_init_services script. Is there a reason reloading would behave differently when run through jsvc?

ryanb commented Aug 29, 2012

I did a bit more testing and found out this problem only happens when Trinidad starts up through the trinidad_init_services script. Is there a reason reloading would behave differently when run through jsvc?

@kares

This comment has been minimized.

Show comment Hide comment
@kares

kares Aug 29, 2012

Member

that's a nice find, it should have behaved the same ... I'll definitely look into that.

Member

kares commented Aug 29, 2012

that's a nice find, it should have behaved the same ... I'll definitely look into that.

@kares

This comment has been minimized.

Show comment Hide comment
@kares

kares Aug 29, 2012

Member

interestingly, seems to somehow work for me with trinidad_init_services-1.1.6 and jsvc compiled on demand during trinidad_init_service, the only minor difference seems to be that I'm on 11.10 and maybe I have a slightly different JRuby 1.7.0 build (except for a different jsvc if you're using an apt installed one none of these should matter) - once again I've checked JDK 6 as well as apt available OpenJDK-7.
maybe you can share the generated init.d script so I can try to arrange things the exact same way as you, otherwise I'm out of ideas why this fails for you right now ...

Member

kares commented Aug 29, 2012

interestingly, seems to somehow work for me with trinidad_init_services-1.1.6 and jsvc compiled on demand during trinidad_init_service, the only minor difference seems to be that I'm on 11.10 and maybe I have a slightly different JRuby 1.7.0 build (except for a different jsvc if you're using an apt installed one none of these should matter) - once again I've checked JDK 6 as well as apt available OpenJDK-7.
maybe you can share the generated init.d script so I can try to arrange things the exact same way as you, otherwise I'm out of ideas why this fails for you right now ...

@ryanb

This comment has been minimized.

Show comment Hide comment
@ryanb

ryanb Aug 29, 2012

Hmm, good to know it's working for you. I emailed you access to the VPS I'm testing on so you can try it out yourself if you don't mind. Perhaps you can duplicate the problem on there. Thanks!

ryanb commented Aug 29, 2012

Hmm, good to know it's working for you. I emailed you access to the VPS I'm testing on so you can try it out yourself if you don't mind. Perhaps you can duplicate the problem on there. Thanks!

@ryanb

This comment has been minimized.

Show comment Hide comment
@ryanb

ryanb Aug 29, 2012

I just tried doing a rolling restart and it looks like that loaded correctly. So for some reason this issue is only present on the default "restart" strategy. The config file is at ~/apps/blog/shared/config/trinidad.yml if you want to experiment with the restart strategy on the VPS which I sent you access to.

ryanb commented Aug 29, 2012

I just tried doing a rolling restart and it looks like that loaded correctly. So for some reason this issue is only present on the default "restart" strategy. The config file is at ~/apps/blog/shared/config/trinidad.yml if you want to experiment with the restart strategy on the VPS which I sent you access to.

@ryanb

This comment has been minimized.

Show comment Hide comment
@ryanb

ryanb Aug 30, 2012

I am unable to duplicate this issue today which is very strange. I am wondering now if it had something to do with memory. Perhaps if Trinidad was running out of memory while restarting that might have strange side effects? That's the only thing I can think of which would cause the problem to be inconsistent.

Either way I'm closing this for now, and I'll reopen if the issue pops up again.

ryanb commented Aug 30, 2012

I am unable to duplicate this issue today which is very strange. I am wondering now if it had something to do with memory. Perhaps if Trinidad was running out of memory while restarting that might have strange side effects? That's the only thing I can think of which would cause the problem to be inconsistent.

Either way I'm closing this for now, and I'll reopen if the issue pops up again.

@ryanb ryanb closed this Aug 30, 2012

@kares

This comment has been minimized.

Show comment Hide comment
@kares

kares Aug 30, 2012

Member

Yeah that's weird but stuff like this happens to me from time to time, thus I perfectly understand ... :)
The memory might be an issue esp. under a load with a lot of requests waiting in the queue, it's definitely easier to have a memory requirement estimate with a single runtime (threadsafe! mode). On the other hand it happened with the default reload strategy which is quite more predictable in terms of memory than the rolling strategy (that's why it was changed as the default in 1.4).

Member

kares commented Aug 30, 2012

Yeah that's weird but stuff like this happens to me from time to time, thus I perfectly understand ... :)
The memory might be an issue esp. under a load with a lot of requests waiting in the queue, it's definitely easier to have a memory requirement estimate with a single runtime (threadsafe! mode). On the other hand it happened with the default reload strategy which is quite more predictable in terms of memory than the rolling strategy (that's why it was changed as the default in 1.4).

@ansel1

This comment has been minimized.

Show comment Hide comment
@ansel1

ansel1 Oct 15, 2012

I've hit the same issue. Ubuntu 12.04.1, jruby 1.6.7.2, using capistrano to deploy to trinidad, using trinidad-init-services, etc.

Definitely looks like trinidad isn't looking at the current symlink. It locks on the whatever current is pointing at initially. For example i noticed if I cap deploy (changing symlink), then touch current/tmp/restart.txt, nothing happens. But if I touch releases/[prior release]/tmp/restart.txt, trinidad reloads. But it only reloads releases/[prior release]. Doesn't pick up new release that current is pointing at.

ansel1 commented Oct 15, 2012

I've hit the same issue. Ubuntu 12.04.1, jruby 1.6.7.2, using capistrano to deploy to trinidad, using trinidad-init-services, etc.

Definitely looks like trinidad isn't looking at the current symlink. It locks on the whatever current is pointing at initially. For example i noticed if I cap deploy (changing symlink), then touch current/tmp/restart.txt, nothing happens. But if I touch releases/[prior release]/tmp/restart.txt, trinidad reloads. But it only reloads releases/[prior release]. Doesn't pick up new release that current is pointing at.

@kares

This comment has been minimized.

Show comment Hide comment
@kares

kares Oct 15, 2012

Member

hmm, that's maybe due expanding the current symlink in the (init.d) shell script. to be sure would be great to know more about you setup e.g. how the init.d script looks like (any customizations ?) and if there are any settings in the trinidad configuration file e.g. trinidad.yml thanks a lot. you can also test if this is "caused" by the init.d script if you (temporarily) start trinidad directly (not with the init.d script) and try redeploy-ing a new release ...

Member

kares commented Oct 15, 2012

hmm, that's maybe due expanding the current symlink in the (init.d) shell script. to be sure would be great to know more about you setup e.g. how the init.d script looks like (any customizations ?) and if there are any settings in the trinidad configuration file e.g. trinidad.yml thanks a lot. you can also test if this is "caused" by the init.d script if you (temporarily) start trinidad directly (not with the init.d script) and try redeploy-ing a new release ...

@ansel1

This comment has been minimized.

Show comment Hide comment
@ansel1

ansel1 Oct 15, 2012

Of course...I should have tried that. Will give it a go.

Haven't done any customizations of the generated trinidad init script. config is pretty much right from the examples. The only thing unusual is that I currently use a non-default restart.txt file, but that was just an attempt to fix the problem. Putting restart.txt outside the scope of the current symlink resolved the issue of trinidad not detected when restart.txt is touched, but it still ends up reloading the prior release instead of the current.

app_path: "/opt/trinidad/current"
trinidad_options: "-e production --monitor /opt/trinidad/restart.txt"
jruby_home: "/usr/local/ruby/jruby-1.6.7.2"
ruby_compat_version: RUBY1_9
trinidad_name: Trinidad
jsvc_path: "/usr/bin/jsvc"
java_home: "/usr/lib/jvm/default-java"
output_path: "/etc/init.d"
pid_file: "/opt/trinidad/shared/pids/trinidad.pid"
log_file: "/opt/trinidad/shared/log/trinidad.log"
run_user: "deploy"

ansel1 commented Oct 15, 2012

Of course...I should have tried that. Will give it a go.

Haven't done any customizations of the generated trinidad init script. config is pretty much right from the examples. The only thing unusual is that I currently use a non-default restart.txt file, but that was just an attempt to fix the problem. Putting restart.txt outside the scope of the current symlink resolved the issue of trinidad not detected when restart.txt is touched, but it still ends up reloading the prior release instead of the current.

app_path: "/opt/trinidad/current"
trinidad_options: "-e production --monitor /opt/trinidad/restart.txt"
jruby_home: "/usr/local/ruby/jruby-1.6.7.2"
ruby_compat_version: RUBY1_9
trinidad_name: Trinidad
jsvc_path: "/usr/bin/jsvc"
java_home: "/usr/lib/jvm/default-java"
output_path: "/etc/init.d"
pid_file: "/opt/trinidad/shared/pids/trinidad.pid"
log_file: "/opt/trinidad/shared/log/trinidad.log"
run_user: "deploy"
@ansel1

This comment has been minimized.

Show comment Hide comment
@ansel1

ansel1 Oct 15, 2012

K...so trinidad isn't reloading from current because trinidad seems to be ignoring the -d command line param. trinidad-init-services runs trinidad in /opt/trinidad/releases/[release], then runs trinidad, passing -d /opt/trinidad/current as an arg. I think trinidad ignores the -d, but runs anyway using cwd as root_dir.

So trinidad is always just monitoring and deploying from current dir.

I confirmed this on the server running trinidad manually, and double checked this on my workstation too. My project folder is:

/Users/ansel1/repositories/myrailsapp

I cd to /Users/ansel1/repositories, rbenv global jruby-1.6.7.2, and do:

trinidad -d /Users/ansel1/repositories -e production

trinidad starts, but when I hit my homepage, I get:

No such file to load -- /Users/ansel1/repositories/config/environment.rb from file:/Users/ansel1/.rbenv/versions/jruby-1.6.7.2/lib/ruby/gems/1.8/gems/jruby-rack-1.1.10/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails/environment2.rb:23:in `load_environment' from file:/Users/ansel1/.rbenv/versions/jruby-1.6.7.2/lib/ruby/gems/1.8/gems/jruby-rack-1.1.10/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails_booter.rb:65:in `load_environment' from

I ran trinidad in the debugger, and confirmed it is correctly parsing the root_dir option, but it doesn't seem to be using it.

ansel1 commented Oct 15, 2012

K...so trinidad isn't reloading from current because trinidad seems to be ignoring the -d command line param. trinidad-init-services runs trinidad in /opt/trinidad/releases/[release], then runs trinidad, passing -d /opt/trinidad/current as an arg. I think trinidad ignores the -d, but runs anyway using cwd as root_dir.

So trinidad is always just monitoring and deploying from current dir.

I confirmed this on the server running trinidad manually, and double checked this on my workstation too. My project folder is:

/Users/ansel1/repositories/myrailsapp

I cd to /Users/ansel1/repositories, rbenv global jruby-1.6.7.2, and do:

trinidad -d /Users/ansel1/repositories -e production

trinidad starts, but when I hit my homepage, I get:

No such file to load -- /Users/ansel1/repositories/config/environment.rb from file:/Users/ansel1/.rbenv/versions/jruby-1.6.7.2/lib/ruby/gems/1.8/gems/jruby-rack-1.1.10/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails/environment2.rb:23:in `load_environment' from file:/Users/ansel1/.rbenv/versions/jruby-1.6.7.2/lib/ruby/gems/1.8/gems/jruby-rack-1.1.10/lib/jruby-rack-1.1.10.jar!/jruby/rack/rails_booter.rb:65:in `load_environment' from

I ran trinidad in the debugger, and confirmed it is correctly parsing the root_dir option, but it doesn't seem to be using it.

@kares

This comment has been minimized.

Show comment Hide comment
@kares

kares Oct 15, 2012

Member

I see, thanks a lot I sure shall look into it ... in the meantime (haven't tried but should work) I can think of a work-around:
Set up a configuration file e.g. /opt/trinidad/shared/trinidad.yml with the following content :

---
  root_dir: /opt/trinidad/current
  # monitor is assumed relative to work_dir ~ defaults to [root_dir]/tmp
  #monitor: restart.txt

Than change the following option and regenerate the init.d script (or simply edit the approciate line in the shell script) :

trinidad_options: "-e production --config /opt/trinidad/shared/trinidad.yml"

Hope it helps ...

Member

kares commented Oct 15, 2012

I see, thanks a lot I sure shall look into it ... in the meantime (haven't tried but should work) I can think of a work-around:
Set up a configuration file e.g. /opt/trinidad/shared/trinidad.yml with the following content :

---
  root_dir: /opt/trinidad/current
  # monitor is assumed relative to work_dir ~ defaults to [root_dir]/tmp
  #monitor: restart.txt

Than change the following option and regenerate the init.d script (or simply edit the approciate line in the shell script) :

trinidad_options: "-e production --config /opt/trinidad/shared/trinidad.yml"

Hope it helps ...

@kares kares reopened this Oct 15, 2012

@kares kares closed this in 179c341 Oct 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment