Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

rack-cache 1.0 + rack master + rails master issues on production env #23

Closed
slbug opened this Issue · 30 comments
@slbug

I posted this issue to rack, but got answer that problems might be in rack-cache.

can you please check - https://github.com/rack/rack/issues/#issue/105

@jfirebaugh

Background: Rack::Lock was recently changed to hold the lock until #close is called on the response body (https://github.com/rack/rack/issues/closed#issue/87).

Rails edge puts Rack::Lock downstream of rack-cache in the default middleware chain.

Rack::Cache::Context sometimes overwrites the response body without calling #close on the original body, thus stranding the lock. For example, in the case that it misses, and fetches a non-cacheable? response that is not_modified?, it calls response.#not_modified! which sets the body to [].

@rtomayko
Owner

Interesting. Sorry, I didn't receive a notification for this issue for some reason. Thanks guys. I'll take a look.

@d3r1v3d

I have the exact same problem using Redhat 5.6 + Apache + Passenger in addition to Ubuntu 10.10 + Nginx + Passenger.

Exact same issue in both places with this (Exception ThreadError... deadlock, recursive locking...) problem.

@buckyflowers

Same issue here. Ubuntu 10.04 + Apache + Passenger+rack-cache 1.0

[ pid=16496 thr=3075607184 file=ext/apache2/Hooks.cpp:821 time=2011-03-31 23:09:02.126 ]: The backend application (process 16672) did not 
send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing b
ugs in this application.
[ pid=16672 thr=76844430 file=utils.rb:176 time=2011-03-31 23:09:02.127 ]: *** Exception ThreadError in application (deadlock; recursive l
ocking) (process 16672, thread #):
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rack-1a5a44ab518c/lib/rack/lock.rb:33:in `lock'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rack-1a5a44ab518c/lib/rack/lock.rb:33:in `call'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:129:in `forward'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:235:in `fetch'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:178:in `lookup'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:65:in `call!'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:50:in `call'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rails-1bedee56314f/railties/lib/rails/engine.rb:422:in `cal
l'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rails-1bedee56314f/railties/lib/rails/railtie/configurable.
rb:28:in `method_missing'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request
'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and
_process_next_request'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:205:in `start_requ
est_handler'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:170:in `block in h
andle_spawn_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/utils.rb:479:in `safe_fork'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:165:in `handle_spa
wn_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously
'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:180:in `start'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:128:in `start'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in s
pawn_rack_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_o
r_add'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_a
pplication'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in 
synchronize'
        from :10:in `synchronize'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchroni
ze'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_applicatio
n'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_applicat
ion'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously
'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/helper-scripts/passenger-spawn-server:99:in `'
@buckyflowers

I uncommented this line in /config/environments/production.rb in my Rails app:

config.threadsafe!

The issue seems to go away with this, but I don't know if it is the fix.

@rtomayko
Owner

This should be fixed in 93846f3:

close body from downstream on not modified and head responses

This should hopefully fix #23 -- Rack::Lock and other middleware
rely on the body being closed to signal the end of the request
cycle.

@rtomayko rtomayko closed this
@cfis cfis referenced this issue from a commit
@rtomayko close body from downstream on not modified and head responses
This should hopefully fix #23 -- Rack::Lock and other middleware
rely on the body being closed to signal the end of the request
cycle.
93846f3
@Bodacious

I'm still having this issue ~

Rails 3.1.0.rc4
Rack 1.3.0

Any idea when the patch is going to be included in the version Rails depends on?

@tc
tc commented

+1, also seeing this on the same config as bodacious.

stack trace on rack/rack#100

@jfirebaugh

@Bodacious, @tc: What version of rack-cache are you using?

The bug is supposed to be fixed in 1.0.1 or later.

@tc
tc commented

rack-cache 1.0.2
here's the stack trace

[ pid=31235 thr=75796900 file=utils.rb:176 time=2011-07-13 22:53:07.173 ]: *** Exception ThreadError in application (deadlock; recursive locking) (process 31235, thread #<Thread:0x9092348>):
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-1.3.0/lib/rack/lock.rb:33:in `lock'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-1.3.0/lib/rack/lock.rb:33:in `call'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:132:in `forward'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:243:in `fetch'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:181:in `lookup'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:65:in `call!'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:50:in `call'
@Bodacious
@sanchitg

I am also using rack-cache 1.0.2 (along with nginx + passenger-3.0.7/lib/phusion_passenger + rack-1.3.2 + railties-3.1.0.rc4 + Linux li283-243 2.6.39.1-linode34) and received same problem. Here are the logs:

2011/08/12 00:00:54 [error] 27052#0: 1580 upstream prematurely closed connection while reading response header from upstream, client: 66.249.72.23, server: beta.cheesplex.com, request: "GET /cheese/cheese_type/70/country/1/milk_type/8 HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "beta.cheeseplex.com"
[ pid=5813 thr=70623890 file=utils.rb:176 time=2011-08-12 00:00:54.861 ]: *
* Exception ThreadError in application (deadlock; recursive locking) (process 5813, thread #Thread:0x86b4524):
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-1.3.2/lib/rack/lock.rb:14:in lock'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-1.3.2/lib/rack/lock.rb:14:in
call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:132:in forward'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:243:in
fetch'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:181:in lookup'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:65:in
call!'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:50:in call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/hoptoad_notifier-2.4.11/lib/hoptoad_notifier/user_informer.rb:12:in
call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc4/lib/rails/engine.rb:438:in call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc4/lib/rails/railtie/configurable.rb:30:in
method_missing'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/request_handler.rb:96:in process_request'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_request_handler.rb:513:in
accept_and_process_next_request'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_request_handler.rb:274:in main_loop'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:205:in
start_request_handler'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:170:in block in handle_spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:479:in
safe_fork'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:165:in handle_spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:180:in
start'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:128:in start'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:253:in
block (2 levels) in spawn_rack_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:132:in lookup_or_add'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:246:in
block in spawn_rack_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:82:in block in synchronize'
from <internal:prelude>:10:in
synchronize'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:79:in synchronize'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:244:in
spawn_rack_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:137:in spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:275:in
handle_spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/helper-scripts/passenger-spawn-server:99:in `

'

@tc
tc commented

Just tested on nginx 1.05/passenger 3.0.8/ruby 1.9.2-p290/rails 3.1.rc5/rack-1.3.2 and the problem still persists.

@itsterry

Yes. I'm seeing it also in rack-cache 1.0.2/rails 3.1.rc5 and ruby 1.9.2-p290 with passenger 3.0.7 and apache2. Should this issue be reopened?

@rtomayko rtomayko reopened this
@rtomayko
Owner

Sorry, can you guys confirm that removing rack-cache from the middleware pipeline causes the issue to go away? The same Rack::Lock error will happen if any component doesn't close the body properly.

@tc
tc commented

Rails 3.1 is shipping with rack-cache, is there a way to easily disable it?

@rtomayko
Owner

Good question. Let me see what I can dig up.

@itsterry

I disabled caching in my environment/production.rb and the problem went away

FROM
config.action_controller.perform_caching = true
TO
config.action_controller.perform_caching = false

@itsterry

Although having done the above, my app is running like a dog. Any news on disabling rack-cache altogether ?

@albus522

Are you using airbrake?

@itsterry

Yes, but it's not an app issue, so Airbrake's not catching anything unfortunately

@Bodacious
@albus522

Airbrake::UserInformer is throwing out the old body likely causing your issue.

add "config.user_information = nil" to your airbrake or hoptoad config and your problem will likely go away

Unless your are actually using the UserInformer in which case you will have to wait for a fix.

@rtomayko
Owner

This sounds like an issue with other middleware in the pipeline. Please reopen if you can tie the issue to rack-cache specifically.

@rtomayko rtomayko closed this
@phuongnd08

@albus522: I'm getting the same error and following your advise doesn't solve mine.
Here my gems:

airbrake (3.0.4)
rack-cache (1.0.3)
rails (3.1.0)
sass (3.1.7)
sass-rails (3.1.2)
sprockets (2.0.0)

Here is what I get from the nginx error.log:

cache: [GET /assets/dolphin.jpg] miss
[ pid=7854 thr=70414960 file=utils.rb:176 time=2011-09-28 23:27:14.750 ]: *** Exception IOError in application (closed stream) (process 7854, thread #Thread:0x864e4e0):
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-1.3.3/lib/rack/body_proxy.rb:12:in close'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:136:in
process_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
from internal:prelude:10:in synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in

'
cache: [GET /] miss
cache error: Permission denied - /home/root/giasu_app/releases/20110928163844/tmp/cache/A89
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in
fu_mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:217:in block (2 levels) in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in
reverse_each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in block in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in
each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in mkdir_p'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:173:in
ensure_cache_path'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:100:in write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/strategy/local_cache.rb:140:in
write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:364:in block in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:520:in
instrument'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:362:in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/actionpack-3.1.0/lib/action_dispatch/http/rack_cache.rb:56:in
write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/metastore.rb:60:in store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:264:in
store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:257:in fetch'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:181:in
lookup'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:65:in call!'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
internal:prelude:10:in synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
cache: [GET /assets/application.css] miss
2011/09/28 23:27:15 [error] 5914#0: *231 upstream prematurely closed connection while reading response header from upstream, client: 116.118.43.44, server: localhost, request: "GET /assets/application.css HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "www.giasudientu.com", referrer: "http://www.giasudientu.com/"
[ pid=5891 thr=3074763664 file=ext/nginx/HelperAgent.cpp:931 time=2011-09-28 23:27:15.718 ]: Uncaught exception in PassengerServer client thread:
exception: Cannot read response from backend process: Connection reset by peer (104)
backtrace:
in 'void Client::forwardResponse(Passenger::SessionPtr&, Passenger::FileDescriptor&, const Passenger::AnalyticsLogPtr&)' (HelperAgent.cpp:603)
in 'void Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:857)
in 'void Client::threadMain()' (HelperAgent.cpp:950)

2011/09/28 23:27:15 [error] 5914#0: 258 upstream prematurely closed connection while reading response header from upstream, client: 116.118.43.44, server: localhost, request: "GET /assets/application.js HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "www.giasudientu.com", referrer: "http://www.giasudientu.com/"
[ pid=7862 thr=70414960 file=utils.rb:176 time=2011-09-28 23:27:15.718 ]: *
* Exception IOError in application (closed stream) (process 7862, thread #Thread:0x864e4e0):
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-1.3.3/lib/rack/body_proxy.rb:12:in close'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/response.rb:230:in
not_modified!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:85:in call!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
from internal:prelude:10:in synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in

'
cache error: Permission denied - /home/root/giasu_app/releases/20110928163844/tmp/cache/B11
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in
fu_mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:217:in block (2 levels) in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in
reverse_each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in block in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in
each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in mkdir_p'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:173:in
ensure_cache_path'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:100:in write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/strategy/local_cache.rb:140:in
write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:364:in block in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:520:in
instrument'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:362:in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/actionpack-3.1.0/lib/action_dispatch/http/rack_cache.rb:56:in
write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/metastore.rb:60:in store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:264:in
store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:257:in fetch'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:181:in
lookup'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:65:in call!'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
internal:prelude:10:in synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
cache: [GET /assets/dolphin.jpg] miss
[ pid=7874 thr=70414960 file=utils.rb:176 time=2011-09-28 23:27:15.771 ]: *** Exception IOError in application (closed stream) (process 7874, thread #Thread:0x864e4e0):
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-1.3.3/lib/rack/body_proxy.rb:12:in close'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/response.rb:230:in
not_modified!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:85:in call!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
from internal:prelude:10:in synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
2011/09/28 23:27:15 [error] 5914#0: *231 upstream prematurely closed connection while reading response header from upstream, client: 116.118.43.44, server: localhost, request: "GET /assets/dolphin.jpg HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "www.giasudientu.com", referrer: "http://www.giasudientu.com/"

@phuongnd08

Don't know why it keep reading cache from /tmp/cache/A89 while all caches are located under /tmp/cache/assets

@phuongnd08

I added config.assets.digest to production.rb to switch over to application-[digest-code].css instead of application.css and get over the rack-cache miss error. Still, if I'm trying to request to application.css then 1 out of every 2 requests will result in a 502 bad gate error

@phuongnd08

Hmm, change /current/tmp folder permission to 777 seem to solve the error. So this must be the nginx worker running as nobody user failed to read and write to /current/tmp/cache folder.

@staii

Problem still occurs with rails 3.2.13, rack-cache 1.2, and running using passenger 4.0.25.

Disabling caching by setting config.action_controller.perform_caching = false stops the error from occurring.

@bobbus

I believe I'm also seeing this problem on rails 3.2, rack 1.4 and unicorn 4.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.