Skip to content
This repository

rack-cache 1.0 + rack master + rails master issues on production env #23

Closed
slbug opened this Issue · 30 comments
bUg.

I posted this issue to rack, but got answer that problems might be in rack-cache.

can you please check - https://github.com/rack/rack/issues/#issue/105

John Firebaugh

Background: Rack::Lock was recently changed to hold the lock until #close is called on the response body (https://github.com/rack/rack/issues/closed#issue/87).

Rails edge puts Rack::Lock downstream of rack-cache in the default middleware chain.

Rack::Cache::Context sometimes overwrites the response body without calling #close on the original body, thus stranding the lock. For example, in the case that it misses, and fetches a non-cacheable? response that is not_modified?, it calls response.#not_modified! which sets the body to [].

Ryan Tomayko
Owner

Interesting. Sorry, I didn't receive a notification for this issue for some reason. Thanks guys. I'll take a look.

d3r1v3d

I have the exact same problem using Redhat 5.6 + Apache + Passenger in addition to Ubuntu 10.10 + Nginx + Passenger.

Exact same issue in both places with this (Exception ThreadError... deadlock, recursive locking...) problem.

Bucky Flowers

Same issue here. Ubuntu 10.04 + Apache + Passenger+rack-cache 1.0


[ pid=16496 thr=3075607184 file=ext/apache2/Hooks.cpp:821 time=2011-03-31 23:09:02.126 ]: The backend application (process 16672) did not 
send a valid HTTP response; instead, it sent nothing at all. It is possible that it has crashed; please check whether there are crashing b
ugs in this application.
[ pid=16672 thr=76844430 file=utils.rb:176 time=2011-03-31 23:09:02.127 ]: *** Exception ThreadError in application (deadlock; recursive l
ocking) (process 16672, thread #):
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rack-1a5a44ab518c/lib/rack/lock.rb:33:in `lock'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rack-1a5a44ab518c/lib/rack/lock.rb:33:in `call'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:129:in `forward'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:235:in `fetch'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:178:in `lookup'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:65:in `call!'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/gems/rack-cache-1.0/lib/rack/cache/context.rb:50:in `call'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rails-1bedee56314f/railties/lib/rails/engine.rb:422:in `cal
l'
        from /var/www/vhosts/beta_thelifeichoose/rails/ruby/1.9.1/bundler/gems/rails-1bedee56314f/railties/lib/rails/railtie/configurable.
rb:28:in `method_missing'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request
'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and
_process_next_request'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:205:in `start_requ
est_handler'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:170:in `block in h
andle_spawn_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/utils.rb:479:in `safe_fork'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:165:in `handle_spa
wn_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously
'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:180:in `start'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/rack/application_spawner.rb:128:in `start'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:253:in `block (2 levels) in s
pawn_rack_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_o
r_add'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:246:in `block in spawn_rack_a
pplication'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server_collection.rb:82:in `block in 
synchronize'
        from :10:in `synchronize'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchroni
ze'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_applicatio
n'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_applicat
ion'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously
'
        from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.5/helper-scripts/passenger-spawn-server:99:in `'
Bucky Flowers

I uncommented this line in /config/environments/production.rb in my Rails app:

config.threadsafe!

The issue seems to go away with this, but I don't know if it is the fix.

Ryan Tomayko
Owner

This should be fixed in 93846f3:

close body from downstream on not modified and head responses

This should hopefully fix #23 -- Rack::Lock and other middleware
rely on the body being closed to signal the end of the request
cycle.

Ryan Tomayko rtomayko closed this
Charlie Savage cfis referenced this issue from a commit
Ryan Tomayko close body from downstream on not modified and head responses
This should hopefully fix #23 -- Rack::Lock and other middleware
rely on the body being closed to signal the end of the request
cycle.
93846f3
Gavin Morrice

I'm still having this issue ~

Rails 3.1.0.rc4
Rack 1.3.0

Any idea when the patch is going to be included in the version Rails depends on?

Tommy Chheng
tc commented

+1, also seeing this on the same config as bodacious.

stack trace on rack/rack#100

John Firebaugh

@Bodacious, @tc: What version of rack-cache are you using?

The bug is supposed to be fixed in 1.0.1 or later.

Tommy Chheng
tc commented

rack-cache 1.0.2
here's the stack trace

[ pid=31235 thr=75796900 file=utils.rb:176 time=2011-07-13 22:53:07.173 ]: *** Exception ThreadError in application (deadlock; recursive locking) (process 31235, thread #<Thread:0x9092348>):
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-1.3.0/lib/rack/lock.rb:33:in `lock'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-1.3.0/lib/rack/lock.rb:33:in `call'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:132:in `forward'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:243:in `fetch'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:181:in `lookup'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:65:in `call!'
    from /data/apps/production/app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:50:in `call'
Gavin Morrice
Sanchit

I am also using rack-cache 1.0.2 (along with nginx + passenger-3.0.7/lib/phusion_passenger + rack-1.3.2 + railties-3.1.0.rc4 + Linux li283-243 2.6.39.1-linode34) and received same problem. Here are the logs:

2011/08/12 00:00:54 [error] 27052#0: 1580 upstream prematurely closed connection while reading response header from upstream, client: 66.249.72.23, server: beta.cheesplex.com, request: "GET /cheese/cheese_type/70/country/1/milk_type/8 HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "beta.cheeseplex.com"
[ pid=5813 thr=70623890 file=utils.rb:176 time=2011-08-12 00:00:54.861 ]: **
Exception ThreadError in application (deadlock; recursive locking) (process 5813, thread #Thread:0x86b4524):
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-1.3.2/lib/rack/lock.rb:14:in lock'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-1.3.2/lib/rack/lock.rb:14:in
call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:132:in forward'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:243:in
fetch'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:181:in lookup'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:65:in
call!'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.2/lib/rack/cache/context.rb:50:in call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/hoptoad_notifier-2.4.11/lib/hoptoad_notifier/user_informer.rb:12:in
call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc4/lib/rails/engine.rb:438:in call'
from /var/www/cheeseplex/shared/bundle/ruby/1.9.1/gems/railties-3.1.0.rc4/lib/rails/railtie/configurable.rb:30:in
method_missing'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/request_handler.rb:96:in process_request'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_request_handler.rb:513:in
accept_and_process_next_request'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_request_handler.rb:274:in main_loop'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:205:in
start_request_handler'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:170:in block in handle_spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/utils.rb:479:in
safe_fork'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:165:in handle_spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:180:in
start'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/rack/application_spawner.rb:128:in start'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:253:in
block (2 levels) in spawn_rack_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:132:in lookup_or_add'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:246:in
block in spawn_rack_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:82:in block in synchronize'
from <internal:prelude>:10:in
synchronize'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server_collection.rb:79:in synchronize'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:244:in
spawn_rack_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:137:in spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/spawn_manager.rb:275:in
handle_spawn_application'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/passenger-3.0.7/helper-scripts/passenger-spawn-server:99:in `'

Tommy Chheng
tc commented

Just tested on nginx 1.05/passenger 3.0.8/ruby 1.9.2-p290/rails 3.1.rc5/rack-1.3.2 and the problem still persists.

Terry S

Yes. I'm seeing it also in rack-cache 1.0.2/rails 3.1.rc5 and ruby 1.9.2-p290 with passenger 3.0.7 and apache2. Should this issue be reopened?

Ryan Tomayko rtomayko reopened this
Ryan Tomayko
Owner

Sorry, can you guys confirm that removing rack-cache from the middleware pipeline causes the issue to go away? The same Rack::Lock error will happen if any component doesn't close the body properly.

Tommy Chheng
tc commented

Rails 3.1 is shipping with rack-cache, is there a way to easily disable it?

Ryan Tomayko
Owner

Good question. Let me see what I can dig up.

Terry S

I disabled caching in my environment/production.rb and the problem went away

FROM
config.action_controller.perform_caching = true
TO
config.action_controller.perform_caching = false

Terry S

Although having done the above, my app is running like a dog. Any news on disabling rack-cache altogether ?

David Genord II

Are you using airbrake?

Terry S

Yes, but it's not an app issue, so Airbrake's not catching anything unfortunately

Gavin Morrice
David Genord II

Airbrake::UserInformer is throwing out the old body likely causing your issue.

add "config.user_information = nil" to your airbrake or hoptoad config and your problem will likely go away

Unless your are actually using the UserInformer in which case you will have to wait for a fix.

Ryan Tomayko
Owner

This sounds like an issue with other middleware in the pipeline. Please reopen if you can tie the issue to rack-cache specifically.

Ryan Tomayko rtomayko closed this
Phuong Nguyen

@albus522: I'm getting the same error and following your advise doesn't solve mine.
Here my gems:

airbrake (3.0.4)
rack-cache (1.0.3)
rails (3.1.0)
sass (3.1.7)
sass-rails (3.1.2)
sprockets (2.0.0)

Here is what I get from the nginx error.log:

cache: [GET /assets/dolphin.jpg] miss
[ pid=7854 thr=70414960 file=utils.rb:176 time=2011-09-28 23:27:14.750 ]: *** Exception IOError in application (closed stream) (process 7854, thread #Thread:0x864e4e0):
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-1.3.3/lib/rack/body_proxy.rb:12:in close'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:136:in
process_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
from internal:prelude:10:in synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
cache: [GET /] miss
cache error: Permission denied - /home/root/giasu_app/releases/20110928163844/tmp/cache/A89
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in
fu_mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:217:in block (2 levels) in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in
reverse_each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in block in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in
each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in mkdir_p'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:173:in
ensure_cache_path'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:100:in write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/strategy/local_cache.rb:140:in
write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:364:in block in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:520:in
instrument'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:362:in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/actionpack-3.1.0/lib/action_dispatch/http/rack_cache.rb:56:in
write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/metastore.rb:60:in store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:264:in
store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:257:in fetch'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:181:in
lookup'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:65:in call!'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
internal:prelude:10:in synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
cache: [GET /assets/application.css] miss
2011/09/28 23:27:15 [error] 5914#0: *231 upstream prematurely closed connection while reading response header from upstream, client: 116.118.43.44, server: localhost, request: "GET /assets/application.css HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "www.giasudientu.com", referrer: "http://www.giasudientu.com/"
[ pid=5891 thr=3074763664 file=ext/nginx/HelperAgent.cpp:931 time=2011-09-28 23:27:15.718 ]: Uncaught exception in PassengerServer client thread:
exception: Cannot read response from backend process: Connection reset by peer (104)
backtrace:
in 'void Client::forwardResponse(Passenger::SessionPtr&, Passenger::FileDescriptor&, const Passenger::AnalyticsLogPtr&)' (HelperAgent.cpp:603)
in 'void Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:857)
in 'void Client::threadMain()' (HelperAgent.cpp:950)

2011/09/28 23:27:15 [error] 5914#0: 258 upstream prematurely closed connection while reading response header from upstream, client: 116.118.43.44, server: localhost, request: "GET /assets/application.js HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "www.giasudientu.com", referrer: "http://www.giasudientu.com/"
[ pid=7862 thr=70414960 file=utils.rb:176 time=2011-09-28 23:27:15.718 ]: **
Exception IOError in application (closed stream) (process 7862, thread #Thread:0x864e4e0):
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-1.3.3/lib/rack/body_proxy.rb:12:in close'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/response.rb:230:in
not_modified!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:85:in call!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
from internal:prelude:10:in synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
cache error: Permission denied - /home/root/giasu_app/releases/20110928163844/tmp/cache/B11
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:243:in
fu_mkdir'
/usr/local/lib/ruby/1.9.1/fileutils.rb:217:in block (2 levels) in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in
reverse_each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:215:in block in mkdir_p'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in
each'
/usr/local/lib/ruby/1.9.1/fileutils.rb:201:in mkdir_p'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:173:in
ensure_cache_path'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/file_store.rb:100:in write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache/strategy/local_cache.rb:140:in
write_entry'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:364:in block in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:520:in
instrument'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/activesupport-3.1.0/lib/active_support/cache.rb:362:in write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/actionpack-3.1.0/lib/action_dispatch/http/rack_cache.rb:56:in
write'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/metastore.rb:60:in store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:264:in
store'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:257:in fetch'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:181:in
lookup'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:65:in call!'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
/home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
internal:prelude:10:in synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
/usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
cache: [GET /assets/dolphin.jpg] miss
[ pid=7874 thr=70414960 file=utils.rb:176 time=2011-09-28 23:27:15.771 ]: *** Exception IOError in application (closed stream) (process 7874, thread #Thread:0x864e4e0):
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-1.3.3/lib/rack/body_proxy.rb:12:in close'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/response.rb:230:in
not_modified!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:85:in call!'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/rack-cache-1.0.3/lib/rack/cache/context.rb:50:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/airbrake-3.0.4/lib/airbrake/user_informer.rb:12:in call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/engine.rb:455:in
call'
from /home/root/giasu_app/shared/bundle/ruby/1.9.1/gems/railties-3.1.0/lib/rails/railtie/configurable.rb:30:in method_missing'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/request_handler.rb:96:in
process_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:513:in accept_and_process_next_request'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_request_handler.rb:274:in
main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:205:in start_request_handler'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:170:in
block in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/utils.rb:479:in safe_fork'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:165:in
handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in
start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:180:in start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/rack/application_spawner.rb:128:in
start'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:253:in block (2 levels) in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:132:in
lookup_or_add'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:246:in block in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:82:in
block in synchronize'
from internal:prelude:10:in synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server_collection.rb:79:in
synchronize'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:244:in spawn_rack_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:137:in
spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/spawn_manager.rb:275:in handle_spawn_application'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:357:in
server_main_loop'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/lib/phusion_passenger/abstract_server.rb:206:in start_synchronously'
from /usr/local/lib/ruby/gems/1.9.1/gems/passenger-3.0.9/helper-scripts/passenger-spawn-server:99:in
'
2011/09/28 23:27:15 [error] 5914#0: *231 upstream prematurely closed connection while reading response header from upstream, client: 116.118.43.44, server: localhost, request: "GET /assets/dolphin.jpg HTTP/1.1", upstream: "passenger:unix:/passenger_helper_server:", host: "www.giasudientu.com", referrer: "http://www.giasudientu.com/"

Phuong Nguyen

Don't know why it keep reading cache from /tmp/cache/A89 while all caches are located under /tmp/cache/assets

Phuong Nguyen

I added config.assets.digest to production.rb to switch over to application-[digest-code].css instead of application.css and get over the rack-cache miss error. Still, if I'm trying to request to application.css then 1 out of every 2 requests will result in a 502 bad gate error

Phuong Nguyen

Hmm, change /current/tmp folder permission to 777 seem to solve the error. So this must be the nginx worker running as nobody user failed to read and write to /current/tmp/cache folder.

Sinan Taifour

Problem still occurs with rails 3.2.13, rack-cache 1.2, and running using passenger 4.0.25.

Disabling caching by setting config.action_controller.perform_caching = false stops the error from occurring.

Adrien Coquio

I believe I'm also seeing this problem on rails 3.2, rack 1.4 and unicorn 4.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.