Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

deadlock; recursive locking errors in rack-1.4.3 #495

Closed
maerzbow opened this Issue · 7 comments

4 participants

@maerzbow

For some time now we are getting deadlock; recursive locking errors.

I actually even do not know if I have to 'blame' Rack for this.
But I am asking now, because I have found a simmilar issue #349 that was closed 2 months ago.
So I hoped it was solved without applying any of the mentioned gists.

Our setup is

rack (1.4.3)
rack-cache (1.2)
rack-mini-profiler (0.1.23)
rack-protection (1.3.2)
rack-ssl (1.3.2)
rails (3.2.11)
unicorn (4.5.0)
app error: deadlock; recursive locking (ThreadError)
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-1.4.3/lib/rack/lock.rb:14:in `lock'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-1.4.3/lib/rack/lock.rb:14:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-ssl-1.3.2/lib/rack/ssl.rb:27:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-mini-profiler-0.1.23/Ruby/lib/mini_profiler/profiler.rb:209:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/railties-3.2.11/lib/rails/engine.rb:479:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/railties-3.2.11/lib/rails/application.rb:223:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/railties-3.2.11/lib/rails/railtie/configurable.rb:30:in `method_missing'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/rack-1.4.3/lib/rack/deflater.rb:13:in `call'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/unicorn-4.5.0/lib/unicorn/http_server.rb:552:in `process_client'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/unicorn-4.5.0/lib/unicorn/http_server.rb:628:in `worker_loop'
/data/lingohub/shared/bundled_gems/ruby/1.9.1/gems/newrelic_rpm-3.5.4.34/lib/new_relic/agent/instrumentation/unicorn_instrumentation.rb:18:in `call'
@raggi
Owner

This is most likely caused by one of your middleware dropping the body without calling close on it. The default rails middleware stack should be fine, as should rack-cache. rack-ssl looks ok.

I honestly can't tell if rack-mini-profiler is doing the right thing by browsing the code on github. It has some huge methods with really odd mixes of tabs and spaces, and they have coupled conditional state (one conditional one place in the method should imply that other code in the method doesn't run). I would recommend having a good look in there to see if your app is being called twice in some cases.

Unfortunately your stack trace doesn't include middleware higher in the list, and it's also a problem that typically this error is thrown the request after the request that causes the problem.

We might be able to help more if you post the contents of your config.ru, and the output of rake middleware.

Please reopen if you find an issue somewhere in rack. The ML may also be able to help you track this down faster.

@raggi raggi closed this
@maerzbow

@raggi Just wanted to thank you for quick reply. As it seems now it works after disabling rack-mini-profiler.

@assembler

I'm having the exact same problem. Here is the output from my rake middleware:

use Airbrake::UserInformer
use Airbrake::Rack
use Rack::Cache
use Rack::Deflater
use ActionDispatch::Static
use Rack::Lock
use #<ActiveSupport::Cache::Strategy::LocalCache::Middleware:0x00000004263190>
use Rack::Runtime
use Rack::MethodOverride
use ActionDispatch::RequestId
use Rails::Rack::Logger
use ActionDispatch::ShowExceptions
use ActionDispatch::DebugExceptions
use ActionDispatch::RemoteIp
use ActionDispatch::Callbacks
use ActiveRecord::ConnectionAdapters::ConnectionManagement
use ActiveRecord::QueryCache
use ActionDispatch::Cookies
use ActionDispatch::Session::CookieStore
use ActionDispatch::Flash
use ActionDispatch::ParamsParser
use ActionDispatch::Head
use Rack::ConditionalGet
use Rack::ETag
use ActionDispatch::BestStandardsSupport
use Warden::Manager
use NewRelic::Rack::BrowserMonitoring
use NewRelic::Rack::ErrorCollector
use HireFireApp::Middleware
use OmniAuth::Builder
run BentPixels::Application.routes

Backtrace:

vendor/bundle/ruby/1.9.1/gems/rack-1.4.4/lib/rack/lock.rb:14:in `lock'
vendor/bundle/ruby/1.9.1/gems/rack-1.4.4/lib/rack/lock.rb:14:in `call'
vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.11/lib/action_dispatch/middleware/static.rb:62:in `call'
vendor/bundle/ruby/1.9.1/gems/rack-1.4.4/lib/rack/deflater.rb:13:in `call'
vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/engine.rb:479:in `call'
vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/application.rb:223:in `call'
vendor/bundle/ruby/1.9.1/gems/railties-3.2.11/lib/rails/railtie/configurable.rb:30:in `method_missing'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in `block in pre_process'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `catch'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `pre_process'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in `process'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in `receive_data'
vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in `start'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in `start'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/controllers/controller.rb:86:in `start'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/runner.rb:187:in `run_command'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/lib/thin/runner.rb:152:in `run!'
vendor/bundle/ruby/1.9.1/gems/thin-1.5.0/bin/thin:6:in `<top (required)>'
vendor/bundle/ruby/1.9.1/bin/thin:23:in `load'
vendor/bundle/ruby/1.9.1/bin/thin:23:in `<main>'
@assembler

exception is raised occasionally (once in every few days). I'm running app on heroku..

@emas80

Hi, I am having the same issues, it started moving from thin server to unicorn.
My application is based on Rails, and has currently a procfile that lauches both the server and Sidekiq, and it runs on Heroku.
Switching back to thin server solves the problem, so I think I will stay on thin.
I wanted to switch to unicorn to deploy on Heroku for using only one dynos for the entire application.

@raggi
Owner

I have previously witnessed body close bugs in both airbrake and newrelic, you also seem to have some other unusual middleware in your app. One of these is likely the cause, not the webserver.

@raggi
Owner

Yup... the bug comes from this one:

use HireFireApp::Middleware

It overrides the body, and doesn't pass on close, so it is not conformant to the rack spec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.