Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

deadlock issue may be related to #349 #604

Closed
jagrati opened this Issue · 11 comments

5 participants

@jagrati

I see this deadlock when links are clicked on quick succession on my app. I am using 1.3.10, which I beleive has this commit by raggi : (8998928)
Still the deadlocks are persistent:( , Please help!

using 1.3.10

Here is the deadlock that I am getting :

13-09-20T01:32:29.562533 #31088] ERROR -- : app error: deadlock; recursive locking (ThreadError)
E, [2013-09-20T01:32:29.562736 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/lock.rb:14:in lock'
E, [2013-09-20T01:32:29.562888 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/lock.rb:14:in
call'
E, [2013-09-20T01:32:29.563084 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in forward'
E, [2013-09-20T01:32:29.563242 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in
fetch'
E, [2013-09-20T01:32:29.563387 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in lookup'
E, [2013-09-20T01:32:29.563530 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in
call!'
E, [2013-09-20T01:32:29.563672 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in call'
E, [2013-09-20T01:32:29.563815 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/railties-3.1.10/lib/rails/engine.rb:456:in
call'
E, [2013-09-20T01:32:29.563983 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/railties-3.1.10/lib/rails/application.rb:143:in call'
E, [2013-09-20T01:32:29.564110 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/railties-3.1.10/lib/rails/railtie/configurable.rb:30:in
method_missing'
E, [2013-09-20T01:32:29.564237 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-rewrite-1.3.3/lib/rack/rewrite.rb:20:in call'
E, [2013-09-20T01:32:29.564361 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/deflater.rb:13:in
call'
E, [2013-09-20T01:32:29.564485 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:552:in process_client'
E, [2013-09-20T01:32:29.564609 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:632:in
worker_loop'
E, [2013-09-20T01:32:29.564732 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:500:in spawn_missing_workers'
E, [2013-09-20T01:32:29.564857 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:511:in
maintain_worker_count'
E, [2013-09-20T01:32:29.564981 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:277:in join'
E, [2013-09-20T01:32:29.565105 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/bin/unicorn_rails:209:in
'
E, [2013-09-20T01:32:29.565230 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/bin/unicorn_rails:23:in load'
E, [2013-09-20T01:32:29.565354 #31088] ERROR -- : /mnt/voylla-staging/shared/bundle/ruby/1.9.1/bin/unicorn_rails:23:in

'

Please find output of rake middleware :

@jagrati

Using rake (10.0.3)
Using Ascii85 (1.0.2)
Using Platform (0.4.0)
Using open4 (1.3.0)
Using POpen4 (0.1.4)
Using multi_json (1.2.0)
Using activesupport (3.1.10)
Using builder (3.0.4)
Using i18n (0.6.1)
Using activemodel (3.1.10)
Using erubis (2.7.0)
Using rack (1.3.10)
Using rack-cache (1.2)
Using rack-mount (0.8.3)
Using rack-test (0.6.2)
Using hike (1.2.1)
Using tilt (1.3.3)
Using sprockets (2.0.4)
Using actionpack (3.1.10)
Using mime-types (1.21)
Using polyglot (0.3.3)
Using treetop (1.4.12)
Using mail (2.3.3)
Using actionmailer (3.1.10)
Using active_reload (0.6.1)
Using active_shipping (0.9.13)
Using braintree (2.22.0)
Using json (1.7.7)
Using activemerchant (1.17.0)
Using arel (2.2.3)
Using tzinfo (0.3.35)
Using activerecord (3.1.10)
Using activeresource (3.1.10)
Using acts_as_list (0.1.4)
Using afm (0.2.0)
Using ansi (1.4.3)
Using archive-tar-minitar (0.5.2)
Using nokogiri (1.5.6)
Using uuidtools (2.1.3)
Using aws-sdk (1.8.3.1)
Using bcrypt-ruby (3.0.1)
Using bundler (1.3.2)
Using cancan (1.6.5)
Using highline (1.6.2)
Using jruby-pageant (1.1.1)
Using net-ssh (2.4.0)
Using net-scp (1.0.4)
Using net-sftp (2.0.5)
Using net-ssh-gateway (1.1.0)
Using capistrano (2.15.4)
Using clamp (0.6.1)
Using cocaine (0.3.2)
Using coderay (1.0.9)
Using coffee-script-source (1.4.0)
Using execjs (1.4.0)
Using coffee-script (2.2.0)
Using rack-ssl (1.3.3)
Using rdoc (3.12.2)
Using thor (0.14.6)
Using railties (3.1.10)
Using coffee-rails (3.1.1)
Using columnize (0.3.6)
Using dalli (2.6.2)
Using rails (3.1.12)
Using deface (0.9.1)
Using delayed_job (3.0.5)
Using delayed_job_active_record (0.4.1)
Using orm_adapter (0.0.7)
Using warden (1.2.1)
Using devise (1.5.3)
Using dynamic_form (1.1.4)
Using excon (0.22.1)
Using faker (1.0.0)
Using multipart-post (1.1.5)
Using faraday (0.8.6)
Using formatador (0.2.4)
Using ruby-hmac (0.4.0)
Using fog (1.9.0)
Using formtastic (2.2.1)
Using geoip (1.2.1)
Using hashery (2.1.0)
Using hashie (1.2.0)
Using httpauth (0.2.0)
Using jquery-rails (2.1.4)
Using jwt (0.1.5)
Using kaminari (0.14.1)
Using kgio (2.8.0)
Using libv8 (3.11.8.17)
Using ruby_core_source (0.1.5)
Using linecache19 (0.5.12)
Using polyamorous (0.5.0)
Using meta_search (1.1.1)
Using method_source (0.8.1)
Using nested_set (1.6.8)
Using oauth2 (0.8.1)
Using omniauth (1.1.3)
Using omniauth-oauth2 (1.1.1)
Using omniauth-facebook (1.4.1)
Using paperclip (2.5.0)
Using ruby-rc4 (0.1.5)
Using ttfunk (1.0.3)
Using pdf-reader (1.3.3)
Using pg (0.14.1)
Using prawn (0.12.0)
Using prawnto_2 (0.2.6)
Using slop (3.4.3)
Using pry (0.9.12)
Using pry-rails (0.2.2)
Using psych (2.0.0)
Using rack-rewrite (1.3.3)
Using rails-dev-tweaks (0.6.1)
Using raindrops (0.10.0)
Using rambling-slider-rails (0.2.0)
Using rd_find_by_param (0.1.1)
Using rd_resource_controller (1.0.1)
Using recaptcha (0.3.5)
Using ref (1.0.4)
Using rubber (2.2.0) from git://github.com/EvilFaeton/rubber.git (at master)
Using ruby-debug-base19 (0.11.25)
Using ruby-debug19 (0.11.6)
Using ruby-ole (1.2.11.6)
Using sass (3.2.6)
Using sass-rails (3.1.7)
Using seamless_database_pool (1.0.6) from git://github.com/wbharding/seamless_database_pool.git (at master)
Using spreadsheet (0.7.1)
Using state_machine (1.0.1)
Using stringex (1.3.0)
Using spree_core (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree_auth (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree_api (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree_dash (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree_promo (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree_sample (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree (0.70.7) from git://github.com/0xe/spree.git (at master)
Using spree_active_shipping (1.0.3) from git://github.com/spree/spree_active_shipping.git (at a85128c)
Using spree_email_to_friend (1.2) from git://github.com/Voylla/spree_email_to_friend.git (at 6edfb10)
Using spree_payu_in (0.0.1) from https://github.com/0xe/spree_payu_in (at 0-70-stable)
Using spree_recently_viewed (1.1) from git://github.com/spree/spree_recently_viewed.git (at 0-70-stable)
Using spree_static_content (0.70.3) from http://github.com/spree/spree_static_content (at 1861f4c)
Using spree_wishlist (1.2) from git://github.com/0xe/spree_wishlist.git (at b9bc718)
Using sqlite3 (1.3.7)
Using therubyracer (0.11.4)
Using turn (0.9.6)
Using uglifier (1.3.0)
Using unicorn (4.6.2)
Using yui-compressor (0.9.6)

@jagrati

Please let me know if any more info is required. .Also tried https://gist.github.com/zporter/2936818 , did not work for me . As we are using unicorn, can I use threadsafe?

@thedarkone

What is your Ruby version?

@jagrati

@thedarkone My ruby version is ruby 1.9.2p320

@jagrati

Removing delayed job significantly brought down deadlock. One more datapoint. Not sure if I am missing something in delayed job

@bjeanes

I am also getting recursive deadlocks with this version of rack on 1.9.2p429 (in a Rails 3.1.12 project). I've been looking into this for several days now but am at the end of my rope wrt to ideas. Is this a confirmed known issue with Rack?

@raggi
Owner

There was a problem with the etag middleware also, it has been fixed in: 4d9e1b2

@raggi
Owner

Please reopen if you find a replication case in a rack app.

@raggi raggi closed this
@jagrati
@bjeanes

To close the loop on my earlier comment, the issue I was seeing turned out to be hassox/warden#82.

@khandu20

After applying the fixes in the two files lib/rack/etag.rb and test/spec_etag.rb I am still seeing the same deadlock issue. Please let me know if you need any more inputs.

(Files present at the locations /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/etag.rb and /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/test/spec_etag.rb)

Logs for your reference
E, [2014-01-29T17:02:44.580355 #12886] ERROR -- : app error: deadlock; recursive locking (ThreadError)
E, [2014-01-29T17:02:44.580450 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/lock.rb:14:in lock'
E, [2014-01-29T17:02:44.580504 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/lock.rb:14:in
call'
E, [2014-01-29T17:02:44.580554 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in forward'
E, [2014-01-29T17:02:44.580601 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in
fetch'
E, [2014-01-29T17:02:44.580648 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in lookup'
E, [2014-01-29T17:02:44.580713 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in
call!'
E, [2014-01-29T17:02:44.580764 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in call'
E, [2014-01-29T17:02:44.580829 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/railties-3.1.10/lib/rails/engine.rb:456:in
call'
E, [2014-01-29T17:02:44.580880 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/railties-3.1.10/lib/rails/application.rb:143:in call'
E, [2014-01-29T17:02:44.580928 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/railties-3.1.10/lib/rails/railtie/configurable.rb:30:in
method_missing'
E, [2014-01-29T17:02:44.580977 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-rewrite-1.3.3/lib/rack/rewrite.rb:20:in call'
E, [2014-01-29T17:02:44.581024 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/rack-1.3.10/lib/rack/deflater.rb:13:in
call'
E, [2014-01-29T17:02:44.581071 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:552:in process_client'
E, [2014-01-29T17:02:44.581117 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:632:in
worker_loop'
E, [2014-01-29T17:02:44.581163 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:500:in spawn_missing_workers'
E, [2014-01-29T17:02:44.581209 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/lib/unicorn/http_server.rb:142:in
start'
E, [2014-01-29T17:02:44.581256 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/gems/unicorn-4.6.2/bin/unicorn_rails:209:in <top (required)>'
E, [2014-01-29T17:02:44.581301 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/bin/unicorn_rails:23:in
load'
E, [2014-01-29T17:02:44.581347 #12886] ERROR -- : /mnt/project-code-repo/shared/bundle/ruby/1.9.1/bin/unicorn_rails:23:in `

'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.