Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

ThreadError: deadlock; recursive locking #2

Closed
johnkchow opened this Issue May 11, 2012 · 9 comments

Comments

Projects
None yet
2 participants
Contributor

johnkchow commented May 11, 2012

Hey @JoshMcKin,

I'm using your em_aws (great idea by the way) but I'm running into an issue. I'm using it for DynamoDB to do massive amounts of writes. On the initial query, it makes an HTTP request which requires a session authorization from AWS, which unfortunately has some locking. Any subsequence requests before the session authorization comes back results in the following exception:

[2012-05-11 23:26:17.954 #18573] ERROR -- : #<ThreadError: deadlock; recursive locking>
[2012-05-11 23:26:17.954 #18573] DEBUG -- : internal:prelude:8:in lock' <internal:prelude>:8:insynchronize'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/session_signer.rb:63:in get_session' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/session_signer.rb:72:insession'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/session_signer.rb:42:in access_key_id' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/dynamo_db/request.rb:31:inadd_authorization!'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:436:in build_request' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:375:inblock (3 levels) in client_request'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/response.rb:65:in call' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/response.rb:65:inrebuild_request'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/response.rb:60:in initialize' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:169:innew'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:169:in new_response' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:375:inblock (2 levels) in client_request'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:287:in log_client_request' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:363:inblock in client_request'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:275:in return_or_raise' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/core/client.rb:362:inclient_request'
(eval):3:in describe_table' /Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/dynamo_db/table.rb:497:inget_resource'
/Users/johnchow/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/aws-sdk-1.4.1/lib/aws/dynamo_db/table.rb:305:in `exists?'

Once the session is returned by AWS, the thing works like a charm.

I was wondering if you had any insight, maybe we could tackle this problem together?

Owner

JoshMcKin commented May 12, 2012

Its probably the mutex, EM-synchrony implements a fiber safe mutex that might help. Can you try this patch and let me know if it helps

require 'em-synchrony/thread'
module AWS
    module Core
      class SessionSigner
        @create_mutex =  EM::Synchrony::Thread::Mutex.new
      end
   end
end
Contributor

johnkchow commented May 12, 2012

Oh nice, I didn't know em-synchrony actually implemented its own fiber-safe mutex.

It looks promising so far, no ThreadErrors occurring. I'll keep you posted once I do a full stress test.

Owner

JoshMcKin commented May 12, 2012

Cool. Assuming this solves your issue, it would be awesome if you could write up a spec that reproduces this error before the patch.

Contributor

johnkchow commented May 12, 2012

Will do, I'll try and get that up for ya as soon as I can.

Owner

JoshMcKin commented May 15, 2012

Hi @jkchow

Hows the testing going? Just released an update to resolve issue https://github.com/JoshMcKin/em_aws/issues/3 with a fix for AWS autoloading. Can you give it a shot see how it performs in your threaded environment.

Thanks

Owner

JoshMcKin commented May 16, 2012

v 0.1.3 uses the fiber safe mutex.

Owner

JoshMcKin commented May 16, 2012

closed for now.

@JoshMcKin JoshMcKin closed this May 16, 2012

Contributor

johnkchow commented May 16, 2012

@JoshMcKin sorry, I've been swamped the past couple of days. I'll keep this issue in mind moving forward and come back to it in the future. Thanks for all your help and support Josh!

Contributor

johnkchow commented Jun 7, 2012

@JoshMcKin Josh, after running some stress tests on the EM server, I've run into the same issue again but only an hour after my app starts. I believe this is due to the session token expiring, and the library is refreshing the session. If you notice in the SessionSigner#initialize, there's another mutex that's stored within the SessionSigner object.

I'm going to monkey patch the #initialize method and use the em-synchrony's version, as you did in the past. Will get back to you by tomorrow with the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment