Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handler (storage) thread-safety #5

Merged
merged 5 commits into from
Nov 18, 2016

Conversation

kares
Copy link
Contributor

@kares kares commented Oct 25, 2016

HGH's design is not thread-safe since most of the use-cases are actors themselves - thus protected from concurrent access. However, not all and esp. under JRuby (with non-GIL concurrency and likely due threaded fibers) some parts tend to break unexpectedly when a handler update is executed by 2 threads.

Here's a failure from the older stack which is still relevant today (with an updated Adhearsion 2.6 stack) :

ERROR Adhearsion::Initializer: <NoMethodError> undefined method `find' for nil:NilClass
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/has-guarded-handlers-1.6.0/lib/has_guarded_handlers.rb:149:in `guarded?'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/has-guarded-handlers-1.6.0/lib/has_guarded_handlers.rb:92:in `trigger_handler'
org/jruby/RubyKernel.java:1254:in `catch'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/has-guarded-handlers-1.6.0/lib/has_guarded_handlers.rb:91:in `trigger_handler'
org/jruby/RubyEnumerable.java:556:in `find'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/has-guarded-handlers-1.6.0/lib/has_guarded_handlers.rb:89:in `trigger_handler'
org/jruby/RubyKernel.java:1254:in `catch'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/has-guarded-handlers-1.6.0/lib/has_guarded_handlers.rb:88:in `trigger_handler'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/punchblock-2.5.3/lib/punchblock/translator/asterisk/call.rb:159:in `process_ami_event'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/punchblock-2.5.3/lib/punchblock/translator/asterisk.rb:209:in `ami_dispatch_to_or_create_call'
org/jruby/RubyHash.java:1357:in `each_pair'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/punchblock-2.5.3/lib/punchblock/translator/asterisk.rb:207:in `ami_dispatch_to_or_create_call'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/punchblock-2.5.3/lib/punchblock/translator/asterisk.rb:90:in `handle_ami_event'
org/jruby/RubyKernel.java:1932:in `public_send'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `handle_message'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `task'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `initialize'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:47:in `initialize'
/srv/phone/apptastic/shared/bundle/jruby/1.9/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `create'

@handlers ||= Hash.new { |h, k| h[k] = Hash.new { |h, k| h[k] = [] } } the underlying issue is that when the same key is "missed" (under 2 threads) the default proc might execute twice ... with TS::Cache there's an atomic guarantee of fetch_or_store method - once set we're only retrieve the same value for the given key

p.s. ThreadSafe::Cache is part of the concurrent-ruby gem these days, if there's interest in getting the changes upstream it should be straightforward to migrate using Concurrent::Hash instead.

@kares kares changed the title Up thread safety 1 handler (storage) thread-safety Oct 25, 2016
@@ -18,6 +18,8 @@ Gem::Specification.new do |s|
s.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
s.require_paths = ["lib"]

s.add_dependency 'thread_safe', ["~> 0.3"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kares Sorry for being very slow on my end. Question for you, should this be [">= 0.3.4"]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when I originally did this 0.3.x was latest - was probably afraid that it might broke around 1.0
but since thread_safe is dead and part of concurrent_ruby gem, latest (last) release is 0.3.5.
so this shouldn't matter much really - but I update to make sure < 0.3.4 is not used - thanks.

@sfgeorge
Copy link
Member

Thanks very much @kares. I think this looks great 👍

@benlangfeld could you take a look when you get a chance?

Copy link
Member

@benlangfeld benlangfeld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please include a changelog entry? Other than that, this looks good.

@benlangfeld benlangfeld merged commit 137d46d into adhearsion:develop Nov 18, 2016
@sfgeorge
Copy link
Member

Awesome! Thank you both!

kares added a commit to kares/has-guarded-handlers that referenced this pull request Jan 14, 2019
* concurrent-ruby:
  [CI] let's test against more (recent) Rubies
  use concurrent/map as a thread_safe replacement
  ~~handler (storage) thread-safety (adhearsion#5)~~
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants