Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High lock contention invalidating "failover global" invalidator #5525

Closed
headius opened this Issue Dec 17, 2018 · 2 comments

Comments

Projects
None yet
2 participants
@headius
Copy link
Member

headius commented Dec 17, 2018

The FailoverSwitchPoint invalidator implements an invalidator that will eventually give up and always be failed. This is to reduce the overhead from global variables that are continually updated and read during the lifetime of the application.

The logic currently uses a synchronized method to do the update of the failure count and the invalidation of the existing switchpoint. As reported by @guizmaii on Gitter, some applications may see this contention in lock profiling if they are updating global variables frequently.

This invalidator is also used for invalidating the base Fixnum and Float methods, but generally I would not expect an application to be continuously redefining math operations.

I have requested that @guizmaii try to determine why global variables are being updated so much in his application, which embeds JRuby into a larger app.

I have an atomics-based improvement for FailoverSwitchPointInvalidator in progress.

Here's a result of a script updating the same global across many threads. The profiler I'm using is from https://github.com/jvm-profiling-tools/async-profiler.

Script:

loop {
  100.times.map {
    Thread.new {
      1000000.times { $foo = 1 }
    }
  }.each(&:join)
}

Top lock contention:

          ns  percent  samples  top
  ----------  -------  -------  ---
3365580097350  100.00%  2297169  org.jruby.runtime.opto.FailoverSwitchPointInvalidator

@headius headius added this to the JRuby 9.2.6.0 milestone Dec 17, 2018

headius added a commit to headius/jruby that referenced this issue Dec 17, 2018

Use atomics to reduce contention within failover invaldator.
Instead of locking around updates to the switchpoint and failure
count, this patch replaces those fields with a stamped atomic
reference that can update both atomically.

Fixes jruby#5525.
@headius

This comment has been minimized.

Copy link
Member Author

headius commented Dec 17, 2018

With the fix in #5526:

Top lock contention:

          ns  percent  samples  top
  ----------  -------  -------  ---
   210857385   98.00%        4  java.util.Collections$SynchronizedSet
     4295257    2.00%      436  java.lang.ref.ReferenceQueue$Lock
@guizmaii

This comment has been minimized.

Copy link

guizmaii commented Dec 18, 2018

@headius impressive !

Is already there a nightly build including this fix I can try with my app ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.