New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make connection pool fair
Conversation
Conflicts: activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb
… up fairness tests.
Reason: If metadata is not cached extra sql statements will be executed, which causes failures tests with assert_queries().
Reason: If metadata is not cached extra sql statements will be executed, which causes failures tests with assert_queries().
Awesome. This definitely deals with some troubles i've been having.
|
Yes, that is true. By "not strict" I mean that if two connections become available at the same time, and thread1 and thread2 are waiting in line, the order in which they re-acquire the monitor is not guaranteed (but thread3 will not be able to "steal" because
I don't see this in the diff. There was a |
Aha, so if two connections become available more or less at once, it's not guaranteed whether thread1 or thread2 goes first, but they are both guaranteed to get a connection ahead of thread3? If that's so, that's plenty good enough.
here is where I see it. I see now it's actually only in the #reap implementation. I don't trust the semantics of the reap implementation already, and lack of fairness when reaping (which if it works right ought to only apply to code that violates AR's contract in the first place) is not too much of a concern. But I think it could be replaced by counting up how many times the reaper reaped, and doing that many |
I'm actually still confused about the fairness.
your |
Ah. It was removed here. The combined diff for the pull request is better: https://github.com/rails/rails/pull/6488/files
That's what |
ugh, sorry, ignore on broadcast, I see I wasn't looking at the final version, which has no broadcast at all in the reap. okay then. still curious about nature of guarantees, but this is a good patch regardless, I think. I actually run into this problem in MRI not jruby -- I'll try to run your test in MRI 1.9.3 next week, cause i'm curious -- i fully expect based on my experiences, it will show similar benefit in MRI. |
I am suspicious of the reaper in general, personally, although @tenderlove may disagree. But I personally don't think the reaper does anything particularly useful atm, so making sure it does what it does properly for fairness... I dunno. The reaper right now will reap only if a connection is still checked out, was last checked out more than Most rdbms have a very long timeout for idleness, MySQL by default (with AR mysql2) will wait hours before closing an idle connection. Which is in fact ordinarily what you want, I think? So I'm not sure how the reaper does anything useful -- it won't reap a 'leaked' connection, under normal conditions, for many minutes or even hours after it was leaked (typo fixed). I may be missing something? Maybe you're expected to significantly reduce the rdbm's idle timeout to make use of the reaper? There are of course times when a connection may be closed because of network or server problems or rdbms restart, unrelated to leaked connections. But the reaper's not meant to deal with that, i don't think, and probably isn't the right way to anyway. (There's already an automatic_reconnect key for some of AR's adapters, although it's semantics aren't entirely clear). Anyhow, this is really a different ticket, I just mention it before you dive into making things 'right' with the reaper, and because you seem to understand this stuff well and I hadn't gotten anyone else to consider or confirm or deny my suspicions about current reaper func yet. :) |
The |
I was planning on just ignoring that :-P |
@jrochkind Oh, and thanks a bunch for taking a look at this. I greatly appreciate the second set of eyes. |
Okay, I think i understand the fairness issue, and it seems pretty damn good. Def understand the issue where it's unpredictable which thread will get the lock first -- that's what requires the I think I'm understanding right that your code will be pretty close to fair -- if there are multiple threads waiting, there's no way the oldest thread waiting will get continually bumped in favor of newer waiters. The issue is only when N>1 threads are checked in at very close to the same time, and even then the first N waiters will all get connections before the N+1st and subsequent waiters. That seems totally good enough. On the reaper.... man, looking at the mysql2 adapter code specifically, I don't think the reaper will ever reap anything, i can't figure out how a connection could ever not be That's really a different issue and for @tenderlove to consider I guess, since he added the code. Personally, I would not ever use the reaper at all, which fortunately is easy to do by making sure |
@pmahoney thanks a lot for doing it, man! I've been struggling with this stuff for a while, and not managing to solve it, and not managing to find anyone else to review my ideas/code for AR either! (I am not a committer, in case that was not clear, def not). Concurrency is def confusing. |
@pmahoney I think you will need to squash your commits. @tenderlove could you review this one? |
Here's a mostly squashed version: #6492 |
This is a second attempt of #6416
It makes the connection pool "fair" with respect to waiting threads. I've done some more measurements here: http://polycrystal.org/2012/05/24/activerecord_connection_pool_fairness.html The patch is also cleaned up compared to the first attempt; the code is much more readable.
It includes some test fixes from @yahonda that this patch triggered (though the failures seem unrelated to the code)
I am still getting test failures, but I see the same failures against master: https://gist.github.com/2788538 And none of these seem related to the connection pool.