Make connection pool fair #6488

pmahoney · May 25, 2012

This is a second attempt of #6416

It makes the connection pool "fair" with respect to waiting threads. I've done some more measurements here: http://polycrystal.org/2012/05/24/activerecord_connection_pool_fairness.html The patch is also cleaned up compared to the first attempt; the code is much more readable.

It includes some test fixes from @yahonda that this patch triggered (though the failures seem unrelated to the code)

I am still getting test failures, but I see the same failures against master: https://gist.github.com/2788538 And none of these seem related to the connection pool.

jrochkind · May 25, 2012

Awesome. This definitely deals with some troubles i've been having.

We don't need strict fairness: if two connections become
available at the same time, it's fine of two threads that were
waiting acquire the connections out of order.

What keeps you from being strictly fair? Strict fairness (or close to it barring edge cases) would work out even better for me, although this will still be a huge improvement. Per our previous conversation where I think you said that if multiple threads were waiting, #signal would always wake up the one that was waiting the longest (verified in both jruby and mri?) -- what prevents strict fairness from being implemented? (The kind of fairness enforced here is still much better, and possibly good enough even for my use case, just curious if it can be improved yet further).
- I notice even though your comments say you don't care about strict fairness -- your test actually does verify strict order with the order test, no? Are the comments outdated, strict fairness really is being guaranteed by the test?
What's the @cond.broadcast needed for? What was failing without this that required this? Related to first point above? I ask because previous implementations (3-2-stable as well as master) did not use a broadcast, but it didn't seem to cause any problems -- the threads that ended up waiting indefinitely in master previously were not caused by a lack of broadcast, they were caused by the situation you fixed with @num_wait and your semi-fair guarantee, as well as code that didn't keep track of total time waiting so threads would keep loop-waiting indefinitely when other threads 'stole' connections.

pmahoney · May 25, 2012

I think you said that if multiple threads were waiting, #signal would always wake up the one that was waiting the longest (verified in both jruby and mri?) -- what prevents strict fairness from being implemented?

Yes, that is true. By "not strict" I mean that if two connections become available at the same time, and thread1 and thread2 are waiting in line, the order in which they re-acquire the monitor is not guaranteed (but thread3 will not be able to "steal" because @num_waiting check forces it to wait).

What's the @cond.broadcast needed for?

I don't see this in the diff. There was a broadcast in the original patch, but this new one should have removed it, unless I missed one.

jrochkind · May 25, 2012

Aha, so if two connections become available more or less at once, it's not guaranteed whether thread1 or thread2 goes first, but they are both guaranteed to get a connection ahead of thread3? If that's so, that's plenty good enough.

I don't see this in the diff. There was an @broadcast in the original patch,

here is where I see it. I see now it's actually only in the #reap implementation. I don't trust the semantics of the reap implementation already, and lack of fairness when reaping (which if it works right ought to only apply to code that violates AR's contract in the first place) is not too much of a concern.

But I think it could be replaced by counting up how many times the reaper reaped, and doing that many signals instead of a broadcast, would that be better?

jrochkind · May 25, 2012

I'm actually still confused about the fairness.

If thread1, thread2, and thread3 are all waiting (yep, thread3 is already waiting too)
and then two connections become avail at more or less the same time
are both thread1 and thread2 guaranteed to get a connection before thread3 (which was also waiting?)

your order check in the test seems to guarantee this in fact is true, I think?

pmahoney · May 25, 2012

Ah. It was removed here. The combined diff for the pull request is better: https://github.com/rails/rails/pull/6488/files

... counting up how many times the reaper reaped ...

That's what @available.add checkout_new_connection if @available.any_waiting? (in #remove which is called by #reap) is supposed to do, though I admit I have not done any testing of the reaper. The reaper attempts to remove stale connections, so I attempt to then create new ones to replace those that have been removed. But what happens if someone checks in a presumed-leaked connection that has been removed? Ugh.

jrochkind · May 25, 2012

ugh, sorry, ignore on broadcast, I see I wasn't looking at the final version, which has no broadcast at all in the reap. okay then.

still curious about nature of guarantees, but this is a good patch regardless, I think.

I actually run into this problem in MRI not jruby -- I'll try to run your test in MRI 1.9.3 next week, cause i'm curious -- i fully expect based on my experiences, it will show similar benefit in MRI.

jrochkind · May 25, 2012

I am suspicious of the reaper in general, personally, although @tenderlove may disagree.

But I personally don't think the reaper does anything particularly useful atm, so making sure it does what it does properly for fairness... I dunno.

The reaper right now will reap only if a connection is still checked out, was last checked out more than @dead_connection_checkout seconds ago (default 5), and has been closed by the actual rdbms (I think that's what active? checks?)

Most rdbms have a very long timeout for idleness, MySQL by default (with AR mysql2) will wait hours before closing an idle connection. Which is in fact ordinarily what you want, I think?

So I'm not sure how the reaper does anything useful -- it won't reap a 'leaked' connection, under normal conditions, for many minutes or even hours after it was leaked (typo fixed).

I may be missing something? Maybe you're expected to significantly reduce the rdbm's idle timeout to make use of the reaper?

There are of course times when a connection may be closed because of network or server problems or rdbms restart, unrelated to leaked connections. But the reaper's not meant to deal with that, i don't think, and probably isn't the right way to anyway. (There's already an automatic_reconnect key for some of AR's adapters, although it's semantics aren't entirely clear).

Anyhow, this is really a different ticket, I just mention it before you dive into making things 'right' with the reaper, and because you seem to understand this stuff well and I hadn't gotten anyone else to consider or confirm or deny my suspicions about current reaper func yet. :)

pmahoney · May 25, 2012

I'm actually still confused about the fairness.

If thread1, thread2, and thread3 are all waiting (yep, thread3 is already waiting too)
and then two connections become avail at more or less the same time
are both thread1 and thread2 guaranteed to get a connection before thread3 (which was also waiting?)

your order check in the test seems to guarantee this in fact is true, I think?

The test_checkout_fairness_by_group is a better test of this. What happens (I think) is that a ConditionVariable does guarantee that the longest waiting thread is the first to wake up. But the first thing a thread does after being woken up is re-acquire the monitor. It's this second action that is a free-for-all. So, yes, thread1 and thread2 will get the connection ahead of thread3 in your example, because thread3 will not be woken up.

pmahoney · May 25, 2012

I just mention it before you dive into making things 'right' with the reaper

I was planning on just ignoring that :-P

pmahoney · May 25, 2012

@jrochkind Oh, and thanks a bunch for taking a look at this. I greatly appreciate the second set of eyes.

jrochkind · May 25, 2012

Okay, I think i understand the fairness issue, and it seems pretty damn good. Def understand the issue where it's unpredictable which thread will get the lock first -- that's what requires the @num_waiting in the first place. And I understand how that guards against a thread that wasn't waiting at all 'stealing' a connection from one that was. (Yes, I have had this problem too).

I think I'm understanding right that your code will be pretty close to fair -- if there are multiple threads waiting, there's no way the oldest thread waiting will get continually bumped in favor of newer waiters. The issue is only when N>1 threads are checked in at very close to the same time, and even then the first N waiters will all get connections before the N+1st and subsequent waiters. That seems totally good enough.

On the reaper.... man, looking at the mysql2 adapter code specifically, I don't think the reaper will ever reap anything, i can't figure out how a connection could ever not be active? even if the rdbms has closed it for idleness -- active? is implemented adapter-specific, but in mysql2 it seems to me a connection will always be active? unless manually disconnected.

That's really a different issue and for @tenderlove to consider I guess, since he added the code. Personally, I would not ever use the reaper at all, which fortunately is easy to do by making sure reap_frequency is unset.

jrochkind · May 25, 2012

@pmahoney thanks a lot for doing it, man! I've been struggling with this stuff for a while, and not managing to solve it, and not managing to find anyone else to review my ideas/code for AR either! (I am not a committer, in case that was not clear, def not).

Concurrency is def confusing.

rafaelfranca · May 25, 2012

@pmahoney I think you will need to squash your commits.

@tenderlove could you review this one?

pmahoney · May 25, 2012

Here's a mostly squashed version: #6492

pmahoney closed this May 25, 2012

pmahoney referenced this pull request May 25, 2012
Merged
Fair connection pool2 #6492

pmahoney	`Make connection pool fair with respect to waiting threads.` … Conflicts: activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb	`8d04939`
pmahoney	`Refactor fair queue into Queue class. Clean up checkout method. Clean…` … … up fairness tests.	`b767418`
pmahoney	`Clean up connection pool fair Queue class.`	`f3e2add`
pmahoney	`Restore Queue#any_waiting deleted by mistake.`	`e409cd1`
yahonda	`Cache metadata in advance to avoid extra sql statements while testing.` … Reason: If metadata is not cached extra sql statements will be executed, which causes failures tests with assert_queries().	`873b0e2`
pmahoney	`Cache metadata in advance to avoid extra sql statements while testing.` … Reason: If metadata is not cached extra sql statements will be executed, which causes failures tests with assert_queries().	`3f74ba3`
pmahoney	`Correct documentation re: which exception is raised.`	`ab54a4a`
pmahoney	`Use @checkout_timeout rather than obsolete @timeout.`	`2884c6b`

@@ @@ -962,6 +962,10 @@ def test_eager_loading_with_order_on_joined_table_preloads @@
    end
    def test_eager_loading_with_conditions_on_joined_table_preloads
++    # cache metadata in advance to avoid extra sql statements executed while testing
++    Tagging.first
++    Tag.first
++
      posts = assert_queries(2) do
        Post.scoped(:select => 'distinct posts.*', :includes => :author, :joins => [:comments], :where => "comments.body like 'Thank you%'", :order => 'posts.id').all
      end
    def test_eager_loading_with_conditions_on_join_model_preloads
      Author.columns
++
++    # cache metadata in advance to avoid extra sql statements executed while testing
++    AuthorAddress.first
      authors = assert_queries(2) do
        Author.scoped(:includes => :author_address, :joins => :comments, :where => "posts.title like 'Welcome%'").all

      author = Author.new(:name => "David")
      assert !author.essays.loaded?
++    # cache metadata in advance to avoid extra sql statements executed while testing
++    Essay.first
++
      assert_queries 1 do
        assert_equal 1, author.essays.size
      end

@@ @@ -36,7 +36,7 @@ def test_expire_mutates_in_use @@
        def test_close
          pool = ConnectionPool.new(ConnectionSpecification.new({}, nil))
--        pool.connections << adapter
++        pool.insert_connection_for_test! adapter
          adapter.pool = pool
          # Make sure the pool marks the connection in use

@@ @@ -200,6 +200,110 @@ def test_checkout_behaviour @@
          end.join
        end
++      # The connection pool is "fair" if threads waiting for
++      # connections receive them the order in which they began
++      # waiting.  This ensures that we don't timeout one HTTP request
++      # even while well under capacity in a multi-threaded environment
++      # such as a Java servlet container.
++      #
++      # We don't need strict fairness: if two connections become
++      # available at the same time, it's fine of two threads that were
++      # waiting acquire the connections out of order.
++      #
++      # Thus this test prepares waiting threads and then trickles in
++      # available connections slowly, ensuring the wakeup order is
++      # correct in this case.
++      def test_checkout_fairness
++        @pool.instance_variable_set(:@size, 10)
++        expected = (1..@pool.size).to_a.freeze
++        # check out all connections so our threads start out waiting
++        conns = expected.map { @pool.checkout }
++        mutex = Mutex.new
++        order = []
++        errors = []
++
++        threads = expected.map do |i|
++          t = Thread.new {
++            begin
++              conn = @pool.checkout # never checked back in
++              mutex.synchronize { order << i }
++            rescue => e
++              mutex.synchronize { errors << e }
++            end
++          }
++          Thread.pass until t.status == "sleep"
++          t
++        end
++
++        # this should wake up the waiting threads one by one in order
++        conns.each { |conn| @pool.checkin(conn); sleep 0.1 }
++
++        threads.each(&:join)
++
++        raise errors.first if errors.any?
++
++        assert_equal(expected, order)
++      end
++
++      # As mentioned in #test_checkout_fairness, we don't care about
++      # strict fairness.  This test creates two groups of threads:
++      # group1 whose members all start waiting before any thread in
++      # group2.  Enough connections are checked in to wakeup all
++      # group1 threads, and the fact that only group1 and no group2
++      # threads acquired a connection is enforced.
++      def test_checkout_fairness_by_group
++        @pool.instance_variable_set(:@size, 10)
++        # take all the connections
++        conns = (1..10).map { @pool.checkout }
++        mutex = Mutex.new
++        successes = []    # threads that successfully got a connection
++        errors = []
++
++        make_thread = proc do |i|
++          t = Thread.new {
++            begin
++              conn = @pool.checkout # never checked back in
++              mutex.synchronize { successes << i }
++            rescue => e
++              mutex.synchronize { errors << e }
++            end
++          }
++          Thread.pass until t.status == "sleep"
++          t
++        end
++
++        # all group1 threads start waiting before any in group2
++        group1 = (1..5).map(&make_thread)
++        group2 = (6..10).map(&make_thread)
++
++        # checkin n connections back to the pool
++        checkin = proc do |n|
++          n.times do
++            c = conns.pop
++            @pool.checkin(c)
++          end
++        end
++
++        checkin.call(group1.size)         # should wake up all group1
++
++        loop do
++          sleep 0.1
++          break if mutex.synchronize { (successes.size + errors.size) == group1.size }
++        end
++
++        winners = mutex.synchronize { successes.dup }
++        checkin.call(group2.size)         # should wake up everyone remaining
++
++        group1.each(&:join)
++        group2.each(&:join)
++
++        assert_equal((1..group1.size).to_a, winners.sort)
++
++        if errors.any?
++          raise errors.first
++        end
++      end
++
        def test_automatic_reconnect=
          pool = ConnectionPool.new ActiveRecord::Base.connection_pool.spec
          assert pool.automatic_reconnect

rails/rails

HTTPS clone URL

Subversion checkout URL

Make connection pool fair #6488

Labels

Milestone

Assignee

4 participants

Fair connection pool2 #6492