Large number of failed jobs after upgrading to ActiveRecord4

[deckchairhq]

I'm looking into a potential issue with ActiveRecord4 and Sidekiq.

Since upgrading to AR4, my failure rate shot through the roof - from under 100 failures in 2 weeks to 1000 failures in the first hour after upgrading.

The error presented is this:

ActiveRecord::ConnectionTimeoutError: could not obtain a database connection within 5.000 seconds (waited 5.001 seconds)

I have an ActiveRecord connection pool of 20. The first 20 jobs go through fine and I can see my connection pool growing, however when it hits 20, the errors start occurring.

It appears that either the connections aren't getting released or checked out appropriately.

If I replace the following line from the ActiveRecord Sidekiq middleware:

::ActiveRecord::Base.clear_active_connections! if defined?(::ActiveRecord)

...with a full disconnect:

::ActiveRecord::Base.clear_all_connections!

All works fine again - though obviously very inefficient.

I'm using a barebones ruby script for my worker that connects to ActiveRecord without rails:

      ::ActiveRecord::Base.establish_connection(
        :adapter  => configuration[:adapter],
        :host     => configuration[:host],
        :username => configuration[:username],
        :password => configuration[:password],
        :database => configuration[:database],
        :pool     => configuration[:pool],
        :port     => configuration[:port],
        :reaping_frequency => 10
      )

      self[:activerecord_cache] = ::ActiveSupport::Cache::MemoryStore.new

It's all very simple and the queue was running fine until now.

I'll plug away at this a bit more and if no luck, will get a skeleton repo up to reproduce it. Any input is appreciated.

[mperham]

You'll probably need +1 connection for the main thread which boots the system. You don't state your Sidekiq version but I assume 2.12.x. Try doubling your pool to 41 and see if that helps. I've noticed some oddness where sometimes 2x connections are needed. I still don't understand the cause.

[deckchairhq]

Ah sorry, should of provided more information..

My DB can accept 500 connections.
I'm currently testing with 1 set as the concurrency level and still see the issue.

[mperham]

Are you spinning off threads in your worker, to do work in parallel?

[deckchairhq]

In a few of my jobs I do create a thread or two.

[mperham]

That'll do it - if they touch ActiveRecord, they'll check out a connection and never return it until stale connections are reaped.

[deckchairhq]

Ah that sounds like the culprit! New AR4 feature I presume?

Best course of action is going to be cleaning up in the thread I take it - is it worth adding any logic to the middleware to help cleanup in such circumstances?

[mperham]

It shouldn't be new.

Best course is to manually check out a connection for your thread's use:

Thread.new do
  ActiveRecord::Base.connection_pool.with_connection do |conn|
    MyModel.find(123)
  end
end

The find will use the connection you checked out because it's already reserved for the thread. When the with_connection block finishes, the connection will be auto-released.

[mperham]

Maybe clear_active_connections! used to clear stale connections in Rails 3 and doesn't anymore in Rails 4?

[deckchairhq]

That sounds like it could be it, I'll have a look this AM, but it did start immediately after I updated.

[deckchairhq]

Just to confirm, wrapping in a block as mentioned above works a treat.. It appears that behaviour has changed for the clear_active_connections! method in the latest version, consider my wrist slapped for being so flippant with AR in Threads.

[aurels]

Hello,

We had the same issue on Rails 4.0.0. Migrating to 4.0.1 fixes the issue for us.

https://github.com/rails/rails/blob/v4.0.1/activerecord/CHANGELOG.md

At the end of the CHANGELOG :

Connections must be closed at the end of a thread. If not, your connection pool can fill and an exception will be raised.

Aaron Patterson

[rubytastic]

Also having this issue, also on 4.0.1

If interpret correctly the suggestion is to use

Thread.new do
  ActiveRecord::Base.connection_pool.with_connection do |conn|
    MyModel.find(123)
  end
end

whenever one needs a database connection from inside the worker right?
With many queries this seems cybersome

How does this relate to loading an module into the worker like:

  require 'games/general'
  include Games::General

These modules hold methods that connect to the database (instead directly from the worker).
Would one need to update these with the suggested block for every database query ? confused
It seems so since I am suffering the same issue mentioned in this thread.

Or is there a way to have initialiser and set this behaviour globally

Thanks in advanche for any suggestions regarding this matter.

[mperham]

@rubytastic The advice is if you create a Thread manually within a Worker, use with_connection inside that thread. 95% of people using Sidekiq don't create their own threads so this advice should be very rare and not needed for most people.

[rubytastic]

@mike so I misinterpreted above. Seems still the issue is caused by active record 4.0.1 regarding not closing the db connections. Im on 4.0.1 and the problem there persists. Do you have a recommendation regarding a solution?