ActiveRecord with Mysql2 outside Rack apps

[jnak]

Hi everyone,

I have an issue that I believe is not 100% specific to em-synchrony, but I didn't know where to post it.

I'm trying to reuse my ActiveRecords models with em-synchrony and em_mysql2 outside of rails. This means that I don't use @mperham rack-fiber_pool. Because of this, it seems that AR will use as many mysql connections as fibers calling the DB, unlike the em-synchrony connexion pool. I tried using the :pool parameter (which defaults to 5 anyway), but it does not change anything.

Am I doing something wrong ? Does anyone use async AR outside of a rack app ?
Some help here would be much appreciated :)

Cheers,
Julien

[igrigorik]

The short answer is you can't use the built-in pool. The way to limit your concurrency here is to either import the FiberPool from the middleware, or use the fiber-aware ConnectionPool from em-synchrony. In case of FiberPool, you're explicitly limiting the number of fibers in which requests can be executed, and in the case of ConnectionPool you're explicitly limiting the number of mysql connections which can be shared between any number of fibers.

[jnak]

Thanks Ilya,

That's what I was suspecting. I already use the ConnectionPool as expected. But I was kind of tired to write some of AR model again (not very DRY) and doing SQL queries by hand.

I've been looking at ActiveRecord ConnectionPool and it seems like it would be possible to get something similar EM::Synchrony Connection Pool, but I'm not sure how. I'll be investigating more soon. If you have any ideas, let me know.

[igrigorik]

Hmm, you don't have to reinvent the wheel.. If you look under the hood, what FiberPool middleware does is basically rip out the default ConnectionPool logic and replace it with Fiber aware stuff. And now that I think about it.. so does the mysql2 gem!

https://github.com/brianmario/mysql2/blob/master/lib/active_record/fiber_patches.rb

In theory, you should be able to use AR + mysql2 + fibers and control the connection pool stuff right through AR (so I guess I retract my original statement :-))

[jnak]

When I use em_mysql2 adapter, it automalically loads active_record/fiber_patches, but it does not work as expected for some reasons. Running something like this will break as Mysql server says it has too many connections.

require 'em-synchrony'
require 'active_record'

require './model/an_ar_model'

ActiveRecord::Base.establish_connection(
  :database => 'test',
  :pool => 3
)

EM.synchrony do
  200.times do
    Fiber.new do
      p AnArModel.find ...
    end.resume
  end
end

I don't understand why the :pool parameter passed to ConnectionPool in the spec hash does not limit the number of connection...

[igrigorik]

Hmm, that sounds like a bug to be opened on the mysql2 repo. If I get the time, can take a look at it over the weekend..

[jnak]

I'll have a closer look too. Getting EM-synchrony to work smoothly with AR would be big plus for the project :)

[mperham]

For the record, Brian and I aren't really interested in maintaining
the mysql2 EM fiber stuff. I'd love to see it migrate to em-synchrony
where it can get better support.

On Wed, Aug 24, 2011 at 7:45 PM, igrigorik
reply@reply.github.com
wrote:

Hmm, that sounds like a bug to be opened on the mysql2 repo. If I get the time, can take a look at it over the weekend..

Reply to this email directly or view it on GitHub:
#51 (comment)

[igrigorik]

@mperham: (to confirm) so, move both the mysql2 driver and the fiber patches into em-synchrony?

[mperham]

That's fine with me. Take the code and let me know when you are happy and ready to support it. I'll remove it from mysql2 and update the docs - I imagine everyone using it is already using em-synchrony anyways.

[igrigorik]

Moved the fiber logic + AR adapter into synchrony: 86feed2

@mperham: in theory, you can drop the patches / AR stuff from mysql2.

[fonzo14]

Outside of a rails app, with :

gem 'em-synchrony', :git => 'https://github.com/igrigorik/em-synchrony.git'
gem 'eventmachine', :git => 'git://github.com/eventmachine/eventmachine.git'
gem 'activesupport', '>= 3.1.0.rc6'
gem 'activerecord', '>= 3.1.0.rc6'
gem 'mysql2', :git => 'https://github.com/brianmario/mysql2.git'

The code

require 'em-synchrony'
require 'em-synchrony/activerecord'

ActiveRecord::Base.establish_connection(config.database_settings)

gives me this error :

Please install the em_mysql2 adapter: gem install activerecord-em_mysql2-adapter (no such file to load -- active_record/connection_adapters/em_mysql2_adapter)
/home/toma/.rvm/gems/ruby-1.9.2-p180@semanteek/gems/activerecord-3.1.0.rc8/lib/active_record/connection_adapters/abstract/connection_specification.rb:71:in rescue in establish_connection' /home/toma/.rvm/gems/ruby-1.9.2-p180@semanteek/gems/activerecord-3.1.0.rc8/lib/active_record/connection_adapters/abstract/connection_specification.rb:68:inestablish_connection'

UPDATE : it works fine if i switch to gem 'mysql2' instead of gem 'mysql2', :git => 'https://github.com/brianmario/mysql2.git'. Rails is looking for the adapter in active_record/connection_adapters/em_mysql2_adapter. So it does not work with master codes em-synchrony/mysql2 as the adapter disappeared from mysql2 and is now under the em-synchrony folder

[jnak]

@fonzo14 Yeah, that s normal since ilya just moved the logic inside em-synchrony. I would be interested on how you deal with the fiber pool sizes (see my message above).
Cheers

[igrigorik]

Hey guys, closing this, let's consolidate the discussion in #58 - same issue.

[jnak]

Ok. Totally makes sense. See you guys there.