Why is it necessary to reconnect each iteration for parallelizing ActiveRecord?

[phene]

Why is it necessary to run ActiveRecord::Base.connection.reconnect! in order to parallelize the processing of ActiveRecord::Base objects? ActiveRecord::Base is thread safe, so what is this actually fixing? Reconnecting every iteration is extremely wasteful.

[grosser]

Try and see if it works, I had lots of issues without this, maybe it's better in AR3,
for forged processes it definetly has to happen.

You can reconnect once per fork, but thats also tricky :)

[phene]

I am also having a lot of issues using the Parallel gem and ActiveRecord, and without a decent explanation from you as to why reconnection for every iteration is necessary, I'm forced to conclude that this gem is unsafe with ActiveRecord.

[grosser]

conclude what you want, I don't have any sla's :D

afaik you have to reconnect affter forking, not sure about threading, but I saw a lot of weird issues so far, so I always use forking + reconnect

[grosser]

whoops wrong button

[grosser]

if you can digg down and find any issue let me know, or add it to the readme
even having some reproducable failing test-case would be nice

[phene]

http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html seems to indicate that though ActiveRecord is thread safe, you have to manage the connection pool manually. This seems like something that this gem could/should support.

[grosser]

This gem should ideally not have to worry about issues of other gems,
if you can fix it without hacking AR and making it opt-in/or a pasteable snippet that would be great :)

[phene]

I found one possible solution using ActiveRecord::Base.connection_pool.with_connection, and I suggest you add it (or a better example) to the readme:

Parallel.each(Message.where(some_condition).all, :in_threads => 8) do |message|
  ActiveRecord::Base.connection_pool.with_connection do
    message.update_attribute(:some_attribute, some_value)
  end
end

[grosser]

Added it to the readme, let's see if it works :)

[chamnap]

It works well for me with mongoid.
Mongoid v2: Mongoid.reconnect!
Mongoid v3: Mongoid.default_session.disconnect