Lost connection to MySQL server during

[elistemann]

Hi,

we are using Octopus for replication, which is mostly working fine, but sometimes we get an "Lost connection to MySQL server during query" (~ for 0.1% of the requests we get).

I found this post, which seems to be the issue we have: http://stackoverflow.com/questions/4983385/lost-connection-to-mysql-server-during-query-error-with-rails3-octopus-replicat

We already have reconnect: true in our config, but you were writing that this is only working with unicorn, but not with passenger.
Could you find a solution for passenger?

Our app stack:

Rails 3.0.10
Octopus 0.3.4
Passenger 3.0.9
MySQL 5.1.41

Our shards.yml:

octopus:
replicated: true
fully_replicated: true
verify_connection: true
environments:
- production

production:
slave1:
encoding: utf8
adapter: mysql
database: database
username: slave
password:
socket: /var/run/mysqld/mysqld.sock
reconnect: true

Backtrace:

…/active_record/connection_adapters/abstract_adapter.rb: 207:in log_instrumentation' …lib/active_record/connection_adapters/mysql_adapter.rb: 289:inexecute'
…lib/active_record/connection_adapters/mysql_adapter.rb: 633:in select' …rd/connection_adapters/abstract/database_statements.rb: 7:inselect_all'
…ive_record/connection_adapters/abstract/query_cache.rb: 56:in select_all' …le/ruby/1.8/gems/ar-octopus-0.3.4/lib/octopus/proxy.rb: 177:insend'
…le/ruby/1.8/gems/ar-octopus-0.3.4/lib/octopus/proxy.rb: 177:in send_queries_to_selected_slave' …le/ruby/1.8/gems/ar-octopus-0.3.4/lib/octopus/proxy.rb: 140:inmethod_missing'
…1.8/gems/activerecord-3.0.10/lib/active_record/base.rb: 473:in find_by_sql_ActiveRecord_self_name_find_by_sql' …gems/activerecord-3.0.10/lib/active_record/relation.rb: 64:into_a'
…rd-3.0.10/lib/active_record/relation/finder_methods.rb: 341:in find_first' …rd-3.0.10/lib/active_record/relation/finder_methods.rb: 122:infirst'
…rd-3.0.10/lib/active_record/relation/finder_methods.rb: 242:in send' …rd-3.0.10/lib/active_record/relation/finder_methods.rb: 242:infind_by_attributes'
…1.8/gems/activerecord-3.0.10/lib/active_record/base.rb: 993:in send' …1.8/gems/activerecord-3.0.10/lib/active_record/base.rb: 993:inmethod_missing_without_create_or_update'
…11110100150/vendor/plugins/create_or_update_by/init.rb: 37:in method_missing' …111110100150/app/controllers/application_controller.rb: 79:infind_site'

Thanks, Enrico

[thiagopradi]

This erro also happens on unicorn, and putting this line on the after_fork seems to fix it:

ActiveRecord::Base.connection_proxy.instance_variable_get(:@shards).each {|k,v| v.clear_reloadable_connections! }

Try putting this on the Passenger config. (I'm not sure how you should do this, but here is my guess):
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
ActiveRecord::Base.connection_proxy.instance_variable_get(:@shards).each {|k,v| v.clear_reloadable_connections! }
end
end
end

[jobster]

FWIW, the fix that @tchandy mentioned above seems to be working for me. A similar fix is also in the StackOverflow post that the OP mentioned...look for the post from Jan 30 at the bottom.

If anyone else can confirm this, then this might be closed!

Edit: the fix that is working for me is the one for Passenger, I haven't tried Unicorn at all.

[sobrinho]

This is not a elegant solution but I think it will work:

if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    if forked
      ActiveRecord::Base.connection_proxy.instance_variable_get(:@shards).each do |shard, connection_pool|
        connection_pool.disconnect!
      end
    end
  end
end

But I'm using puma these days which not require this kind of configuration.

Closing :)

[mshibuya]

We also have been suffering from intermittent 'Lost connection' and 'ActiveModel::MissingAttributeError' errors.
After applying @sobrinho 's solution, those errors have gone completely.
Thanks so much!

[sobrinho]

😆