Mysql2::Error: Lost connection to MySQL server during query #69

Closed
parasew opened this Issue Oct 12, 2010 · 71 comments

Comments

Projects
None yet

parasew commented Oct 12, 2010

is there any way to configure the timeout (similar to the ActiveRecord::Base.verification_timeout setting) with rails3?

Contributor

wvanbergen commented Oct 12, 2010

I can confirm that the connection problems is appearing again in 0.2.4. It started reappearing when moving from Ruby 1.8.7 to Ruby 1.9.2. Version 0.2.3 works fine in both Rubies. I am using Rails 3.0.

Owner

brianmario commented Oct 13, 2010

Are you using Unicorn or Passenger?

Owner

brianmario commented Oct 13, 2010

Also could you guys give master a try?

Contributor

wvanbergen commented Oct 14, 2010

I am getting this issue in development mode, using mongrel 1.2 beta. 0.2.4 triggers this issue really quickly, master takes some more time/requests before the exception is raised, but it still exists.

Owner

brianmario commented Oct 14, 2010

Would it be at all possible for you to build me a simple Rails app that reproduces the issue for you locally? I can give it a try on my end and see what I can find, I'm still unable to reproduce the issue :(

Contributor

wvanbergen commented Oct 15, 2010

I have been trying to build a sample app that triggers to bug, but currently I'm not able to reproduce it.

Some characteristics of my app: I am using different database connections in my app, all database connectivity uses ActiveRecord or uses ActiveRecord::Base.connection directly. I tend do to a lot of "quote" calls, and then execute a rather slow MySQL query. Pages that trigger the exception most often usually execute several (5-20) of these queries.

parasew commented Oct 15, 2010

i am doing exactly the same thing as wvanbergen - a change of the mysql timeouts in my.cnf also did not resolve those isses.

parasew commented Oct 15, 2010

i am using the thin webserver, version 1.2.7

Owner

brianmario commented Oct 15, 2010

Can you guys verify the wait_timeout Mysql variable is set to a number (seconds) higher than the number of seconds these queries are taking?
You can verify by checking the output of:

SHOW VARIABLES LIKE "wait_timeout"

from Rails console, or even log it to the Rails log during a request.

In some testing I did with another user, this value was set to 60 seconds. After having him increase this value his issue went away. Would you guys mind giving this a try to see what your values are?

Owner

brianmario commented Oct 15, 2010

Actually, I just added the ability to set wait_timeout in database.yml (which defaults to a pretty high number). Could you all give master a try?

Contributor

wvanbergen commented Oct 15, 2010

The exceptions still occur for me with master. Wait timeout is 2592000 for me. The queries that are causing the issues all take less than a second, but more than a trivial amount of time. I execute about 20 of them on my page. Every ~3 reloads, the exception will occur.

Owner

brianmario commented Oct 15, 2010

Would it be possible to allow me to remote connect to your machine (or the machine this is happening on) to debug? I'd love to get this figured out asap but am having a hell of a time reproducing it anywhere

Contributor

wvanbergen commented Oct 15, 2010

I guess we can set something up next week. Can you send me some contact details?

Owner

brianmario commented Oct 15, 2010

Sounds good, my email is seniorlopez at gmail - do you have ichat or something? Would be great if we could screen share

Owner

brianmario commented Oct 16, 2010

omg... I think I finally found and fixed the bug, could you all give master a try? :)

Contributor

wvanbergen commented Oct 16, 2010

ZOMG. PONIES!

parasew commented Oct 17, 2010

i tried with master and still have the same issues. although, not so often anymore.

Owner

brianmario commented Oct 18, 2010

ok, I'm skeptical the latest patches will help much - but still curious nonetheless ;)
Would you mind giving master another try?

Really sorry to keep having you guys try these out, I'm still (and always have been) unable to reproduce any of these issues on my machine or in any of my VMs, or else I'd test it myself :)

parasew commented Oct 18, 2010

hello brianmario - unfortunately the issues are still there. here is a suggestion on how to reproduce those issues: create an app with lots of test-data. then create two databases that run on another machine. leave the main database as sqlite3 database and connect to the other two databases with "establish_connection" as outlined here: http://j.mp/9gOPjt - when you call some queries on data of all the three databases, the error should occur.
wait_timeout is set to 28800 for me. i will try to create an example app with remote connections that raise this error - but it might take some time since i am quite busy this week.

Owner

brianmario commented Oct 18, 2010

Did you have to manually fix the gemfile? I was broken until this morning. How are you testing master?

wakiki commented Oct 19, 2010

Hi,

I also had the same problem with 0.2.4 and it went away 0.2.3. I'm using passenger 3 with ruby enterprise edition. Interestingly the problem was not there with ruby 1.9.2.

It's a very high traffic server with 100GB+ data. I'm happy for you to log in remotely to debug it, but it can't be too long since this is a live site and the errors effectively causes the site to go down.

Drop me an email and we can set it up.

Steve

Owner

brianmario commented Oct 19, 2010

Steve, have you tried master? It has some pretty important bugfixes, including one of which should fix your problem with 0.2.4 - I've had people test it out who had issues before and it appears to fix it for them. I'm planning on pushing 0.2.5 really soon (today?) but would like to hear a couple more success stories first ;)

wakiki commented Oct 19, 2010

OK I've just updated to master and indeed the error is gone - good stuff!

Let me know when you release 0.2.5 as I'll update the gemfile - don't like tracking head on production.

Again thanks!

Steve

Owner

brianmario commented Oct 19, 2010

That said, if you still have issues with master I'd love a chance to remotely debug it. I'm sure you've read above that I've had a really hard time reproducing the issue, so any chance I get to debug it on a machine that has the problem, I'll take.

parasew commented Oct 19, 2010

brianmario - please contact me directly via email on parasew at gmail so i can send you my code that triggers the error. since there is confidential information (and passwords inside) i want to send it to you directly.

Owner

brianmario commented Oct 19, 2010

just pushed 0.2.5 guys, thanks for all your help

wakiki commented Oct 20, 2010

Excellent!

I'm on 0.2.6, passenger 3.0, rails 3.0.3, ruby 1.9.2 .. and I'm getting this problem intermittently on production. Any suggestions? I switched off to the mysql gem and everything works fine there. My wait_timeout in mysql is set to 28800.

baconpat commented Feb 2, 2011

I just deployed an application to production using mysql2 v0.2.6, passenger v3.0.2, rails v2.3.10, ruby v1.8.7. Like a previous commenter I am using establish_connection to switch between different databases from request to request. The morning after the new deploy, when no one had hit the site for quite a few hours, the first hit to the DB got the "Lost connection" error.

The app had previously been using the regular mysql gem for several years without running into this problem. I am probably going to switch back to the mysql gem, but wanted to let you know this is still something that is occurring (at least occasionally) with 0.2.6.

The database has the default 28800 value for wait_timeout - but like I said this has been running with that default value for a couple of years without ever having this issue with the mysql gem.

Please let me know if there is any more information I can provide to help.

dedman commented Apr 5, 2011

I also am having a similar issue even after upgrading to Mysql2 0.2.6 (rails 2.3.8), connecting to 2 different dbs using establish_connection we will get the error. It seems to be intermittent, but I havn't been able to investigate fully yet.

For the moment switching one of the db's adapters back to mysql fixes the issue.

Let me know if I can provide any more info or test out fixes.

Thanks for your hard work.

Owner

brianmario commented Apr 5, 2011

Try upgrading to 0.2.7, let me know

Great! My setup that was not working on 0.2.6 works on 0.2.7

I'm getting this issue again on 0.2.7 / ruby-1.8.7-p334 / Rails 2.3.10

however 0.2.6 works fine.

cfjohan commented Dec 5, 2011

What's the status on this? I'm getting this error practically every day with rails 3.0.5, ruby 1.9.2, apache 2.2 and passenger 3.0.9. I'm connecting to a remote mysql server. I've tried mysql2 v.0.2.7, 0.2.6 and 0.2.13, and get the same thing on all of them.

This only seems to happen during UPDATEs when I'm trying to write about 50kb of data to a blob. The blob can hold 1 mb though, so there's no lack of space. max_allowed_packet = 32mb.

Owner

brianmario commented Dec 5, 2011

Whats your timeout set to?

cfjohan commented Dec 6, 2011

wait_timeout is 604 800 seconds (= 7 days, a bit overkill I admit). I assume that's the timeout you were asking for, it's the only timeout I've changed anyway (as far as I can remember). The MySQL log sheds some light on my particular problem though, it's filled with "Got an error reading communication packets" and "Got timeout writing communication packets". After some googling on that I found the possible solution of increasing the net_read_timeout. So I did yesterday, but that hasn't made any difference.

The standard value for net_read_timeout is 30 seconds though, so I'm wondering.. is it really supposed to take over 30 seconds to write less than 100 kb of data?

Update: increasing net_read_timeout has not made the error go away. Btw, I also get the error described in issue #66, although in my case the error is:

undefined method each' for nil:NilClass mysql2 (0.2.6) lib/active_record/connection_adapters/mysql2_adapter.rb:635:inselect'

Could these bugs be related? Neither of them go away when I switch to 0.2.7 or 0.2.13.

Owner

brianmario commented Dec 6, 2011

I wouldn't think it would take more than 750ms at most but I guess it all depends on what kind of load the system(s) is/are under. Have you tried using 0.2.18?

cfjohan commented Dec 9, 2011

The application runs on a pretty good machine and only has a few users, so system load should not be a problem at all. But I haven't tried 0.2.18, no. I thought 0.2.13 was the latest. I'll try it out and see if it works better.

It's interesting to note that I'm also running another rails application against the same database on the same machine as the database in question. That application runs under almost exactly the same environment (with mysql2 v.0.2.6), but I never get any issues there.

cfjohan commented Dec 13, 2011

0.2.18 doesn't make the problem go away, still get the same error several times a day. Usually when the application hasn't been used for a few hours. In fact, I get another (additional) error now:

Mysql2::Error: Lock wait timeout exceeded; try restarting transaction: INSERT INTO...

Although that doesn't happen very often. The MySQL log is filled with messages like (regarding the first error, not the new one):

[Warning] Aborted connection 37 to db: 'X' user: 'X' host: 'X.X.X.X' (Got an error reading communication packets)

Owner

brianmario commented Dec 13, 2011

If you're getting lock timeouts from your database, it could br over-utilized or misconfigured? Do you have this issue with the regular mysql gem?

cfjohan commented Dec 14, 2011

Hardly over-utilized. Maybe misconfigured, but I doubt it (I could post my config if that helps). The lock wait timeout is the standard 50s at the moment. I've tried switching to the regular mysql gem, but when I do I get encoding errors everywhere for some reason (lots of swedish characters in the app). So I'm stuck with this for now. This problem came up when I upgraded from rails 2.3 to 3.0 though, and the mysql2 gem came with that. Never saw this with rails 2.3 and the regular mysql gem.

Stranger still is that the other application never has this error, even though it's almost exactly the same environment. Only major difference is that that application is running on the same machine as the database. Although the particular database requests that seem to cause the error are never run in that app..

I am also getting this pretty randomly all over my site.

Why was this issue closed? Is there a fix or resolution not mentioned?

Owner

brianmario commented Feb 18, 2012

What is your setup like?

Using passenger or unicorn?
What version of ruby?
What operating system and architecture (32 or 64bit)?
Which version of MySQL server is being used, and which version of libmysql was mysql2 linked against?

I'm running mariadb 5.2.8 built from source on Lion 1.7.2, ruby 1.9.3 and mysql2 0.3.11 installed via gem. I'm testing with fatfreecrm pulled from git. By default, fatfree uses mongrel and postgres. I change the db to mysql (mariadb). The libmysql is that provided by mariadb. Here is the Gemfile:


gem 'rails', '3.1.3'
gem 'prototype-rails'

# Uncomment the database that you have configured in config/database.yml
# ----------------------------------------------------------------------
gem "mysql2", "0.3.11"
# gem "sqlite3"
gem "pg", "~> 0.12.2"

gem 'authlogic',           '~> 3.1.0'
gem 'acts_as_commentable', '~> 3.0.1'
gem 'acts-as-taggable-on', '~> 2.2.1'
gem 'haml',                '~> 3.1.3'
gem 'paperclip',           '~> 2.4.5'
gem 'will_paginate',       '~> 3.0.2'
gem 'acts_as_list',        '~> 0.1.4'
gem 'simple_form',         '~> 1.5.2'
gem 'ffaker',              '>= 1.12.0' # For loading demo data
gem 'uglifier'

group :heroku do
  gem 'unicorn', :platform => :ruby
end

 # Gems used only for assets and not required
 # in production environments by default.
group :assets do
  gem 'sass-rails', '>= 3.1.1'
  gem 'coffee-rails', '>= 3.1.1'
  gem 'therubyracer', :platform => :ruby  # C Ruby (MRI) or Rubinius, but NOT Windows
end

group :development, :test do
  unless ENV["CI"]
    gem 'ruby-debug',   :platform => :mri_18
    gem 'ruby-debug19', :platform => :mri_19, :require => 'ruby-debug' if RUBY_VERSION == "1.9.2"
    gem 'awesome_print'
  end

  gem 'test-unit',          '~> 2.4.3',  :platform => :mri_19, :require => false
  gem 'rspec-rails',        '~> 2.8.0'
  gem 'factory_girl'
end

group :development do
  platforms :ruby do
    # These gems give you an awesome development environment.
    gem 'thin'
    gem 'guard'             # https://github.com/guard/guard
    gem 'guard-rails'       # https://github.com/guard/guard-rails
    gem 'guard-sass'        # https://github.com/guard/guard-sass
    gem 'guard-spork'       # https://github.com/guard/guard-spork
    gem 'guard-rspec'       # https://github.com/guard/guard-rspec
    gem 'guard-livereload'  # https://github.com/guard/guard-livereload
    gem 'ruby_gntp'
    gem 'yajl-ruby'
  end
  # For annotating models with Schema information
  gem 'annotate', '~> 2.4.1.beta', :require => false, :group => :development
end

group :test do
  gem 'spork'
  gem 'factory_girl_rails', '~> 1.4.0'
  gem 'simplecov', :platform => :mri_19 unless ENV["CI"]  # Until Travis supports build artifacts
  gem 'fuubar'
end

Steps to reproduce:

pull fatfreecrm from git (i used fatfreecrm-fat_free_crm-8f2f679)
create the db in mysql
bundle exec rake crm:setup
and it fails while creating records

output is:

-- drop_table(:open_id_authentication_associations)
   -> 0.0013s
-- drop_table(:open_id_authentication_nonces)
   -> 0.0007s
==  DropOpenidTables: migrated (0.0022s) ======================================

==  AddAdminToUsers: migrating ================================================
-- add_column(:users, :admin, :boolean, {:null=>false, :default=>false})
   -> 0.0126s
rake aborted!
An error has occurred, all later migrations canceled:

<snip>
Mysql2::Error: Lost connection to MySQL server during query: SELECT  `users`.* FROM `users`  WHERE `users`.`deleted_at` IS NULL LIMIT 1

Tasks: TOP => db:migrate:reset => db:migrate
(See full trace by running task with --trace)
air11:fatfreecrm-fat_free_crm-8f2f679 peterk$ mysql -v
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 74
Server version: 5.2.8-MariaDB Source distribution
<snip>
Owner

brianmario commented Feb 18, 2012

Do you have this problem when using an official MySQL build? Just curious from a troubleshooting standpoint, I haven't actually tested using MariaDB's libmysql.

Funny you should ask: the reason I went with MariaDB several months back was that this problem was pervasive. For the last couple of years I've used nothing but official, Oracle sanctioned binaries and regularly encountered this bug with mysql2 (not sure about plain mysql). Until today MariaDB worked flawlessly.

Owner

brianmario commented Feb 18, 2012

interesting, I'll have to see if I can get an OSX vm setup to test this with

On Feb 18, 2012, at 1:47 PM, Peter Krnjevicreply@reply.github.com wrote:

Funny you should ask: the reason I went with MariaDB several months back was that this problem was pervasive. For the last couple of years I've used nothing but official, Oracle sanctioned binaries and regularly encountered this bug with mysql2 (not sure about plain mysql). Until today MariaDB worked flawlessly.


Reply to this email directly or view it on GitHub:
#69 (comment)

I don't know if this helps, but here is the rake stack trace:

air11:fatfreecrm-fat_free_crm-8f2f679 peterk$ bundle exec rake crm:setup --trace** Invoke crm:setup (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute crm:setup

Your database is about to be reset, so if you choose to proceed all the existing data will be lost.

Continue [yes/no]: yes
** Invoke db:migrate:reset (first_time)
** Invoke db:drop (first_time)
** Invoke db:load_config (first_time)
** Invoke rails_env (first_time)
** Execute rails_env
** Execute db:load_config
** Execute db:drop
** Invoke db:create (first_time)
** Invoke db:load_config 
** Execute db:create
fat_free_crm_test already exists
** Invoke db:migrate (first_time)
** Invoke environment 
** Invoke db:load_config 
** Execute db:migrate
==  CreateSessions: migrating =================================================
-- create_table(:sessions)
   -> 0.0122s
-- add_index(:sessions, :session_id)
   -> 0.0128s
-- add_index(:sessions, :updated_at)
   -> 0.0349s
==  CreateSessions: migrated (0.0602s) ========================================

==  CreateUsers: migrating ====================================================
-- create_table(:users, {:force=>true})
   -> 0.0115s
-- add_index(:users, [:username, :deleted_at], {:unique=>true})
   -> 0.0118s
-- add_index(:users, :email)
   -> 0.0132s
-- add_index(:users, :last_request_at)
   -> 0.0307s
-- add_index(:users, :remember_token)
   -> 0.0118s
-- add_index(:users, :perishable_token)
   -> 0.0130s
==  CreateUsers: migrated (0.0925s) ===========================================

==  CreateOpenidTables: migrating =============================================
-- create_table(:open_id_authentication_associations, {:force=>true})
   -> 0.0119s
-- create_table(:open_id_authentication_nonces, {:force=>true})
   -> 0.0109s
==  CreateOpenidTables: migrated (0.0230s) ====================================

==  CreateAccounts: migrating =================================================
-- create_table(:accounts, {:force=>true})
   -> 0.0102s
-- add_index(:accounts, [:user_id, :name, :deleted_at], {:unique=>true})
   -> 0.0116s
-- add_index(:accounts, :assigned_to)
   -> 0.0116s
==  CreateAccounts: migrated (0.0338s) ========================================

==  CreatePermissions: migrating ==============================================
-- create_table(:permissions, {:force=>true})
   -> 0.0137s
-- add_index(:permissions, :user_id)
   -> 0.0129s
==  CreatePermissions: migrated (0.0267s) =====================================

==  CreateSettings: migrating =================================================
-- create_table(:settings, {:force=>true})
   -> 0.0104s
-- add_index(:settings, :name)
   -> 0.0285s
==  CreateSettings: migrated (0.0391s) ========================================

==  CreatePreferences: migrating ==============================================
-- create_table(:preferences)
   -> 0.0106s
-- add_index(:preferences, [:user_id, :name])
   -> 0.0122s
==  CreatePreferences: migrated (0.0229s) =====================================

==  CreateCampaigns: migrating ================================================
-- create_table(:campaigns, {:force=>true})
   -> 0.0115s
-- add_index(:campaigns, [:user_id, :name, :deleted_at], {:unique=>true})
   -> 0.0122s
-- add_index(:campaigns, :assigned_to)
   -> 0.0123s
==  CreateCampaigns: migrated (0.0362s) =======================================

==  CreateLeads: migrating ====================================================
-- create_table(:leads, {:force=>true})
   -> 0.0116s
-- add_index(:leads, [:user_id, :last_name, :deleted_at], {:unique=>true})
   -> 0.0126s
-- add_index(:leads, :assigned_to)
   -> 0.0122s
==  CreateLeads: migrated (0.0367s) ===========================================

==  CreateContacts: migrating =================================================
-- create_table(:contacts, {:force=>true})
   -> 0.0113s
-- add_index(:contacts, [:user_id, :last_name, :deleted_at], {:unique=>true, :name=>"id_last_name_deleted"})
   -> 0.0124s
-- add_index(:contacts, :assigned_to)
   -> 0.0125s
==  CreateContacts: migrated (0.0365s) ========================================

==  CreateOpportunities: migrating ============================================
-- create_table(:opportunities, {:force=>true})
   -> 0.0109s
-- add_index(:opportunities, [:user_id, :name, :deleted_at], {:unique=>true, :name=>"id_name_deleted"})
   -> 0.0140s
-- add_index(:opportunities, :assigned_to)
   -> 0.0136s
==  CreateOpportunities: migrated (0.0389s) ===================================

==  CreateAccountContacts: migrating ==========================================
-- create_table(:account_contacts, {:force=>true})
   -> 0.0108s
==  CreateAccountContacts: migrated (0.0109s) =================================

==  CreateAccountOpportunities: migrating =====================================
-- create_table(:account_opportunities, {:force=>true})
   -> 0.0097s
==  CreateAccountOpportunities: migrated (0.0098s) ============================

==  CreateContactOpportunities: migrating =====================================
-- create_table(:contact_opportunities, {:force=>true})
   -> 0.0124s
==  CreateContactOpportunities: migrated (0.0125s) ============================

==  CreateTasks: migrating ====================================================
-- create_table(:tasks, {:force=>true})
   -> 0.0133s
-- add_index(:tasks, [:user_id, :name, :deleted_at], {:unique=>true})
   -> 0.0119s
-- add_index(:tasks, :assigned_to)
   -> 0.0253s
==  CreateTasks: migrated (0.0508s) ===========================================

==  CreateComments: migrating =================================================
-- create_table(:comments, {:force=>true})
   -> 0.0071s
==  CreateComments: migrated (0.0072s) ========================================

==  CreateActivities: migrating ===============================================
-- create_table(:activities, {:force=>true})
   -> 0.0121s
-- add_index(:activities, :user_id)
   -> 0.0282s
-- add_index(:activities, :created_at)
   -> 0.0144s
==  CreateActivities: migrated (0.0550s) ======================================

==  CreateAvatars: migrating ==================================================
-- create_table(:avatars)
   -> 0.0098s
==  CreateAvatars: migrated (0.0099s) =========================================

==  RenameRememberToken: migrating ============================================
-- rename_column(:users, :remember_token, :persistence_token)
   -> 0.0150s
-- remove_column(:users, :openid_identifier)
   -> 0.0114s
==  RenameRememberToken: migrated (0.0266s) ===================================

==  DropOpenidTables: migrating ===============================================
-- drop_table(:open_id_authentication_associations)
   -> 0.0016s
-- drop_table(:open_id_authentication_nonces)
   -> 0.0007s
==  DropOpenidTables: migrated (0.0025s) ======================================

==  AddAdminToUsers: migrating ================================================
-- add_column(:users, :admin, :boolean, {:null=>false, :default=>false})
   -> 0.0131s
rake aborted!
An error has occurred, all later migrations canceled:

Mysql2::Error: Lost connection to MySQL server during query: SELECT  `users`.* FROM `users`  WHERE `users`.`deleted_at` IS NULL LIMIT 1
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/mysql2_adapter.rb:687:in `query'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/mysql2_adapter.rb:687:in `block in exec_query'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:244:in `block in log'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activesupport-3.1.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract_adapter.rb:239:in `log'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/mysql2_adapter.rb:685:in `exec_query'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/mysql2_adapter.rb:679:in `select'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/database_statements.rb:18:in `select_all'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/query_cache.rb:63:in `select_all'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/base.rb:470:in `find_by_sql'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/relation.rb:111:in `to_a'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/relation.rb:129:in `to_a'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/relation/finder_methods.rb:376:in `find_first'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/relation/finder_methods.rb:122:in `first'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/base.rb:441:in `first'
/Users/peterk/Code/fatfreecrm-fat_free_crm-8f2f679/db/migrate/20100928030618_add_admin_to_users.rb:4:in `up'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:356:in `up'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:394:in `block (2 levels) in migrate'
/Users/peterk/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/benchmark.rb:280:in `measure'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:394:in `block in migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:185:in `with_connection'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:375:in `migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:507:in `migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:687:in `block (2 levels) in migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:744:in `call'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:744:in `ddl_transaction'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:686:in `block in migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:671:in `each'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:671:in `migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:549:in `up'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/migration.rb:530:in `migrate'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/activerecord-3.1.3/lib/active_record/railties/databases.rake:161:in `block (2 levels) in <top (required)>'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/Users/peterk/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:176:in `block in invoke_prerequisites'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:174:in `each'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:174:in `invoke_prerequisites'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:157:in `block in invoke_with_call_chain'
/Users/peterk/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/Users/peterk/Code/fatfreecrm-fat_free_crm-8f2f679/lib/tasks/fat_free_crm.rake:101:in `block (2 levels) in <top (required)>'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/Users/peterk/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block (2 levels) in top_level'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block in top_level'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `block in run'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/bin/rake:19:in `load'
/Users/peterk/.rvm/gems/ruby-1.9.3-p0/bin/rake:19:in `<main>'
Tasks: TOP => db:migrate:reset => db:migrate

I can confirm this is still happening on a virtual machine. Only when the host-machine is under heavy load though. Not like it has to wait for something - it fails almost instantly.

Newest updated version of Ubuntu (natty) with the mysql2 gem.

cfjohan commented Feb 20, 2012

This is still happening for me as well (same setup as before), but my solution is to move the application to the same external server the DB is running on. That seems to solve the problem completely.

Hmm, that won't work for me - both app and db are on the same box. My hope is that this will work on my CentOS production box and that the problem is only with OSX.

cfjohan commented Feb 24, 2012

You might be on to something there - the app that has this problem for me is actually running on OSX (server 10.6.4), the external server I'm moving it to is running Ubuntu.

I'm seeing this on Rails 3.2.2 and mysql2 0.3.11. I have a model called Tree. In the rails console I can call Tree.all and see all 316 rows returned. In my rake task, I make the same call and it fails with "Mysql2::Error: Lost connection to MySQL server during query". If I limit the query by Tree.all(:limit => 100) it works fine. I'm on OS X.
Thoughts?

I noticed that, same as for rgrimard above, the problem only occurs with rake tasks.

Well, I've found a solution but I'm not sure which step made the difference.

I was getting warnings building some code, unrelated to RoR, suggesting I update from Xcode 4.2 to 4.3.1. After a couple of false starts installing the command line tools, I found these two articles that helped get my build tools environment in order:

http://stackoverflow.com/questions/4390833/what-versions-of-gcc-llvm-and-clang-are-in-xcode-4
http://old.nabble.com/Xcode-4.3-Command-Line-Tools-installer-glitch-td33340567.html

Since this build went very smoothly, I tried re-brewing mariadb but now it wouldn't build at all (though mysql 5.5 brewed fine). Turns out the problem is clang compatibility - the answer is to brew with --use-llvm to force gcc-llvm instead of clang. But I only found that later.

To be thorough, I figured I should rebuild all components with Xcode 4.3.1. I found this excellent recipe from shigeya on SO describing exactly how to add an Xcode 4.3.1 built ruby 1.9.3-p125 to my rvm installation:

http://stackoverflow.com/questions/9651670/issue-updating-ruby-on-mac-with-xcode-4-3-1

Now that ruby was working, I gem installed bundler, dropped into my project and did a bundle install - bundler ran around for a while reloading/building the required gems. The only hiccup was nokogiri that choked on Apple's 'different' iconv. The answer was to link the inconv installed by rvm (for ruby) to /usr/local/lib using this:

ln -s ~/.rvm/usr/lib/libiconv.dylib /usr/local/lib/

Next, ran a few tests:

bundle exec rake db:setup
bundle exec rake db:populate -- this never passed before
bundle exec rake db:test:prepare
bundle exec rspec spec

And it all worked like a charm with mysql 5.5!

I then rebrewed mariadb 5.3.5 using my minimally updated brew recipe: https://gist.github.com/2041302
and this command: (NB. remove mysql beforehand or bad things will happen!)

brew install mariadb --use-llvm

This is the Gemfile I used to test: https://gist.github.com/2041325
While I didn't specify the versions when the tests were failing previously, the versions haven't changed, though now they are built against a slightly newer ruby (previously ruby 1.9.3-p0) with Xcode 4.3.1 instead of 4.2.

I'm afraid there are too many changes here to say exactly what solved the problem with mysql2, but I didn't touch mysql2 or it's config in any way.

Thanks to Brian for his prompt help and perhaps this information can help someone else stuck on OSX Lion.

Owner

brianmario commented Mar 15, 2012

Wow, thanks for posting all that! I'll keep this in mind for future issues like this

I'm using xcode 4.2.1 and have run into this same error when running a db:populate task:

Mysql2::Error: Lost connection to MySQL server during query: SELECT `

As a temporary hack -- I have no idea how this works or why -- I have been able to get the populate task to run by adding the --trace tag to it like so

$ bundle exec rake db:populate --trace

I hope this helps someone smarter than me with debugging. Paradoxically, since running --trace is part of debugging, and adding that tag makes the populate task work, I'm at a loss.

Why is this closed? Was there a general solution somewhere?

I'm still having the same problem on OS X Lion, with mysql2 (head), using ruby 1.8.7. Looks like there is a pretty involved work around above, but I couldn't get that working with 1.8.7.

Other options/workarounds that I'm missing?

I hope this is helpful for others and for debugging (if this issue is still under consideration):

I just discovered issue #43, and when I changed my import from LOAD DATA LOCAL INFILE to LOAD DATA INFILE the problem disappeared.

Like another user wrote above, for me moving the database to another machine solves the problem (somehow the problem only occurs for me when connected to localhost on a machine under heavy load?).

bdurand commented Apr 3, 2012

I had a similar issue recently and found this thread in researching the problem. In my case, the application contained a call to Kernel.fork that was used to run some code asynchronously in a child process. When a child process would exit, it would closed the open file descriptors including the MySQL connection it shared with the parent process causing the parent process to get random connection closed errors.

This is not a bug in the mysql2 gem, but might help anyone investigating a similar issue.

hadiS commented May 14, 2012

Hi, I have the same error in my rake task only, but it worked a few weeks ago. What i changed on my machine was upgrading to Xcode 4.3.2 and updating homebrew (homebrew forced me to execute 'brew link ....' on some files to be able to install rbenv through homebrew.). Since then i get this error and i have no idea how i can solve this.

Hi I have the same issue with hadiS and I have no clue to solve it either. :(

Hi, I'm getting this very same problem while doing queries to a "heavy" mysql view.

The server is not under high load, and, somehow, after this error occurs, everything stop working.. I have to restart the entire application..

fireflyk commented Apr 3, 2015

Is it solved now?

fireflyk commented Apr 3, 2015

I found that it is OK in 0.3.15. But the problem reappears from 0.3.16.
Is there anyone found that?

@fireflyk I am facing the same issue

I am also facing this problem. I am using
mysql2 0.3.18
ruby 2.1.2
centos 6.5. (EC2)
MySQL 5.6.22 (RDS)

I just downgraded to 0.3.15 and issue still occurs.

Mysql2::Error: Lost connection to MySQL server at 'reading initial communication packet', system error: 110

My issue was a false alarm. Sorry about that. It turned out to be that I had 2 hosts defined. This didn't jump out at me till I downgraded to 0.3.15 which gave me the ip address it was having problems with.

@jamesfebin We read the C source code. There is a problem. We start to research how to solve it.

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment