using rails connect hangs when initialized from script/server, but not from script/console

[angelf]

Hi,

A certain action in a rails 2.3.5 app stalls when using the mysql2 gem, after investigation we have narrowed this to mysql2 establishing the connection. The loop is on client.c in the function nogvl_connect. When stalled, the script/server process cannot be interrupted and only a kill -KILL will stop it.

The same connection arguments work just fine if the connection is established from script/console (or from a stand-alone ruby process, if that matters). Also the old mysql gem works fine from rails, and the mysql native client can connect perfectly to the server.

To make it clearer, this code hangs forever:

class TestController < ApplicationController
  def test
    Mysql2::Client.new(:host => "our_host", :username => "xxx", :password => "xxx", :port => 3306, :database => "database", :socket => nil, :flags => 2)
  end
end

but this works perfectly:

>script/console
>>  Mysql2::Client.new(:host => "our_host", :username => "xxx", :password => "xxx", :port => 3306, :database => "database", :socket => nil, :flags => 2)

This has been replicated in two machines (Mac OS X, Snow Leopard, 64 bits). The target server is

mysql> status;
--------------
mysql  Ver 14.14 Distrib 5.1.30, for suse-linux-gnu (i686) using readline 5.2
Server version:     5.1.30-log SUSE MySQL RPM
..

What information can we provide for a useful bug report? We have the C and ruby call stack but that doesn't seem very useful.

[brianmario]

What version of ruby (also - 32bit/64bit?), which webserver are you using?

[brianmario]

Any possibility of a smaller reproducible test-case?

I'm unable to replicate the issue.

[angelf]

• ruby is 1.8.7 (2009-06-12 patchlevel 174) [universal-darwin10.0]
• server is webrick (script/server)
• architecture is x86_64
• OS is snow leopard

I will try to create a sample rails app with the minimum code that can reproduce the problem. I think I mentioned that only the connection to some servers stall, connecting to other servers works ok. The surprising element is that the troublesome servers work perfectly from script/console AND when using the older mysql gem.

Also it may be relevant that the app that is having the issue has a about 10 mysql connections to other servers (it's a sharded app, lots of databases, etc.) already stablished when the problem occurs. Don't know if we may be finding some global maximum (will do some tests).

Thanks for your help,

[angelf]

Hi,

This is the simplest test case I could create.

SETUP:

• gem install mysql2 -v 0.2.3

• create a new rails rails 2.3.5 app

• add this controller

  class TestController < ApplicationController
  def test
  require 'mysql2'
  Mysql2::Client.new(:host => "our_host1", :username => "our_user", :password => "pwd", :port => 3306, :database => "database", :socket => nil, :flags => 2)
  render :text => "hi"
  end
  end

BEHAVIOUR:

webrick fails:

• boot webrick with script/server
• load http//localhost:3000/test/test --- ruby process gets locked in a non-interruptible state

mongrel_rails fails:

• boot mongrel with mongrel_rails
• load http//localhost:3000/test/test --- ruby process gets locked in a non-interruptible state

script/console works ok:

• booting with script/console and then running app.get("test/test") works fine (returns 200 code)

In order to reproduce the bug we need:

webrick + mysql2 + connection to a certain server
or
mongrel_cluster + mysql2 + connection to a certain server

mysql2 can connect to other mysql servers without any problem, using mysql gem solves the problem, running the code from script/console solves the problem

This is the info of one target mysql servers that work well:
mysql Ver 14.14 Distrib 5.1.43, for debian-linux-gnu (x86_64) using EditLine wrapper

And this is two servers that do not
Server version: 5.0.67-log SUSE MySQL RPM

Server version: 5.1.47-rel11.0-log (Percona Server (GPL), 11.0 , Revision 47)

so it doesn't seem related to a certain mysql version.

I don't have any idea at all of what could be happening here...

[angelf]

could a network sniffer help in any way?

[brianmario]

A recent patch fixed a long standing bug that was probably the culprit of all of the strange hard to reproduce (for me) bugs people have been seeing. Would you mind giving master a try?

[angelf]

sure! will get back to you tomorrow

[angelf]

Sorry for the delay. Unfortunately the problem persists.

[thhooommmm]

I'm experiencing the exact same problem with rails 2.3.5+ and the latest source (updated as of the last commit). It simply hangs on boot every time with webrick, thin, mongrel, etc.

[npadgett]

Same thing here.

$ rails -v
Rails 2.3.8
$ ruby -v
ruby 1.8.7 (2009-06-12 patchlevel 174) [universal-darwin10.0]
$ gem list | grep mysql2
mysql2 (0.2.4)
$ script/server
=> Booting WEBrick
=> Rails 2.3.8 application starting on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2010-10-02 20:43:36] INFO WEBrick 1.3.1
[2010-10-02 20:43:36] INFO ruby 1.8.7 (2009-06-12) [universal-darwin10.0]
[2010-10-02 20:43:36] INFO WEBrick::HTTPServer#start: pid=275 port=3000
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.6.4
BuildVersion: 10F569

[thhooommmm]

Also, the problem persists with ubuntu 10.04, both 32-bit and 64-bit, though it occasionally starts and finishes loading webrick on the 64-bit once in a blue moon. I usually have to restart the AWS instance to get it to do that again, however.

[brianmario]

Could you recreate me a small sample Rails application that reproduces the issue for you? Unfortunately there's nothing I can do until I can reproduce it on my end as well.

[brianmario]

Or if you guys don't mind giving me temporary access to a server that has this issue (and has some debugging tools installed) that would be awesome as well

[angelf]

Hi Brian,

I believe the problem must be associated to our specific environment, I actually was able to reproduce it in app that was created from scratch (see Sep 3th message)

I'll try to arrange an ssh access to a server where you can see the problem live.

Thanks for your help!

[angelf]

Hi Brian,

While on the process of setting up the server to do the test you requested, I have found that the issue could be related to network latency.

The problem cannot be reproduced when I connect to a server from a machine in the same datacenter, but happens always when I connect:

• from a different datacenter using a VPN and private IP
• from a different datacenter using a public IP address
• from a different datacenter connecting through a public machine that does (unencrypted) port forwarding

I have also verified that the problem occurs both when the client is running Mac OS X and Ubuntu 9.10 (karmic).

Can you send me (angel{at}vlex.com) your IP and I'll set-up the firewall permissions so you can test this from your own machine?

Thanks!

[thhooommmm]

Brian, I'll set up a test server right now and send you a message with contact details & an ssh key (its going to be on aws) for you to access it.

[brianmario]

Awesome thank you!

[npadgett]

Following John@AWS's recommendation (http://developer.amazonwebservices.com/connect/thread.jspa?messageID=153250&#153250), I set character_set_server=utf8 and collation_server=utf8_general_ci. Still no luck. I also tried upgrading to 0.2.6.

[npadgett]

This appears to be something environment specific. I am not able to connect to my staging server from my machine with mysql2. However, I am able to connect to my staging server from Heroku with mysql2. In both cases I am using the same webapp.

[npadgett]

I am now able to connect to my mysql server with mysql2. However, my co-worker cannot. I have no idea what differences there are between our two machines.

[brianmario]

What do your database.yml files look like?

[npadgett]

development: 
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: schema_name
  pool: 5
  wait_timeout: 5
  username: user
  password: pass
  host: data.base.com

[npadgett]

We discovered that once we installed RVM (http://rvm.beginrescueend.com/), this problem went away.

[brianmario]

That's definitely interesting...
I've run into other strange issues with the system ruby that ships with OSX, and have since always used whatever the latest ruby is (1.8 and 1.9 at whatever patch-level) via rvm.

Are you guys still having issues on Ubuntu as well?

[npadgett]

Ubuntu was another team.

[brianmario]

Could you try setting a connect_timeout in database.yml to something very low?

[jazminschroeder]

I'm experiencing the same problem with rails3.0.4 on Windows7, Is there a known solution for this issue? (OSX and rails3.0.4 works just fine)

[brianmario]

Could you guys give this another try with 0.2.7?

[dw3w4at]

I'm in same hang-up too (with rails3.0.6 on Windows7). Even if I set wait_timeout or connect_timeout.

Then I try to connect manually in Rails console,

dbconf = ActiveRecord::Base.configurations['development']
ActiveRecord::Base.establish_connection dbconf

It's okay until above, but

dbconn = ActiveRecord::Base.connection

is hanged, And stack-traces are shown when Ctrl+C, and it indicates "mysql2-0.2.7/lib/active_reco
rd/connection_adapters/mysql2_adapter.rb:312" .

[Ritesh-Kumar]

Hi all,

This issue is turning out to be a nightmare for many users including me. 'rake' task and rails server hanging with mysql2.
If you go through this post - "http://rorguide.blogspot.com/2011/03/installing-mysql2-gem-on-ruby-192-and.html", you will find many questions and scenarios where it happened but unfortunately I have no answers for them. Hope this might help you in getting to the root cause and solving it.

Thanks

[inspire22]

I'm having issues with mysql2 hanging indefinitely as well. Hard to tell why or what's causing it. Using unicorn and sending an ActiveRecord::Base.establish_connection after forking.

[trevorsmith]

We just dealt with this same maddening issue on one of our projects, and found HTTPClient to be the culprit, specifically its custom implementation of Timeout.

The Ruby process hangs as libmysqlclient.so attempts to establish a connection over the network. Attaching strace revealed an infinite loop of 20 system calls:

socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
[...]
connect(5, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("xx.xx.xx.xx")}, 16) = -1 EINPROGRESS (Operation now in progress)
fcntl(5, F_SETFL, O_RDWR)               = 0
poll([{fd=5, events=POLLIN|POLLPRI}], 1, 30000) = ? ERESTART_RESTARTBLOCK (To be restarted)
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a)                      = -1 EINTR (Interrupted system call)
shutdown(5, 2 /* send and receive */)   = 0
close(5)                                = 0

SIGVTALRM is used by Ruby to emulate threading, interrupting the current "thread" using an interval timer. EINTR is returned from the signal handler, and MySQL (or the mysql2 gem, I haven't investigated) responds by aborting and retrying connect().

We looked through our gems and found HTTPClient uses threads in a custom implementation of Timeout.timeout. Sure enough, we stopped loading the library, and the hang vanished.

I don't have the time now to familiarize myself with the differences between HTTPClient's timeout.rb and the core implementation and discover where the bug lies. Is mysql2 or ActiveRecord doing something with threads or timeouts that might have similar effects?

Our environment:

Ubuntu 10.04 LTS
Ruby 1.8.7 patchlevel 249
ActiveRecord 3.0.7
HTTPClient 2.2.0.2
mysql-client 5.1.41

[rdp]

@windows users, see my comments to #142

@Others: I wasn't able to reproduce this with mingw 1.8.7, mysql2 0.2.7, rails 2.3.8

[thobruk]

I am having a similar problem. If I create a virtual machine in VirtualBox with Ubuntu 11.04, install rails 3.0.7 and mysql2 0.2.7, create an empty application with an empty model and controller, modify Gemfile to add mysql2, modify database.yml to point a remote mysql database, then run 'rails server', it will hang on the first access of the controller. I have tried both 64bit and 32bit. I am using ruby 1.8.7.

[brianmario]

Would you give 0.2.10 a try?

[thobruk]

Just tried 0.2.10. The problem still occurs. There appears to be a specific problem with the ruby package that is built for ubuntu. If I build it myself (1.8.7 p302) or use rvm (which does the same thing) the problem goes away.

[brianmario]

What's the version of Ruby that ships with ubuntu that you're having an issue with?

[dw3w4at]

For me (MariaDB on Windows), gem installation itself is failed after 0.2.9. Until 0.2.8, gem installation is succeeded.

[thobruk]

1.8.7 p302, I believe. A basic installation of Ubuntu 11.04 (for example, but I get the problem with older Ubuntus), run 'apt-get install ruby rubygems ruby-devel' will give you the problem.

[dw3w4at]

For me (MariaDB on Windows), both gem installation and rails execution go successfully in version 0.2.11 !

[johnkolen]

Here's another datapoint. I'm access mysql and mongo locally and tried adding a remote mysql connection today and ran into the problems described above ('rails console' works, 'rails server' hangs) on ubuntu 11.04. I switched to serving with thin, and the problem went away. I noticed that bundle grabbed 'eventmachine' during the install. Tried mongrel--it hangs too.

[brianmario]

closing due to inactivity. please reopen if you're still having trouble on the latest gems

[TheMadDeveloper]

I second johnkolen's datapoint. I just tried starting up a fresh app connecting to a remote mysql database and was experiencing the very same hanging behavior described throughout the thread. Switching from WEBrick to thin seemingly has bypassed the issue.

FYI, I'm running on OSX (Lion) with the prepackaged ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin11.0], Rails 3.2.8, and mysql2 0.3.11 (also had same issues with 0.3.12b4).