Mysql2::Error: MySQL server has gone away under Rails 3.1.1 #213

Closed
thoughtafter opened this Issue Oct 7, 2011 · 23 comments

Projects

None yet
@thoughtafter

I upgraded an app to Rails 3.1.1 and pushed it to a staging server. I'm seeing lots of "Mysql2::Error: MySQL server has gone away" including in the console. This could easily be a bug in Rails 3.1.1 but I thought I would report it here to see if anyone else encounters it.

I get the issue as well, It happens sporadically and I can't consistently reproduce it. Running Rails 3.1.0

I'm getting it as well, can't seem to reconnect when this error happens as well. Only happens for me when there is an file upload though....

flackou commented Dec 22, 2011

I have the exact same issue (I asked a question on stackoverflow). Have you find a way to solve this ?

Haven't had time yet to go back and check it. Was going to try and solve it over Christmas, I'll let you know if I get it working

Oh actually, wrong issue. I found my problem, haven't figured out why it is happening though. For me it was an issue with the ruby-audioinfo gem. Below is the issue:

moumar/ruby-audioinfo#2 (comment)

flackou commented Dec 22, 2011

In fact, I don't have the exact same issue as you (not upload related in my
case). My previous message was confusing, sorry. Thanks for your help. @
thoughtafter https://github.com/thoughtafter
@sgriegoliethttps://github.com/sgriegoliet have
you found an explanation to this bug ?

On Thu, Dec 22, 2011 at 11:46 PM, Jon <
reply@reply.github.com

wrote:

Oh actually, wrong issue. I found my problem, haven't figured out why it
is happening though. For me it was an issue with the ruby-audioinfo gem.
Below is the issue:

moumar/ruby-audioinfo#2 (comment)


Reply to this email directly or view it on GitHub:
#213 (comment)

flackou commented Dec 23, 2011

I think I solved my issue : everything is explained in my stackoverflow question. The wait_timeout set by ActiveRecord for mysql2 seems to be too high and exceeds the MySQL max value for this param. Could you tell me if setting wait_timeout: 2147483 in your database.yml solve your bugs ?

As strange as it seems, installing imagemagick solved this problem for me. During a mysql transaction, the app was shelling out out to Open3.popen3('identify', ...), but since imagemagick wasn't installed, it failed and somehow that caused the problem. I'd bet it's some threading error, or possibly a fork problem...

This code consistently throws this error

ActiveRecord::StatementInvalid: Mysql2::Error: MySQL server has gone away: SELECT * FROM my_models WHERE (my_models.id = 1)

MyModel.find(1) 
Open3.popen3('any_nonexistent_command') { |x,y,z|  }
sleep(2) # race!
MyModel.find(1) # always throws an error

or, distilled to its essence:

connection = MyModel.connection.instance_variable_get(:@connection)
connection.query("SELECT 1")
pid = fork {
  fork {
    # Open3.popen3 redirects STD(IN|OUT|ERR) here.
    exec("any_nonexistent_command")
  }
  exit!(0)
}
Process.waitpid(pid)
sleep(2) # race!
connection.query("SELECT 1")

It seems like an exception is being raised in the grand-child process. The exception is not bubbled up to the original process, but since the grand-child process still has access to the mysql2 connection resource owned by the original process, it's being disconnected by mysql2 spec it "should close the connection when an exception is raised"

Open3.popen3 redirects STDERR in the grand-child process, so the exception is never seen. You can see the exception if you read it like this:

MyModel.find(1) 
puts Open3.popen3('any_nonexistent_command') { |stdin,stdout,stderr| stderr.read } # prints the exception that is uncaught in the grandchild process
sleep(2) # race!
MyModel.find(1) # always throws an error

@johndouthat, I played a bit with your examples and they worked for me, but a slightly
modified version did reproduce the problem (or perhaps expose a new one)

MyModel.transaction do
  if fork
    # parent
    Process.wait
    MyModel.first
  else
    # child does nothing
  end
end

It looks like forking inside transactions causes bad things to happen. A work around
is to use Kernel.exit! in the child process.

@andykitchen It looks like a race condition. It happens reliably on my quad core machine w/HT, but on a single core VM, it never seems to happen. Adding sleep(2) before the second query will likely "fix" the race. I'll add the sleep statement to my earlier comment.

I'm having this issue as well, it happens sporadically and I can't consistently reproduce it. Running Rails 3.1.1;
For now I've set wait_timeout: 2147483 in database.yml and seems that fix it (not sure yet - doing tests).

sgonyea commented May 7, 2012

@andreydjason, @thoughtafter etc...

Have you tried setting "reconnect: true" in your database.yml? I just ran into this issue, and was on the verge of blaming wait_timeout as well. The max wait_timeout value of "2147483" only applies when MySQL is run on Windows.

I'm using this option now for tests, I removed the "wait_timeout" option. As soon I get some results, I'll comment here.

any results?

Collaborator

Is this still a problem in current Rails and mysql2's for anybody here? If not I will close the issue.

Can somebody please confirm or refute if Rails 3.2.14 supports configuring wait_timeout via database.yml as discussed here?

Collaborator

Closing for lack of updates.

@sodabrew sodabrew closed this Oct 17, 2013
@eam eam added a commit to eam/mysql2 that referenced this issue Dec 18, 2013
@tamird @eam tamird + eam Failing test for #213 24f3cc5
@eam eam added a commit to eam/mysql2 that referenced this issue Dec 18, 2013
@eam eam WIP #213 8cb96ca
@eam eam added a commit to eam/mysql2 that referenced this issue Dec 18, 2013
@eam eam WIP #213. 8b9e52f
@eam eam added a commit to eam/mysql2 that referenced this issue Dec 18, 2013
@eam eam When the Ruby interpreter forks after creating a mysql2 instance, eac…
…h mysql2

object will share a socket and the first object to be garbage collected will
destroy the socket state for the second object. This appears to be the
underlying bug in issue #213.

Because the GC is non-deterministic, sometimes child processes will exit
without calling the GC on copies of the mysql2 client object before ending the
child process. In our tests we force the GC to run, exposing the issue in a
reliable manner.

I work around this issue by preventing the GC from collecting instances in
child processes entirely. This patch causes the GC to only collect mysql2
objects within the same process which spawned them (within the parent). #close
continues to function as it has, in case it is desired to force cleanup of the
socket state.

It would be ideal to have a more granular or configurable interface for cleanup
than mysql_close(), allowing us to perform everything but the call to shutdown().
e285778
@eam eam added a commit to eam/mysql2 that referenced this issue Dec 18, 2013
@eam eam Don’t close shared sockets (fixes #213)
When the Ruby interpreter forks after creating a mysql2 object, each mysql2
object will share the socket and the first instance to be garbage collected
will destroy the socket state for other objects in other processes. This
appears to be the underlying bug in issue #213.

Because the GC is non-deterministic sometimes child processes will exit
without calling the GC on copies in children before the children exit. In our
tests we force the GC to run, exposing the issue in a reliable manner.

I work around this issue by preventing the GC from collecting instances in
child processes entirely. This patch causes the GC to only collect mysql2
objects within the same process which spawned them (within the parent). #close
continues to function as it has, in case it is desired to force cleanup of the
socket state.

It would be ideal to have a more granular or configurable interface for cleanup
than mysql_close(), allowing us to perform everything but the call to shutdown().
b0a165a
@eam eam added a commit to eam/mysql2 that referenced this issue Dec 18, 2013
@eam eam Don’t shutdown() shared sockets (fixes #213)
When the Ruby interpreter forks after creating a mysql2 object, each mysql2
object will share the socket and the first instance to be garbage collected
will destroy the socket state for other objects in other processes by calling
shutdown(). This appears to be the underlying bug in issue #213.

Because the GC is non-deterministic sometimes child processes will exit
without calling the GC on copies in children before the children exit. In our
tests we force the GC to run, exposing the issue in a reliable manner.

I work around this issue by preventing the GC from collecting instances in
child processes entirely. This patch causes the GC to only collect mysql2
objects within the same process which spawned them (within the parent). #close
continues to function as it has, in case it is desired to force cleanup of the
socket state.

It would be ideal to have a more granular or configurable interface for cleanup
than mysql_close(), allowing us to perform everything but the call to
shutdown(), and to only close() descriptors in the child without side effects.
1153eae
Collaborator

Reopening for @eam has found the cause!

@sodabrew sodabrew reopened this Dec 18, 2013
Owner

Sounds possibly related to http://bugs.mysql.com/bug.php?id=65956

jogaco commented Feb 9, 2014

Same issue here, with Rails 3.2.16 and mysql2 0.3.13.

@eam eam added a commit to eam/mysql2 that referenced this issue Feb 10, 2014
@eam eam Avoid shutdown() on shared sockets (fixes #213)
Close the mysql socket before calling mysql_close(). mysql_close() calls
shutdown() on the attached socket. In a situation involving fork() and copies
of an object spanning multiple interpreter instances this will result in the
first instance to GC breaking all other instances sharing the socket.

dylanahsmith notes that we can avoid this by simply closing the socket before
entering mysql_close(), preventing the shutdown() from having any impact on the
shared socket state.
789f8ab
@eam eam added a commit to eam/mysql2 that referenced this issue Feb 10, 2014
@eam eam Avoid shutdown() on shared sockets (fixes #213)
Close the mysql socket before calling mysql_close(). mysql_close() calls
shutdown() on the attached socket. In a situation involving fork() and copies
of an object spanning multiple interpreter instances this will result in the
first instance to GC breaking all other instances sharing the socket.

dylanahsmith notes that we can avoid this by simply closing the socket before
entering mysql_close(), preventing the shutdown() from having any impact on the
shared socket state.
d5f812f
@eam eam added a commit to eam/mysql2 that referenced this issue Feb 10, 2014
@eam eam Avoid shutdown() on shared sockets (fixes #213)
Close the mysql socket before calling mysql_close(). mysql_close() calls
shutdown() on the attached socket. In a situation involving fork() and copies
of an object spanning multiple interpreter instances this will result in the
first instance to GC breaking all other instances sharing the socket.

dylanahsmith notes that we can avoid this by simply closing the socket before
entering mysql_close(), preventing the shutdown() from having any impact on the
shared socket state.
e492dff
@eam eam added a commit to eam/mysql2 that referenced this issue Feb 10, 2014
@eam eam Avoid shutdown() on shared sockets (fixes #213)
Close the mysql socket before calling mysql_close(). mysql_close() calls
shutdown() on the attached socket. In a situation involving fork() and copies
of an object spanning multiple interpreter instances this will result in the
first instance to GC breaking all other instances sharing the socket.

dylanahsmith notes that we can avoid this by simply closing the socket before
entering mysql_close(), preventing the shutdown() from having any impact on the
shared socket state.
d7d8e93
@sodabrew sodabrew pushed a commit that closed this issue Feb 27, 2014
@eam eam Avoid shutdown() on shared sockets (fixes #213)
Close the mysql socket before calling mysql_close(). mysql_close() calls
shutdown() on the attached socket. In a situation involving fork() and copies
of an object spanning multiple interpreter instances this will result in the
first instance to GC breaking all other instances sharing the socket.

dylanahsmith notes that we can avoid this by simply closing the socket before
entering mysql_close(), preventing the shutdown() from having any impact on the
shared socket state.
a2bcfd6
@sodabrew sodabrew closed this in a2bcfd6 Feb 27, 2014
@sodabrew sodabrew added a commit to sodabrew/mysql2 that referenced this issue Mar 21, 2014
@sodabrew sodabrew After closing the MySQL fd, set it to -1 to prevent a race condition …
…(updates #213).

The MySQL library also calls close() on the fd, so it is possible for another
thread to have a new use for the same fd value. By setting the fd to -1, the
second close() will return EBADF without the risk of closing the wrong file.
46335ce

I think I have this error again with ruby-audioinfo: moumar/ruby-audioinfo#2

Creating the AudioInfo object (and not even accessing it), make the database connection drop. Strange because as far as I can see AudioInfo does not use the database at all!

I have this error again with sneakers, it is a multi-process framework.

Collaborator

@anhuiliujun This particular ticket is closed. If you can provide a full bug report, enough to help reproduce and narrow down the problem, please open a new issue. You can reference this issue #id.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment