Mysql2::Error: This connection is still waiting for a result, try again once you have the result #772
Comments
|
After some investigation it appeared that the issue was caused by incorrect signal handling.
|
|
I ran into another, closely related variant of this issue. It can be triggered when someone uses #!/usr/bin/env ruby
require 'mysql2'
def long_query(connection)
begin
connection.query('SELECT SLEEP(1000);')
rescue StandardError => e
puts "Caught #{e}"
ensure
puts "Ensured"
end
puts 'done'
end
conn = Mysql2::Client.new(:host => "localhost",
:username => "root",
:password => 'p4ssw0rd')
t = Thread.new { long_query(conn) }
sleep 1
puts "Killing #{t}, #{t.status}"
t.kill
sleep 1
puts "Killed #{t}, #{t.status}"
conn.query('SELECT 1;')On my machine, the output is: ie. the killed thread is still marked as actively using the connection, even after it's been killed. This variant is a little bit worse than the ones caused by signals or remotely-triggered exceptions, because there is no reasonable way to mask the I imagine this has been discussed before, but I haven't found any reference to it. Would it be reasonable to try to solve this with an exception-handler in the C extension code? The exception handlers should get run in the |
|
@richardlarocque I cannot think of any reasonable workaround for a |
|
For those who are using Passenger to serve their app and use a script to kill passenger processes, check your script. We're running a script via cron to kill processes which are consuming too much ram. Thanks to the research by @dmitry-g, I looked into our script and was able to improve it to remove the use of kill. Passenger 5.0 comes with the passenger-config command which can be used to gracefully shut down passenger processes. We switched to that command recently and since we've started using it, this exception went away. Here is the script we're using to kill the processes for those who are interested (thanks to whoever came up with this script, it has been very helpful). for i in |
|
It looks like #811 would provide a fix for this problem. |
|
Thanks for the help, everyone! We no longer encounter this issue. 👍 |
|
this happened again, we are running the latest gem. here is the log can you help me debug it and see why this is still happening? @sodabrew @darnaut |
|
What are your initializers? |
|
Example:
|
|
Interesting re: Aurora failovers! Do you have metrics for Aurora failover events? I can imagine a relationship between these two scenarios: Rails app sends query |
|
The event that @eredi93 reported was isolated to a single host & wasn't triggered during any database failover, which is why we're at a bit of loss trying to figure out what the cause could be here. :( |
|
@austinhalpin Thanks for sharing your initializer, it helped me! I'm also curious if you've encountered any other gotchas in the last year since your post. :) I have been using Aurora a bit recently, and encountered the same issue with failovers. At first I thought that connections were not reset when the failover happened. After some more testing it does appear that all connections are reset, but that Rails often manages to reconnect before the DNS update has propagated fully. The Aurora cluster endpoint is a CNAME record with a 5 second TTL. So my conclusion is that the mysql process on the old master restarts too quickly and is able to receive connections again (but now as a slave) before the DNS has propagated fully. Clients that manage to connect are oblivious to what happened and tries to write to the slave. I changed your initializer slightly, using require 'active_record'
module ActiveRecord
module Mysql2ReadonlyReconnect
def execute(*)
super
rescue Mysql2::Error, ActiveRecord::StatementInvalid => e
if e.message.include?('The MySQL server is running with the --read-only option')
pool.disconnect!
end
raise
end
end
end
ActiveRecord::ConnectionAdapters::Mysql2Adapter.prepend(ActiveRecord::Mysql2ReadonlyReconnect)Finally, this document has been helpful to adopt best practices for Aurora: https://d1.awsstatic.com/whitepapers/RDS/amazon-aurora-mysql-database-administrator-handbook.pdf The document mentions "smart drivers" for Aurora. Basically the table There are some other Aurora-specific variables available: |
We've been experiencing this issue in production to varying degrees for the last two years:
Mysql2::Error: This connection is still waiting for a result, try again once you have the resultI've looked at previous issues (like #566, #532, #99), but didn't find anything that helps. We've also tried multiple things ourselves, but did not manage to solve it.
We're running:
Stacktraces don't show anything interesting, here's one example:
We didn't notice any particular pattern to when it happens either:
Any help would be appreciated 😄
The text was updated successfully, but these errors were encountered: