New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AR adapter connection pool management problem #31
Comments
Would you mind trying again from master? |
It is still failing, but now it actually raises the error on the select_values call I described:
|
Moreover, the error is not persistent anymore. Previously, when the error occurred, I had to restart the server. Now, I can just refresh and it sometimes is successfully able to serve the request again, but not always. Weird. |
Maybe this is also relevant: I am using multiple databases from database.yml. The initial AR.find is done on the development database, and the custom queries are done on another connection. Both are using mysql2. |
If it can help Brian, I have the same issue, but I have only one database. |
I am trying to create a minimal sample app that recreates the problem, but so far, I have not been successful. |
It seems to be something external. |
For now, I have worked around the problem by using the MySQL2 API directly for the custom query I am using. This means I am using a lot more connections, but at least it is working reliably now. This does seem to indicate that the problem is in the AR adapter. |
I pushed 0.1.9 on the 17th, could you give 0.1.8 a try and see if you have the same issue? |
Pretty sure I tracked down the issue, could you give master another try? |
Seems to be working fine for me now. Thanks! It's amazing that you are able to track down an issue with a vague description like this :-) |
I actually decided to revert that commit, and will try and track down the issue in the AR adapter... hold tight! |
Cool ! |
+1 here...also started just in the last couple days, but we have had a lot of development happen so no one knows exactly what changed. We are using Bundler with Rails 2.3.8. A few devs are experiencing this consistently, including myself. Thanks for checking it out Brian |
Hey guys, |
Hi Brian, the problem with this ref. is that we might get some random errors while running the app. Thanks Brian for this gem |
Ok guys, |
I am also using a Rails 3 application so it wasn't Rails 2.3 specific. That said, this change seems to solve the problem for me. Thanks! |
Hi Brian, Hope it'll work ! Many thanks again ! |
Ah, this may have been a recent fix (post beta4) to AR in rails3. I'll double check on that and wait for one more of you to have a chance to test it before closing the ticket. Thanks for your help in tracking it down guys! |
Looks like it was fixed/changed (again?) in Rails3 in this commit: http://github.com/rails/rails/commit/62c4e4d3856b38ee9869f4ad6342e712788c8635 |
I'm still seeing this error on your master. Let me know if there is other info I can provide. It's highly intermittent, though, so hard to reproduce. |
What version of rails and ruby? |
Ruby Enterprise Edition 2010.02 and Rails 3 (master). I also wonder if this error has something to do with serving via Unicorn... |
Hi Brian, |
I'm having the same error using rails3.rc, latest unicorn and latest mysql2 gem. The error only occurs if you set "preload_app true". My unicorn is configured correctly with: before_fork do |server, worker| ActiveRecord::Base.connection.disconnect! end after_fork do |server, worker| ActiveRecord::Base.establish_connection end I also tried the latest snapshot from github for mysql2, but the error still occurs. The only workaround for now it to set "preload_app false". |
After some debugging it seems the error is inside the mysql2_ext.h file. The macro REQUIRE_OPEN_DB check some internal field of the MYSQL structure and assumes the connection is closed if the net.vio field is NULL. But this seems to be a wrong assumption. Even thought net.vio is NULL sometimes, the connection is still alive and working perfectly. I'd suggest simply removing this macro completely (it works fine for me then). If the connection is closed the mysql functions won't crash, they'll return an error anyway. And I think it's not a good practice to work with any internal fields (at least I think it's an internal field because I couldn't find any documentation on it). If you want to give it a try, please try my fork. Please let me know how it works :-) http://github.com/gucki/mysql2/commit/15932686254749f2a66962a1565b0637a61d1364 |
I actually tried this recently and was still getting errors. They were just coming from libmysql instead of the ruby extension itself. They were the typical "MySQL server has gone away..." error I'm sure we've all seen many times. I suspect the issue is something to do with the AR driver telling the connection poll the wrong thing... Still trying to track it down. |
I'm able to reproduce this using Unicorn, although I can't with thin/webrick/mongrel. I can also reproduce it with "preload_app false" - something is definitely up with the combination of AR's connection pool and mysql2's AR adapter. |
I'm running my app with the patched mysql2 for some time now and didn't have any problems so far. Without the patch I got the errors just after a few seconds. |
stumbled upon same bug here, rails 3 rc on ruby 1.9.2. |
For now you can point to the Rails repos 3-0-stable branch and also mysql2's master branch in your Gemfile. The next RC of Rails is due any day now and I'll be pushing 0.2.0 very soon as well. |
confirming that using the edge versions works.
|
Awesome! Thanks for testing that out |
Seems to work fine! |
NICE thanks for testing man |
0.2.0 pushed |
still have been running onto this issue with 0.2.0 and 0.2.1 now giving 0.2.3 a try. as for now it's been running smoothly for a few hours now. |
still the same with mysql2 in 0.2.3 version :( |
Could you give me a snippet to reproduce? Also, what other details: ruby version and architecture, operating system, rails/sinatra, etc |
No snippet to be shown -- it's just that after some hours application throws a 500 with "Mysql2::Error: closed MySQL connection" in production.log System is Gentoo Linux Maybe I could provide you with some system/mysql logs? Which ones? |
Could you give Rails3 RC2 a try? Which I'm pretty sure was fixed in RC2. |
OK, I gave Rails3 final a try with mysql2 latest version. Let's see how it goes :) |
I'm seeing similar issues sparodically in backend scripts with latest master version, Merb 1.0.1, activerecord 2.3.8. These are for constantly used connections, and no threading. Example tracebacks with different messages : ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/connection_adapters/abstract_adapter.rb:221:in `log': Errno::EBADF: Bad file descriptor: SELECT * FROM `static_locations` WHERE latitude = 24.43359375 and longitude = 54.31640625 ORDER BY weighting desc LIMIT 1 (ActiveRecord::StatementInvalid) from ./vendor/gems/gems/mysql2-0.2.3/lib/active_record/connection_adapters/mysql2_adapter.rb:314:in `execute' from ./vendor/gems/gems/mysql2-0.2.3/lib/active_record/connection_adapters/mysql2_adapter.rb:627:in `select' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/connection_adapters/abstract/query_cache.rb:62:in `select_all' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:664:in `find_by_sql' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:1578:in `find_every' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:1535:in `find_initial' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:616:in `find' ... ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/connection_adapters/abstract_adapter.rb:221:in `log': Mysql2::Error: This connection is still waiting for a result, try again once you have the result: SELECT * FROM `static_locations` WHERE latitude = 29.8828125 and longitude = 29.970703125 (ActiveRecord::StatementInvalid) from ./vendor/gems/gems/mysql2-0.2.3/lib/active_record/connection_adapters/mysql2_adapter.rb:314:in `execute' from ./vendor/gems/gems/mysql2-0.2.3/lib/active_record/connection_adapters/mysql2_adapter.rb:627:in `select' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all_without_query_cache' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/connection_adapters/abstract/query_cache.rb:62:in `select_all' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:664:in `find_by_sql' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:1578:in `find_every' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:1535:in `find_initial' from ./vendor/gems/gems/activerecord-2.3.8/lib/active_record/base.rb:616:in `find' ... |
afaik Merb uses EventMachine which could be attempting concurrency in your app. Which mysql driver were you using before? |
I was using the old 'mysql' driver previously, v 2.8.1 with no problem. I'm not sure what concurrency might be being attempted in these. Originally we did have some concurrency issues with the backend scripts, & in fact was solved by loading the environment explicitly, and not using the concurrent setup you get running offline scripts normally with Merb. Some other notes. There haven't been any of these errors reported from the webserver which is running through nginx+passenger. The problem happens randomly (at most every hours hour for an individual script), and occurs much more frequently during peak usage periods for the database, and the machines the scripts are running. Thanks for your help! |
Forgot to ask which version of ruby? |
Sorry, I should have provided in the first place! Its using Ruby Enterprise Edition : ruby 1.8.7 (2009-12-24 patchlevel 248) [x86_64-linux], MBARI 0x6770, Ruby Enterprise Edition 2010.01 |
A recent patch fixed a long standing bug that was probably the culprit of all of the strange hard to reproduce (for me) bugs people have been seeing. If any of you are still having issues would you mind giving master a try? |
FWIW, I am still not seeing these issues return with the latest HEAD. |
bad. ass. |
Hi Mario, I've lot of Mysql2::Error: closed MySQL connection errors with latest gem (0.2.4) on a production env with rails 3 and ruby 1.9.2 reconnect: true |
forgot mysql is a AWS RDS instance (so 5.1 version) |
Same error here: 0.2.4 upd: no such problem on staging server (same config) |
Also have this error with 0.2.4. Downgraded to 0.2.3, error disappeared. I'm using REE and Rails 3. |
I have the same error with 0.2.4 when using multiple databases. Used Ruby 1.9.2 via RVM and Rails 3. Will downgrade to 0.2.3. |
Could you guys give master a try? |
Tried out master, and I cannot reproduce the error in rails console anymore. Until now it looks good to me. BTW, thanks for this awesome gem! |
Tried out master, and it seems to work. Thanks! |
I have the same issue with Mysql2::Error closed MySQL connection on a custom Ruby-only app with EM with latest Ruby 1.9.1. I will give master a try as well. |
I am having troubles with the connection management of the ActiveRecord adapter. For some reason, I am getting "Mysql2::Error: closed MySQL connection" after a couple of refreshes of the same page.
The page uses one
ActiveRecord.find(id)
call, and several (about 10) calls to theActiveRecord::Base.connection.select_values(sql)
method. It usually works fine the first time and after the first and second refresh of the page, but it'll fail eventually.The error occurs outside of my own code. This is the backtrace:
When I comment out the calls to the custom
select_values
call, the problem does not seem to occur anymore. This is the code I am using:These errors only occur in development mode; in production everything seems to be working OK. Also, when I rewrite the method above to use
execute
and handle the result myself, the problem persists. Moreover, the exact same code works fine with the normal mysql gem, except for some typecasting issues (which is why I want to use mysql2).The text was updated successfully, but these errors were encountered: