Lost Connections Crashing. Better dberrhandle Implementation #79

metaskills · 2012-04-05T03:38:07Z

Based on this James K. Lowden's excellent stack trace review.

http://lists.ibiblio.org/pipermail/freetds/2012q2/027793.html

metaskills · 2012-04-19T17:06:51Z

It is important to read the full thread from above, the link is only the last part. Will need to QA out client.c code.

Also, this is another link on SQL Server and it's keep alive setting. Which could also play into poor connections. But the main emphasis should be on a better dberrhandle implementation.

http://blogs.msdn.com/b/sql_protocols/archive/2006/03/09/546852.aspx

metaskills · 2012-09-25T11:40:11Z

http://lists.ibiblio.org/pipermail/freetds/2012q3/028026.html

metaskills · 2012-11-26T12:42:14Z

This thread has some interesting keep-alive talk.
http://lists.ibiblio.org/pipermail/freetds/2012q4/028157.html

neongrau · 2013-02-01T11:08:05Z

Is there any way to handle these exceptions within my Rails application?
Having the problem that my thin instances are crashing hard when some Backup Tool is stopping the SQL Server at night.

metaskills · 2013-02-01T13:03:21Z

Technically yes, read up here in the adapter. https://github.com/rails-sqlserver/activerecord-sqlserver-adapter#auto-connecting

neongrau · 2013-02-01T13:09:23Z

ActiveRecord::ConnectionAdapters::SQLServerAdapter.auto_connect returns nil
So all i need is to set it to true ?

metaskills · 2013-02-01T13:30:50Z

No, you have to define your own methods to do what you want. These are callbacks so you can get more info and debug. As the docs say, the currently log.

https://github.com/rails-sqlserver/activerecord-sqlserver-adapter/blob/master/lib/active_record/connection_adapters/sqlserver_adapter.rb#L44-L50

So if you want to add more to that or change it, do something like this in a confi/initializers/ file.

module ActiveRecord
  module ConnectionAdapters
    class SQLServerAdapter < AbstractAdapter

      protected

      def self.did_retry_sqlserver_connection(connection,count)
        # ...
      end

      def self.did_lose_sqlserver_connection(connection)
        # ...
      end

    end
  end
end

neongrau · 2013-02-01T13:38:28Z

hmm, if
def self.did_retry_sqlserver_connection(connection,count)
logger.info "CONNECTION RETRY: #{connection.class.name} retry ##{count}."
end

def self.did_lose_sqlserver_connection(connection)
  logger.info "CONNECTION LOST: #{connection.class.name}"
end

is the default, shouldn't i see at least one of these messages in the log?
Because even when i turn on debug logging, i never get any of these messages. All i get is a windows popup telling me that ruby.exe has crashed. When i press "OK" then the process just ends with nothing else happening on the rails part.

And this is most i get when i run from command-line. When i run this via wrapper (with rubyw.exe) as a service that crash messages doesn't even show anywhere and the process still stays up as a zombie so noone can even tell that the service died.

scaryguy · 2013-05-02T10:25:54Z

Is there a solution for this or not? I really have difficulty to understand. Setting

ActiveRecord::ConnectionAdapters::SQLServerAdapter.auto_connect = true

does not work for me? At a post above you're saying that wouldn't be enough. It's not me who created this adapter who come I can fix this by my self? :|

HOW to prevent my app to die because of that kind of dead connections?? Doesn't this question have an answer? With a plain english?

krzcho · 2013-07-05T13:21:56Z

Both versions 0.6.0.rc1 x86-mingw32 and 0.5.1 x86-mingw32 crash after connection is lost; no reconnection attempt, no single entry in the log
Ken: was initial dump/log investigation implemented somehow as I still see the same results: infinite loop of
dblib.c:5780:dbgetuserdata(0591C858)
dblib.c:5780:dbgetuserdata(0591C858)
dblib.c:4880:dbdead(0591C858) [alive]
dblib.c:4639:dbsqlok(0591C858)
dblib.c:4669:dbsqlok() not done, calling tds_process_tokens()
token.c:540:tds_process_tokens(05797EA8, 000939A0, 0009399C, 0x6914)
util.c:331:tdserror(05933A90, 05797EA8, 20004, 10054)
dblib.c:7929:dbperror(0591C858, 20004, 10054)
dblib.c:7981:20004: "Read from the server failed"
ending with a crash? freetds now and before is reported as 0.91 - nothing changed actually?
As I read consecutive time James K. Lowden's analysis I think it is a client problem:
"It would appear -- whether by design or not -- that the
client-installed error handler makes db-lib calls, in particular
dbsqlok(). Don't do that. It engenders infinite recursion".
My guess is that you receive SYBEWRIT (atually a SYBEREAD which is 20004 above) error before dbdead is able to return true and you are somehow trying to cancel query on a failing socket. Please move SYBEWRIT/SYBEREAD error to the same handling routine as SYBESOCK (ending connection without cancelling query) and let me test it.

metaskills · 2013-07-05T18:37:55Z

Someone needs to provide a suggested patch to save me some time till I get a chance to look at this.

Also, I have this test that is in skip mode which would help with debugging.

krzcho · 2013-07-05T20:32:09Z

I have built tiny_tds with my patch (BTW great job with the mini_portile!) and I can confirm it solves the issue: after dropping the connection (using tcpview utility from sysinternals) it is now nicely reconnecting (CONNECTION RETRY: ActiveRecord::ConnectionAdapters::SQLServerAdapter retry #0. in the log file) instead of blowing up the the process.
Patch changes handling routine only for SYBEREAD - maybe it is enough as you have some constraints for canceling a query for SYBEWRIT. Time will tell.
Please consider a fast merge/release as this issue of crashing web server could be a huge annoyance for windows users with little load applications (connection drop due to inactivity, like mine) or unstable network infrastructure.

metaskills · 2013-07-07T01:55:15Z

Working on pushing a v0.6.0 release soon.

Ken

On Jul 5, 2013, at 4:32 PM, Krzysztof Chodak notifications@github.com wrote:

I have built tiny_tds with my patch (BTW great job with the mini_portile!) and I can confirm it solves the issue: after dropping the connection (using tcpview utility from sysinternals) it is now nicely reconnecting (CONNECTION RETRY: ActiveRecord::ConnectionAdapters::SQLServerAdapter retry #0. in the log file) instead of blowing up the the process.
Patch changes handling routine only for SYBEREAD - maybe it is enough as you have some constraints for canceling a query for SYBEWRIT. Time will tell.
Please consider a fast merge as this issue of crashing web server could be a huge annoyance for windows users with little load applications (connection drop due to inactivity, like mine) or unstable network infrastructure.

—
Reply to this email directly or view it on GitHub.

metaskills · 2013-07-07T03:11:33Z

Closing this since issue #124 is the fix.

jh125486 · 2013-11-16T06:01:48Z

I don't know if this a problem in TinyTds or FreeTDS or just a crappy SqlServer that is on it's last legs:
I was using TinyTds 0.5.2 as of last week, and started getting 'Something has gone wrong" when querying my sqlserver. A second or third attempt would query fine though.
I'm running the latest FreeTDS 0.91, and reading through this issue I decided to pull down the master TinyTds and upgrade to it.
It still fails at the same rate, but now I am getting: 'INT_EXIT returned for msgno 20004' .
A log of FreeTDS shows the same "db-lib is exiting because the client error handle returned INT_EXIT returned for msgno 20004".
I am unfortunately not a "windows" guy, but our windows sysads say nothing is wrong the sql server.

So I have two questions:

Is the sqlserver sick?
Where can I intercept these connection drops and have it manually reconnect? (I'm using Rails 3).

Thanks, this Gem has been a lifesaver btw!

krzcho · 2013-11-16T07:22:03Z

Please check awl log file first
16 lis 2013 07:01 "Jacob Hochstetler" notifications@github.com napisał(a):

I don't know if this a problem in TinyTds or FreeTDS or just a crappy
SqlServer that is on it's last legs:
I was using TinyTds 0.5.2 as of last week, and started getting 'Something
has gone wrong" when querying my sqlserver. A second or third attempt would
query fine though.
I'm running the latest FreeTDS 0.91, and reading through this issue I
decided to pull down the master TinyTds and upgrade to it.
It still fails at the same rate, but now I am getting: 'INT_EXIT returned
for msgno 20004' .
A log of FreeTDS shows the same "db-lib is exiting because the client
error handle returned INT_EXIT returned for msgno 20004".
I am unfortunately not a "windows" guy, but our windows sysads say nothing
is wrong the sql server.

So I have two questions:

Is the sqlserver sick?

Where can I intercept these connection drops and have it manually
reconnect? (I'm using Rails 3).

Thanks, this Gem has been a lifesaver btw!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-28620559
.

krzcho · 2013-11-16T07:22:41Z

awl=sql
16 lis 2013 08:21 "Krzysztof Chodak" krzysztof.chodak@gmail.com
napisał(a):

Please check awl log file first
16 lis 2013 07:01 "Jacob Hochstetler" notifications@github.com
napisał(a):

I don't know if this a problem in TinyTds or FreeTDS or just a crappy
SqlServer that is on it's last legs:
I was using TinyTds 0.5.2 as of last week, and started getting 'Something
has gone wrong" when querying my sqlserver. A second or third attempt would
query fine though.
I'm running the latest FreeTDS 0.91, and reading through this issue I
decided to pull down the master TinyTds and upgrade to it.
It still fails at the same rate, but now I am getting: 'INT_EXIT returned
for msgno 20004' .
A log of FreeTDS shows the same "db-lib is exiting because the client
error handle returned INT_EXIT returned for msgno 20004".
I am unfortunately not a "windows" guy, but our windows sysads say
nothing is wrong the sql server.

So I have two questions:

Is the sqlserver sick?

Where can I intercept these connection drops and have it manually
reconnect? (I'm using Rails 3).

Thanks, this Gem has been a lifesaver btw!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-28620559
.

jh125486 · 2013-11-16T07:31:02Z

I'm tracking. I'll get with our support team to pour through the logs.
Thanks again.

krzcho · 2013-11-16T07:42:19Z

Yes it is
16 lis 2013 08:31 "Jacob Hochstetler" notifications@github.com napisał(a):

Is this log located on the SqlServer?
Thanks!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/79#issuecomment-28621547
.

metaskills · 2013-11-16T13:32:34Z

Why don't you build a gem from master and try a newer version. We have fixes around this area.

metaskills mentioned this issue Apr 19, 2012

WSAENOBUFS failure and subsequent crash #70

Closed

metaskills mentioned this issue Apr 19, 2012

tinytds incredibly slow over VPN #82

Closed

metaskills mentioned this issue Sep 19, 2012

Freezing on parsing of execute statements. #93

Closed

metaskills mentioned this issue May 1, 2013

ActiveRecord::LostConnection (TinyTds::Error: DBPROCESS is dead or not enabled rails-sqlserver/activerecord-sqlserver-adapter#267

Closed

metaskills closed this as completed Jul 7, 2013

BMahabir mentioned this issue Jan 26, 2015

The same issue as issue #79. (DBPROCESS is dead). #185

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lost Connections Crashing. Better dberrhandle Implementation #79

Lost Connections Crashing. Better dberrhandle Implementation #79

metaskills commented Apr 5, 2012

metaskills commented Apr 19, 2012

metaskills commented Sep 25, 2012

metaskills commented Nov 26, 2012

neongrau commented Feb 1, 2013

metaskills commented Feb 1, 2013

neongrau commented Feb 1, 2013

metaskills commented Feb 1, 2013

neongrau commented Feb 1, 2013

scaryguy commented May 2, 2013

krzcho commented Jul 5, 2013

metaskills commented Jul 5, 2013

krzcho commented Jul 5, 2013

metaskills commented Jul 7, 2013

metaskills commented Jul 7, 2013

jh125486 commented Nov 16, 2013

krzcho commented Nov 16, 2013

krzcho commented Nov 16, 2013

jh125486 commented Nov 16, 2013

krzcho commented Nov 16, 2013

metaskills commented Nov 16, 2013

Lost Connections Crashing. Better dberrhandle Implementation #79

Lost Connections Crashing. Better dberrhandle Implementation #79

Comments

metaskills commented Apr 5, 2012

metaskills commented Apr 19, 2012

metaskills commented Sep 25, 2012

metaskills commented Nov 26, 2012

neongrau commented Feb 1, 2013

metaskills commented Feb 1, 2013

neongrau commented Feb 1, 2013

metaskills commented Feb 1, 2013

neongrau commented Feb 1, 2013

scaryguy commented May 2, 2013

krzcho commented Jul 5, 2013

metaskills commented Jul 5, 2013

krzcho commented Jul 5, 2013

metaskills commented Jul 7, 2013

metaskills commented Jul 7, 2013

jh125486 commented Nov 16, 2013

krzcho commented Nov 16, 2013

krzcho commented Nov 16, 2013

jh125486 commented Nov 16, 2013

krzcho commented Nov 16, 2013

metaskills commented Nov 16, 2013