Net::HTTP::Persistent::Error: too many connection resets (due to Connection reset by peer - Errno::ECONNRESET) after 2 requests on 14759220

[bjoseph]

I am trying to do some simple screen scraping on etrade's website but am getting a similar issue that was reported by someone earlier. I read through that message thread but it doesn't look like it was ever resolved.

Here is the error I am getting back:

Net::HTTP::Persistent::Error: too many connection resets (due to Connection reset by peer - Errno::ECONNRESET) after 2 requests on 14759220
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/net-http-persistent-1.8/lib/net/http/persistent.rb:446:in `rescue in request'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/net-http-persistent-1.8/lib/net/http/persistent.rb:422:in `request'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize/http/agent.rb:204:in `fetch'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize.rb:628:in `post_form'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize.rb:520:in `submit'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/mechanize-2.0.1/lib/mechanize/form.rb:167:in `submit'
    from (irb):74
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/railties-3.0.9/lib/rails/commands/console.rb:44:in `start'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/railties-3.0.9/lib/rails/commands/console.rb:8:in `start'
    from /Users/benny/.rvm/gems/ruby-1.9.2-p180@taxhaven/gems/railties-3.0.9/lib/rails/commands.rb:23:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'

Here is my code:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
login_page = agent.get("https://www.etrade.com")
form = login_page.form_with(:action => '/login.fcc') 
form.USER     = "test"
form.PASSWORD = "test12"
form.submit

Any ideas?

Thanks

[chip]

I am getting the same error when attempting an http POST, but it only occurs intermittently.

[shaiguitar]

Same here.

[lankz]

I encountered this too after upgrading from 1.0.0, along with this one in a few other places:

too many connection resets (due to end of file reached - EOFError) after 3 requests on 70046648458580

Easily resolved by sticking with 1.0.0, which seems much more stable compared to 2.x at the moment.

[styx]

I'll try to find out when the regression came up.

[styx]

Net::Http::Persistent introduced in 4d074f4

Some stuff from debug log:

I, [2011-08-01T11:01:20.015280 #7916]  INFO -- : Net::HTTP::Post: /login.fcc
D, [2011-08-01T11:01:20.015352 #7916] DEBUG -- : request-header: accept => */*
D, [2011-08-01T11:01:20.015384 #7916] DEBUG -- : request-header: user-agent => WWW-Mechanize/1.0.0 (http://rubyforge.org/projects/mechanize/)
D, [2011-08-01T11:01:20.015415 #7916] DEBUG -- : request-header: connection => keep-alive
D, [2011-08-01T11:01:20.015449 #7916] DEBUG -- : request-header: keep-alive => 300
D, [2011-08-01T11:01:20.015479 #7916] DEBUG -- : request-header: accept-encoding => gzip,identity
D, [2011-08-01T11:01:20.015509 #7916] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2011-08-01T11:01:20.015540 #7916] DEBUG -- : request-header: host => us.etrade.com
D, [2011-08-01T11:01:20.015570 #7916] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2011-08-01T11:01:20.015600 #7916] DEBUG -- : request-header: cookie => WRC_ID=93.125.111.29-1312185679759; TB=8785
D, [2011-08-01T11:01:20.015630 #7916] DEBUG -- : request-header: referer => https://us.etrade.com/e/t/home
D, [2011-08-01T11:01:20.015665 #7916] DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
D, [2011-08-01T11:01:20.015695 #7916] DEBUG -- : request-header: content-length => 54
E, [2011-08-01T11:01:20.306272 #7916] ERROR -- : Rescuing EOF error
E, [2011-08-01T11:01:21.374473 #7916] ERROR -- : Rescuing EOF error
E, [2011-08-01T11:01:22.435259 #7916] ERROR -- : Rescuing EOF error

I also checked headers in Firefox plugin: Tamper Data
It shows that the size of 3 first consecutive requests(POST, GET, GET) is -1.

[bhochhi]

Any solution to this issue? I am getting similar error with no effect increasing keep_alive_time or open_time:
too many connection resets (due to end of file reached - EOFError) after 3 requests on 3084924
Net::HTTP::Persistent::Error

[simonmd]

Same here, seems to fluctuate between EOF and ECONNRESET

[jg]

Confirmed with Mechanize 2.0.1. Please fix this issue.

[GBH]

+1 I'm seeing the same thing

[knu]

Same here. It occurs regardless of https or http.

If you know re-posting is harmless (as in a login form) you can temporarily set:

  agent.agent.http.retry_change_request = true

To force a re-post, and seems it works for me.

[knu]

Another ugly workaround is to manually reset the connection before posting.

  #...
  agent.agent.http.tap { |http|
    http.reset http.connection_for(login_page.uri + form.action)
  }
  form.submit

[matthewbjones]

I'm also seeing this quite frequently on 2.0.1, going to look into downgrading to 1.0.0 as a short-term solution to the problem.

[jinschoi]

Here is what looks to be going on:

I see this error when I run a request, then after a brief pause, run another request, using SSL. It does not happen when I run the requests back to back. I got a full backtrace from line 460 of persistent.rb, and that showed that the IOError was being raised from buffering.rb:145 in read_nonblock at a call to sysread_nonblock, called from net/protocol.rb:135 in rbuf_fill.

Digging down, it looks like sysread_nonblock is implemented (in 1.9.2) in the core file ossl_ssl.c, and only returns an IOError if one of two errors occurred, SSL_ERROR_ZERO_RETURN or SSL_ERROR_SYSCALL. The documentation here: http://www.openssl.org/docs/ssl/SSL_get_error.html
indicates that either the connection has been closed, or an EOF was read that violates the SSL protocol. I'm guessing it is the first case because of the behavior with delays. So it looks like the problem is that regardless of the Connection: Keep-Alive setting, the SSL connection can go down and you get an EOFError. In the non-SSL situation, there is a similar problem with IO.read_nonblock() also having the possibility of returning EOFError.

The problem is going to be figuring out when that is the case so you can retry safely for non-idempotent queries. knu's workaround will work if you know it is safe to do so. You can't just always retry on EOFError because other levels can throw EOFError for different reasons.

[dgmdan]

I'm getting a similar error using 2.0.1. It happens on a POST request in which the server takes about 2-3 min to respond. The error I get is a little different though: "too many connection resets (due to Resource temporarily unavailable - Timeout::Error)" Fixed it temporarily by reverting back to 1.0.0 and setting high agent.read_timeout and agent.open_timeout settings.