Skip to content

Commit

Permalink
retry on request limits for persistent connections
Browse files Browse the repository at this point in the history
nginx has a keepalive_requests directive that limits the
number of requests a client may make over a single keepalive
connection.  By default, this limit is 100 requests.

With the test script below, I get IOError on the 101st request
when hitting bogomips.org (my server).

$prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock': closed stream (IOError)
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
	from $prefix/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
	from $prefix/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:298:in `pipeline_receive'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:179:in `pipeline'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.3.2/lib/net/http/persistent.rb:543:in `pipeline'
	from t.rb:21:in `block in <main>'
	from t.rb:19:in `times'
	from t.rb:19:in `<main>'

Ideally, Ruby could avoid raising IOError by not reading a
closed stream, but I don't have time to dig into net/http
internals so rescuing on IOError and retrying is easier.

Feel free to hit bogomips.org with the following test script
to confirm/test/debug this issue:

  # tested with ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  require 'net/http/pipeline'
  require 'net/http/persistent' # tested with 2.3.2
  require 'uri'

  nr = (ARGV.shift || 101).to_i
  np = (ARGV.shift || 1).to_i
  puts "requests per pipeline call: #{nr}"
  puts "pipeline calls: #{np}"

  # bogomips.org runs nginx with the default
  # keepalive_requests=100 directive
  # Apache servers have a similar MaxKeepaliveRequests directive
  # https://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests
  uri = URI('http://bogomips.org/')
  http = Net::HTTP::Persistent.new('x')
  requests = nr.times.map { Net::HTTP::Get.new(uri.path) }

  # if nr is > keepalive_requests value on the server,
  # IOError gets raised
  np.times do |j|
    i = -1
    http.pipeline(uri, requests.dup) do |resp|
      p [ j, i += 1,  resp ]
    end
  end
  • Loading branch information
Eric Wong authored and drbrain committed Apr 13, 2012
1 parent 5a79031 commit 22005b9
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion lib/net/http/pipeline.rb
Expand Up @@ -312,7 +312,7 @@ def pipeline_receive in_flight, responses


responses responses
rescue Timeout::Error, EOFError, Errno::ECONNABORTED, Errno::ECONNRESET, rescue Timeout::Error, EOFError, Errno::ECONNABORTED, Errno::ECONNRESET,
Errno::EPIPE, Net::HTTPBadResponse => e Errno::EPIPE, Net::HTTPBadResponse, IOError => e
pipeline_finish pipeline_finish


raise ResponseError.new(e, in_flight, responses) raise ResponseError.new(e, in_flight, responses)
Expand Down

0 comments on commit 22005b9

Please sign in to comment.