Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

retry on request limits for persistent connections

nginx has a keepalive_requests directive that limits the
number of requests a client may make over a single keepalive
connection.  By default, this limit is 100 requests.

With the test script below, I get IOError on the 101st request
when hitting bogomips.org (my server).

$prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock': closed stream (IOError)
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
	from $prefix/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
	from $prefix/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:298:in `pipeline_receive'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:179:in `pipeline'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.3.2/lib/net/http/persistent.rb:543:in `pipeline'
	from t.rb:21:in `block in <main>'
	from t.rb:19:in `times'
	from t.rb:19:in `<main>'

Ideally, Ruby could avoid raising IOError by not reading a
closed stream, but I don't have time to dig into net/http
internals so rescuing on IOError and retrying is easier.

Feel free to hit bogomips.org with the following test script
to confirm/test/debug this issue:

  # tested with ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  require 'net/http/pipeline'
  require 'net/http/persistent' # tested with 2.3.2
  require 'uri'

  nr = (ARGV.shift || 101).to_i
  np = (ARGV.shift || 1).to_i
  puts "requests per pipeline call: #{nr}"
  puts "pipeline calls: #{np}"

  # bogomips.org runs nginx with the default
  # keepalive_requests=100 directive
  # Apache servers have a similar MaxKeepaliveRequests directive
  # https://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests
  uri = URI('http://bogomips.org/')
  http = Net::HTTP::Persistent.new('x')
  requests = nr.times.map { Net::HTTP::Get.new(uri.path) }

  # if nr is > keepalive_requests value on the server,
  # IOError gets raised
  np.times do |j|
    i = -1
    http.pipeline(uri, requests.dup) do |resp|
      p [ j, i += 1,  resp ]
    end
  end
  • Loading branch information...
commit 22005b9013920c8ae5c6fd39b92cfbf6beabe692 1 parent 5a79031
Eric Wong authored committed

Showing 1 changed file with 1 addition and 1 deletion. Show diff stats Hide diff stats

  1. +1 1  lib/net/http/pipeline.rb
2  lib/net/http/pipeline.rb
@@ -312,7 +312,7 @@ def pipeline_receive in_flight, responses
312 312
313 313 responses
314 314 rescue Timeout::Error, EOFError, Errno::ECONNABORTED, Errno::ECONNRESET,
315   - Errno::EPIPE, Net::HTTPBadResponse => e
  315 + Errno::EPIPE, Net::HTTPBadResponse, IOError => e
316 316 pipeline_finish
317 317
318 318 raise ResponseError.new(e, in_flight, responses)

0 comments on commit 22005b9

Please sign in to comment.
Something went wrong with that request. Please try again.