Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

retry on request limits for persistent connections

nginx has a keepalive_requests directive that limits the
number of requests a client may make over a single keepalive
connection.  By default, this limit is 100 requests.

With the test script below, I get IOError on the 101st request
when hitting bogomips.org (my server).

$prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock': closed stream (IOError)
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
	from $prefix/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
	from $prefix/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
	from $prefix/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:298:in `pipeline_receive'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-pipeline-1.0/lib/net/http/pipeline.rb:179:in `pipeline'
	from $prefix/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.3.2/lib/net/http/persistent.rb:543:in `pipeline'
	from t.rb:21:in `block in <main>'
	from t.rb:19:in `times'
	from t.rb:19:in `<main>'

Ideally, Ruby could avoid raising IOError by not reading a
closed stream, but I don't have time to dig into net/http
internals so rescuing on IOError and retrying is easier.

Feel free to hit bogomips.org with the following test script
to confirm/test/debug this issue:

  # tested with ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  require 'net/http/pipeline'
  require 'net/http/persistent' # tested with 2.3.2
  require 'uri'

  nr = (ARGV.shift || 101).to_i
  np = (ARGV.shift || 1).to_i
  puts "requests per pipeline call: #{nr}"
  puts "pipeline calls: #{np}"

  # bogomips.org runs nginx with the default
  # keepalive_requests=100 directive
  # Apache servers have a similar MaxKeepaliveRequests directive
  # https://httpd.apache.org/docs/2.2/mod/core.html#maxkeepaliverequests
  uri = URI('http://bogomips.org/')
  http = Net::HTTP::Persistent.new('x')
  requests = nr.times.map { Net::HTTP::Get.new(uri.path) }

  # if nr is > keepalive_requests value on the server,
  # IOError gets raised
  np.times do |j|
    i = -1
    http.pipeline(uri, requests.dup) do |resp|
      p [ j, i += 1,  resp ]
    end
  end
  • Loading branch information...
commit 22005b9013920c8ae5c6fd39b92cfbf6beabe692 1 parent 5a79031
Eric Wong authored committed
Showing with 1 addition and 1 deletion.
  1. +1 −1  lib/net/http/pipeline.rb
2  lib/net/http/pipeline.rb
View
@@ -312,7 +312,7 @@ def pipeline_receive in_flight, responses
responses
rescue Timeout::Error, EOFError, Errno::ECONNABORTED, Errno::ECONNRESET,
- Errno::EPIPE, Net::HTTPBadResponse => e
+ Errno::EPIPE, Net::HTTPBadResponse, IOError => e
pipeline_finish
raise ResponseError.new(e, in_flight, responses)
Please sign in to comment.
Something went wrong with that request. Please try again.