Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Initial commit

  • Loading branch information...
commit 0f942cafbf3d29e337145cf36b9cdd85d0da0860 0 parents
Brad Pardee authored October 03, 2010
4  .gitignore
... ...
@@ -0,0 +1,4 @@
  1
+/TAGS
  2
+/pkg
  3
+/doc
  4
+*.swp
4  History.txt
... ...
@@ -0,0 +1,4 @@
  1
+=== 1.0.0 / 2010-09-16
  2
+
  3
+* Major Enhancements
  4
+  * Based on drbrain/net-http-persistent but uses a connection pool instead of connection/thread
20  LICENSE
... ...
@@ -0,0 +1,20 @@
  1
+Copyright (c) 2010 Eric Hodel, Aaron Patterson
  2
+
  3
+Permission is hereby granted, free of charge, to any person obtaining
  4
+a copy of this software and associated documentation files (the
  5
+'Software'), to deal in the Software without restriction, including
  6
+without limitation the rights to use, copy, modify, merge, publish,
  7
+distribute, sublicense, and/or sell copies of the Software, and to
  8
+permit persons to whom the Software is furnished to do so, subject to
  9
+the following conditions:
  10
+
  11
+The above copyright notice and this permission notice shall be
  12
+included in all copies or substantial portions of the Software.
  13
+
  14
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
  15
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  16
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
  17
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
  18
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
  19
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
  20
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
58  README.rdoc
Source Rendered
... ...
@@ -0,0 +1,58 @@
  1
+= persistent_http
  2
+
  3
+* http://github.com/bpardee/persistent_http
  4
+
  5
+== DESCRIPTION:
  6
+
  7
+Persistent connections using Net::HTTP with a connection pool.
  8
+
  9
+This is based on Eric Holder's Net::HTTP::Persistent libary but uses
  10
+a connection pool of Net::HTTP objects instead of a connection per 
  11
+thread.  C/T is fine if you're only using your http threads to make 
  12
+connections but if you use them in child threads then I suspect you 
  13
+will have a thread memory leak.  Also, you will generally get less
  14
+connection resets if the most recently used connection is always 
  15
+returned.
  16
+
  17
+== FEATURES/PROBLEMS:
  18
+
  19
+* Supports SSL
  20
+* Thread-safe
  21
+* Pure ruby
  22
+* Timeout-less speed boost for 1.8 (by Aaron Patterson)
  23
+
  24
+== INSTALL:
  25
+
  26
+  gem install persistent_http
  27
+
  28
+== EXAMPLE USAGE:
  29
+
  30
+  require 'persistent_http'
  31
+
  32
+  class MyHTTPClient
  33
+    @@persistent_http = PersistentHTTP.new(
  34
+      :name         => 'MyHTTPClient',
  35
+      :logger       => Rails.logger,
  36
+      :pool_size    => 10,
  37
+      :warn_timeout => 0.25,
  38
+      :force_retry  => true,
  39
+      :url          => 'https://www.example.com/echo/foo'  # equivalent to :use_ssl => true, :host => 'www.example.com', :default_path => '/echo/foo'
  40
+    )
  41
+
  42
+    def send_get_message
  43
+      response = @@persistent_http.request
  44
+      ... Handle response as you would a normal Net::HTTPResponse ...
  45
+    end
  46
+
  47
+    def send_post_message
  48
+      request = Net::HTTP::Post.new('/perform_service)
  49
+      ... Modify request as needed ...
  50
+      response = @@persistent_http.request(request)
  51
+      ... Handle response as you would a normal Net::HTTPResponse ...
  52
+    end
  53
+  end
  54
+
  55
+
  56
+== Copyright
  57
+
  58
+Copyright (c) 2010 Eric Hodel, Aaron Patterson, Brad Pardee.  See LICENSE for details.
16  Rakefile
... ...
@@ -0,0 +1,16 @@
  1
+require 'rubygems'
  2
+require 'rake'
  3
+
  4
+begin
  5
+  require 'jeweler'
  6
+  Jeweler::Tasks.new do |gemspec|
  7
+    gemspec.name = "persistent_http"
  8
+    gemspec.summary = "Persistent HTTP connections using a connection pool"
  9
+    gemspec.description = "Persistent HTTP connections using a connection pool"
  10
+    gemspec.email = "bradpardee@gmail.com"
  11
+    gemspec.homepage = "http://github.com/bpardee/persistent_http"
  12
+    gemspec.authors = ["Brad Pardee"]
  13
+  end
  14
+rescue LoadError
  15
+  puts "Jeweler not available. Install it with: gem install jeweler"
  16
+end
1  VERSION
... ...
@@ -0,0 +1 @@
  1
+1.0.0
444  lib/persistent_http.rb
... ...
@@ -0,0 +1,444 @@
  1
+require 'net/http'
  2
+require 'persistent_http/faster'
  3
+require 'uri'
  4
+require 'gene_pool'
  5
+
  6
+##
  7
+# Persistent connections for Net::HTTP
  8
+#
  9
+# PersistentHTTP maintains a connection pool of Net::HTTP persistent connections.
  10
+# When connections fail due to resets or bad responses, the connection is renewed
  11
+# and the request is retried per RFC 2616 (POST requests will only get retried if
  12
+# the :force_retry option is set to true).
  13
+#
  14
+# Example:
  15
+#
  16
+#   @@persistent_http = PersistentHTTP.new(
  17
+#     :name         => 'MyHTTPClient',
  18
+#     :logger       => Rails.logger,
  19
+#     :pool_size    => 10,
  20
+#     :warn_timeout => 0.25,
  21
+#     :force_retry  => true,
  22
+#     :url          => 'https://www.example.com/echo/foo'  # equivalent to :use_ssl => true, :host => 'www.example.com', :default_path => '/echo/foo'
  23
+#   )
  24
+#  
  25
+#   def send_get_message
  26
+#     response = @@persistent_http.request
  27
+#     ... Handle response as you would a normal Net::HTTPResponse ...
  28
+#   end
  29
+#  
  30
+#   def send_post_message
  31
+#     request = Net::HTTP::Post.new('/perform_service)
  32
+#     ... Modify request as needed ...
  33
+#     response = @@persistent_http.request(request)
  34
+#     ... Handle response as you would a normal Net::HTTPResponse ...
  35
+#   end
  36
+
  37
+class PersistentHTTP
  38
+
  39
+  ##
  40
+  # The version of PersistentHTTP use are using
  41
+  VERSION = '1.0.0'
  42
+
  43
+  ##
  44
+  # Error class for errors raised by PersistentHTTP.  Various
  45
+  # SystemCallErrors are re-raised with a human-readable message under this
  46
+  # class.
  47
+  class Error < StandardError; end
  48
+
  49
+  ##
  50
+  # An SSL certificate authority.  Setting this will set verify_mode to
  51
+  # VERIFY_PEER.
  52
+  attr_accessor :ca_file
  53
+
  54
+  ##
  55
+  # This client's OpenSSL::X509::Certificate
  56
+  attr_accessor :certificate
  57
+
  58
+  ##
  59
+  # Sends debug_output to this IO via Net::HTTP#set_debug_output.
  60
+  #
  61
+  # Never use this method in production code, it causes a serious security
  62
+  # hole.
  63
+  attr_accessor :debug_output
  64
+
  65
+  ##
  66
+  # Default path for the request
  67
+  attr_accessor :default_path
  68
+  
  69
+  ##
  70
+  # Retry even for non-idempotent (POST) requests.
  71
+  attr_accessor :force_retry
  72
+
  73
+  ##
  74
+  # Headers that are added to every request
  75
+  attr_accessor :headers
  76
+
  77
+  ##
  78
+  # Host for the Net:HTTP connection
  79
+  attr_reader :host
  80
+  
  81
+  ##
  82
+  # HTTP version to enable version specific features.
  83
+  attr_reader :http_version
  84
+
  85
+  ##
  86
+  # The value sent in the Keep-Alive header.  Defaults to 30.  Not needed for
  87
+  # HTTP/1.1 servers.
  88
+  #
  89
+  # This may not work correctly for HTTP/1.0 servers
  90
+  #
  91
+  # This method may be removed in a future version as RFC 2616 does not
  92
+  # require this header.
  93
+  attr_accessor :keep_alive
  94
+
  95
+  ##
  96
+  # Logger for message logging.
  97
+  attr_accessor :logger
  98
+
  99
+  ##
  100
+  # A name for this connection.  Allows you to keep your connections apart
  101
+  # from everybody else's.
  102
+  attr_reader :name
  103
+
  104
+  ##
  105
+  # Seconds to wait until a connection is opened.  See Net::HTTP#open_timeout
  106
+  attr_accessor :open_timeout
  107
+
  108
+  ##
  109
+  # The maximum size of the connection pool
  110
+  attr_reader :pool_size
  111
+
  112
+  ##
  113
+  # Port for the Net:HTTP connection
  114
+  attr_reader :port
  115
+  
  116
+  ##
  117
+  # This client's SSL private key
  118
+  attr_accessor :private_key
  119
+
  120
+  ##
  121
+  # The URL through which requests will be proxied
  122
+  attr_reader :proxy_uri
  123
+
  124
+  ##
  125
+  # Seconds to wait until reading one block.  See Net::HTTP#read_timeout
  126
+  attr_accessor :read_timeout
  127
+
  128
+  ##
  129
+  # Use ssl if set
  130
+  attr_reader :use_ssl
  131
+  
  132
+  ##
  133
+  # SSL verification callback.  Used when ca_file is set.
  134
+  attr_accessor :verify_callback
  135
+
  136
+  ##
  137
+  # HTTPS verify mode.  Defaults to OpenSSL::SSL::VERIFY_NONE which ignores
  138
+  # certificate problems.
  139
+  #
  140
+  # You can use +verify_mode+ to override any default values.
  141
+  attr_accessor :verify_mode
  142
+
  143
+  ##
  144
+  # The threshold in seconds for checking out a connection at which a warning 
  145
+  # will be logged via the logger
  146
+  attr_accessor :warn_timeout
  147
+
  148
+  ##
  149
+  # Creates a new PersistentHTTP.
  150
+  #
  151
+  # Set +name+ to keep your connections apart from everybody else's.  Not
  152
+  # required currently, but highly recommended.  Your library name should be
  153
+  # good enough.  This parameter will be required in a future version.
  154
+  #
  155
+  # +proxy+ may be set to a URI::HTTP or :ENV to pick up proxy options from
  156
+  # the environment.  See proxy_from_env for details.
  157
+  #
  158
+  # In order to use a URI for the proxy you'll need to do some extra work
  159
+  # beyond URI.parse:
  160
+  #
  161
+  #   proxy = URI.parse 'http://proxy.example'
  162
+  #   proxy.user     = 'AzureDiamond'
  163
+  #   proxy.password = 'hunter2'
  164
+
  165
+  def initialize(options={})
  166
+    @name            = options[:name]            || 'PersistentHTTP'
  167
+    @ca_file         = options[:ca_file]
  168
+    @certificate     = options[:certificate]
  169
+    @debug_output    = options[:debug_output]
  170
+    @default_path    = options[:default_path]
  171
+    @force_retry     = options[:force_retry]
  172
+    @headers         = options[:header]          || {}
  173
+    @host            = options[:host]
  174
+    @keep_alive      = options[:keep_alive]      || 30
  175
+    @logger          = options[:logger]
  176
+    @open_timeout    = options[:open_timeout]
  177
+    @pool_size       = options[:pool_size]       || 1
  178
+    @port            = options[:port]
  179
+    @private_key     = options[:private_key]
  180
+    @read_timeout    = options[:read_timeout]
  181
+    @use_ssl         = options[:use_ssl]
  182
+    @verify_callback = options[:verify_callback]
  183
+    @verify_mode     = options[:verify_mode]
  184
+    @warn_timeout    = options[:warn_timeout]    || 0.5
  185
+    
  186
+    url              = options[:url]
  187
+    if url
  188
+      url = URI.parse(url) if url.kind_of? String
  189
+      @default_path ||= url.request_uri
  190
+      @host         ||= url.host
  191
+      @port         ||= url.port
  192
+      @use_ssl      ||= url.scheme == 'https'          
  193
+    end
  194
+    
  195
+    @port ||= (@use_ssl ? 443 : 80)
  196
+
  197
+    # Hash containing the request counts based on the connection
  198
+    @count_hash = Hash.new(0)
  199
+
  200
+    raise 'host not set' unless @host
  201
+    net_http_args = [@host, @port]
  202
+    connection_id = net_http_args.join ':'
  203
+
  204
+    proxy = options[:proxy]
  205
+
  206
+    @proxy_uri = case proxy
  207
+                 when :ENV      then proxy_from_env
  208
+                 when URI::HTTP then proxy
  209
+                 when nil       then # ignore
  210
+                 else raise ArgumentError, 'proxy must be :ENV or a URI::HTTP'
  211
+                 end
  212
+
  213
+    if @proxy_uri then
  214
+      @proxy_args = [
  215
+        @proxy_uri.host,
  216
+        @proxy_uri.port,
  217
+        @proxy_uri.user,
  218
+        @proxy_uri.password,
  219
+      ]
  220
+
  221
+      @proxy_connection_id = [nil, *@proxy_args].join ':'
  222
+
  223
+      connection_id << @proxy_connection_id
  224
+      net_http_args.concat @proxy_args
  225
+    end
  226
+
  227
+    @pool = GenePool.new(:name         => name + '-' + connection_id,
  228
+                         :pool_size    => @pool_size,
  229
+                         :warn_timeout => @warn_timeout,
  230
+                         :logger       => @logger) do
  231
+      begin
  232
+        connection = Net::HTTP.new(*net_http_args)
  233
+        connection.set_debug_output @debug_output if @debug_output
  234
+        connection.open_timeout = @open_timeout if @open_timeout
  235
+        connection.read_timeout = @read_timeout if @read_timeout
  236
+
  237
+        ssl connection if @use_ssl
  238
+
  239
+        connection.start
  240
+        connection
  241
+      rescue Errno::ECONNREFUSED
  242
+        raise Error, "connection refused: #{connection.address}:#{connection.port}"
  243
+      rescue Errno::EHOSTDOWN
  244
+        raise Error, "host down: #{connection.address}:#{connection.port}"
  245
+      end
  246
+    end
  247
+  end
  248
+
  249
+  ##
  250
+  # Makes a request per +req+.  If +req+ is nil a Net::HTTP::Get is performed
  251
+  # against +default_path+.
  252
+  #
  253
+  # If a block is passed #request behaves like Net::HTTP#request (the body of
  254
+  # the response will not have been read).
  255
+  #
  256
+  # +req+ must be a Net::HTTPRequest subclass (see Net::HTTP for a list).
  257
+  #
  258
+  # If there is an error and the request is idempontent according to RFC 2616
  259
+  # it will be retried automatically.
  260
+
  261
+  def request req = nil, &block
  262
+    retried      = false
  263
+    bad_response = false
  264
+
  265
+    req = Net::HTTP::Get.new @default_path unless req
  266
+
  267
+    headers.each do |pair|
  268
+      req.add_field(*pair)
  269
+    end
  270
+
  271
+    req.add_field 'Connection', 'keep-alive'
  272
+    req.add_field 'Keep-Alive', @keep_alive
  273
+
  274
+    @pool.with_connection do |connection|
  275
+      begin
  276
+        response = connection.request req, &block
  277
+        @http_version ||= response.http_version
  278
+        @count_hash[connection.object_id] += 1
  279
+        return response
  280
+
  281
+      rescue  Timeout::Error => e
  282
+        due_to = "(due to #{e.message} - #{e.class})"
  283
+        message = error_message connection
  284
+        @logger.info "#{name}: Removing connection #{due_to} #{message}" if @logger
  285
+        remove connection
  286
+        raise
  287
+        
  288
+      rescue Net::HTTPBadResponse => e
  289
+        message = error_message connection
  290
+        if bad_response or not (idempotent? req or @force_retry)
  291
+          @logger.info "#{name}: Removing connection because of too many bad responses #{message}" if @logger
  292
+          remove connection
  293
+          raise Error, "too many bad responses #{message}"
  294
+        else
  295
+          bad_response = true
  296
+          @logger.info "#{name}: Renewing connection because of bad response #{message}" if @logger
  297
+          connection = renew connection
  298
+          retry
  299
+        end
  300
+
  301
+      rescue IOError, EOFError, Errno::ECONNABORTED, Errno::ECONNRESET, Errno::EPIPE => e
  302
+        due_to = "(due to #{e.message} - #{e.class})"
  303
+        message = error_message connection
  304
+        if retried or not (idempotent? req or @force_retry)
  305
+          @logger.info "#{name}: Removing connection #{due_to} #{message}" if @logger
  306
+          remove connection
  307
+          raise Error, "too many connection resets #{due_to} #{message}"
  308
+        else
  309
+          retried = true
  310
+          @logger.info "#{name}: Renewing connection #{due_to} #{message}" if @logger
  311
+          connection = renew connection
  312
+          retry
  313
+        end
  314
+      end
  315
+    end
  316
+  end
  317
+
  318
+  ##
  319
+  # Shuts down all connections.
  320
+
  321
+  def shutdown
  322
+    raise 'Shutdown not implemented'
  323
+    # TBD - need to think about this one
  324
+    @count_hash = nil
  325
+  end
  326
+
  327
+  #######
  328
+  private
  329
+  #######
  330
+
  331
+  ##
  332
+  # Returns an error message containing the number of requests performed on
  333
+  # this connection
  334
+
  335
+  def error_message connection
  336
+    requests = @count_hash[connection.object_id] || 0
  337
+    "after #{requests} requests on #{connection.object_id}"
  338
+  end
  339
+
  340
+  ##
  341
+  # URI::escape wrapper
  342
+
  343
+  def escape str
  344
+    URI.escape str if str
  345
+  end
  346
+
  347
+  ##
  348
+  # Finishes the Net::HTTP +connection+
  349
+
  350
+  def finish connection
  351
+    @count_hash.delete(connection.object_id)
  352
+    connection.finish
  353
+  rescue IOError
  354
+  end
  355
+
  356
+  ##
  357
+  # Is +req+ idempotent according to RFC 2616?
  358
+
  359
+  def idempotent? req
  360
+    case req
  361
+    when Net::HTTP::Delete, Net::HTTP::Get, Net::HTTP::Head,
  362
+         Net::HTTP::Options, Net::HTTP::Put, Net::HTTP::Trace then
  363
+      true
  364
+    end
  365
+  end
  366
+
  367
+  ##
  368
+  # Adds "http://" to the String +uri+ if it is missing.
  369
+
  370
+  def normalize_uri uri
  371
+    (uri =~ /^https?:/) ? uri : "http://#{uri}"
  372
+  end
  373
+
  374
+  ##
  375
+  # Creates a URI for an HTTP proxy server from ENV variables.
  376
+  #
  377
+  # If +HTTP_PROXY+ is set a proxy will be returned.
  378
+  #
  379
+  # If +HTTP_PROXY_USER+ or +HTTP_PROXY_PASS+ are set the URI is given the
  380
+  # indicated user and password unless HTTP_PROXY contains either of these in
  381
+  # the URI.
  382
+  #
  383
+  # For Windows users lowercase ENV variables are preferred over uppercase ENV
  384
+  # variables.
  385
+
  386
+  def proxy_from_env
  387
+    env_proxy = ENV['http_proxy'] || ENV['HTTP_PROXY']
  388
+
  389
+    return nil if env_proxy.nil? or env_proxy.empty?
  390
+
  391
+    uri = URI.parse(normalize_uri(env_proxy))
  392
+
  393
+    unless uri.user or uri.password then
  394
+      uri.user     = escape ENV['http_proxy_user'] || ENV['HTTP_PROXY_USER']
  395
+      uri.password = escape ENV['http_proxy_pass'] || ENV['HTTP_PROXY_PASS']
  396
+    end
  397
+
  398
+    uri
  399
+  end
  400
+
  401
+  ##
  402
+  # Finishes then removes the Net::HTTP +connection+
  403
+
  404
+  def remove connection
  405
+    finish connection
  406
+    @pool.remove(connection)
  407
+  end
  408
+
  409
+  ##
  410
+  # Finishes then renews the Net::HTTP +connection+.  It may be unnecessary 
  411
+  # to completely recreate the connection but connections that get timed out
  412
+  # in JRuby leave the ssl context in a frozen object state.
  413
+
  414
+  def renew connection
  415
+    finish connection
  416
+    connection = @pool.renew(connection)
  417
+  end
  418
+
  419
+  ##
  420
+  # Enables SSL on +connection+
  421
+
  422
+  def ssl connection
  423
+    require 'net/https'
  424
+    connection.use_ssl = true
  425
+
  426
+    # suppress warning but allow override
  427
+    connection.verify_mode = OpenSSL::SSL::VERIFY_NONE unless @verify_mode
  428
+
  429
+    if @ca_file then
  430
+      connection.ca_file = @ca_file
  431
+      connection.verify_mode = OpenSSL::SSL::VERIFY_PEER
  432
+      connection.verify_callback = @verify_callback if @verify_callback
  433
+    end
  434
+
  435
+    if @certificate and @private_key then
  436
+      connection.cert = @certificate
  437
+      connection.key  = @private_key
  438
+    end
  439
+
  440
+    connection.verify_mode = @verify_mode if @verify_mode
  441
+  end
  442
+  
  443
+end
  444
+
27  lib/persistent_http/faster.rb
... ...
@@ -0,0 +1,27 @@
  1
+require 'net/protocol'
  2
+
  3
+##
  4
+# Aaron Patterson's monkeypatch (accepted into 1.9.1) to fix Net::HTTP's speed
  5
+# problems.
  6
+#
  7
+# http://gist.github.com/251244
  8
+
  9
+class Net::BufferedIO #:nodoc:
  10
+  alias :old_rbuf_fill :rbuf_fill
  11
+
  12
+  def rbuf_fill
  13
+    if @io.respond_to? :read_nonblock then
  14
+      begin
  15
+        @rbuf << @io.read_nonblock(65536)
  16
+      rescue Errno::EWOULDBLOCK => e
  17
+        retry if IO.select [@io], nil, nil, @read_timeout
  18
+        raise Timeout::Error, e.message
  19
+      end
  20
+    else # SSL sockets do not have read_nonblock
  21
+      timeout @read_timeout do
  22
+        @rbuf << @io.sysread(65536)
  23
+      end
  24
+    end
  25
+  end
  26
+end if RUBY_VERSION < '1.9'
  27
+
49  persistent_http.gemspec
... ...
@@ -0,0 +1,49 @@
  1
+# Generated by jeweler
  2
+# DO NOT EDIT THIS FILE DIRECTLY
  3
+# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
  4
+# -*- encoding: utf-8 -*-
  5
+
  6
+Gem::Specification.new do |s|
  7
+  s.name = %q{gene_pool}
  8
+  s.version = "1.0.1"
  9
+
  10
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
  11
+  s.authors = ["Brad Pardee"]
  12
+  s.date = %q{2010-09-12}
  13
+  s.description = %q{Generic pooling library for creating a connection pool}
  14
+  s.email = %q{bradpardee@gmail.com}
  15
+  s.extra_rdoc_files = [
  16
+    "LICENSE",
  17
+     "README.rdoc"
  18
+  ]
  19
+  s.files = [
  20
+    ".gitignore",
  21
+     "History.txt",
  22
+     "LICENSE",
  23
+     "README.rdoc",
  24
+     "Rakefile",
  25
+     "VERSION",
  26
+     "gene_pool.gemspec",
  27
+     "lib/gene_pool.rb",
  28
+     "test/gene_pool_test.rb"
  29
+  ]
  30
+  s.homepage = %q{http://github.com/bpardee/gene_pool}
  31
+  s.rdoc_options = ["--charset=UTF-8"]
  32
+  s.require_paths = ["lib"]
  33
+  s.rubygems_version = %q{1.3.6}
  34
+  s.summary = %q{Generic pooling library for creating a connection pool}
  35
+  s.test_files = [
  36
+    "test/gene_pool_test.rb"
  37
+  ]
  38
+
  39
+  if s.respond_to? :specification_version then
  40
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
  41
+    s.specification_version = 3
  42
+
  43
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
  44
+    else
  45
+    end
  46
+  else
  47
+  end
  48
+end
  49
+
647  test/persistent_http_test.rb
... ...
@@ -0,0 +1,647 @@
  1
+require 'rubygems'
  2
+require 'test/unit'
  3
+require 'shoulda'
  4
+require 'persistent_http'
  5
+require 'openssl'
  6
+require 'stringio'
  7
+require 'logger'
  8
+
  9
+CMD_SUCCESS      = 'success'
  10
+CMD_SLEEP        = 'sleep'
  11
+CMD_BAD_RESPONSE = 'bad_response'
  12
+CMD_EOF_ERROR    = 'eof_error'
  13
+CMD_CONNRESET    = 'connreset'
  14
+CMD_ECHO         = 'echo'
  15
+
  16
+PASS = 'pass'
  17
+FAIL = 'fail'
  18
+
  19
+DUMMY_OPEN_TIMEOUT_FOR_HOSTDOWN    = 9000
  20
+DUMMY_OPEN_TIMEOUT_FOR_CONNREFUSED = 9001
  21
+
  22
+$debug = false
  23
+$count = -1
  24
+
  25
+class Net::HTTP
  26
+  def connect
  27
+    raise Errno::EHOSTDOWN    if open_timeout == DUMMY_OPEN_TIMEOUT_FOR_HOSTDOWN
  28
+    raise Errno::ECONNREFUSED if open_timeout == DUMMY_OPEN_TIMEOUT_FOR_CONNREFUSED
  29
+  end
  30
+
  31
+  def successful_response
  32
+    r = Net::HTTPResponse.allocate
  33
+    def r.http_version() '1.1'; end
  34
+    def r.read_body() :read_body; end
  35
+    yield r if block_given?
  36
+    r
  37
+  end
  38
+
  39
+  def request(req, &block)
  40
+    $count += 1
  41
+    puts "path=#{req.path} count=#{$count}" if $debug
  42
+    args = req.path[1..-1].split('/')
  43
+    cmd = args.shift
  44
+    i = $count % args.size if args.size > 0
  45
+    puts "i=#{i}" if $debug
  46
+    if cmd == CMD_ECHO
  47
+      res = successful_response(&block)
  48
+      eval "def res.body() \"#{req.body}\" end"
  49
+      return res
  50
+    elsif cmd == CMD_SUCCESS || args[i] == PASS
  51
+      return successful_response(&block)
  52
+    end
  53
+    case cmd
  54
+    when CMD_SLEEP
  55
+      sleep args[i].to_i
  56
+      return successful_response(&block)
  57
+    when CMD_BAD_RESPONSE
  58
+      raise Net::HTTPBadResponse.new('Dummy bad response') 
  59
+    when CMD_EOF_ERROR
  60
+      raise EOFError.new('Dummy EOF error') 
  61
+    when CMD_CONNRESET
  62
+      raise Errno::ECONNRESET
  63
+    else
  64
+      return successful_response(&block)
  65
+    end 
  66
+  end
  67
+end
  68
+
  69
+class PersistentHTTP
  70
+  attr_reader :pool
  71
+
  72
+  # Make private methods public
  73
+  send(:public, *(self.private_instance_methods - Object.private_instance_methods))
  74
+end
  75
+
  76
+class PersistentHTTPTest < Test::Unit::TestCase
  77
+
  78
+  def clear_proxy_env
  79
+    ENV.delete 'http_proxy'
  80
+    ENV.delete 'HTTP_PROXY'
  81
+    ENV.delete 'http_proxy_user'
  82
+    ENV.delete 'HTTP_PROXY_USER'
  83
+    ENV.delete 'http_proxy_pass'
  84
+    ENV.delete 'HTTP_PROXY_PASS'
  85
+  end
  86
+
  87
+  def uri_for(*args)
  88
+    '/' + args.join('/')
  89
+  end
  90
+
  91
+  def get_request(*args)
  92
+    puts "uri=#{uri_for(args)}" if $debug
  93
+    $count = -1
  94
+    return Net::HTTP::Get.new(uri_for(args))
  95
+  end
  96
+  
  97
+  def post_request(*args)
  98
+    puts "uri=#{uri_for(args)}" if $debug
  99
+    $count = -1
  100
+    return Net::HTTP::Post.new(uri_for(args))
  101
+  end
  102
+
  103
+  def http_and_io(options={})
  104
+    io = StringIO.new
  105
+    logger = Logger.new(io)
  106
+    logger.level = Logger::INFO
  107
+    default_options = {:name => 'TestNetHTTPPersistent', :logger => logger, :pool_size => 1}
  108
+    http = PersistentHTTP.new(default_options.merge(options))
  109
+    [http, io]
  110
+  end
  111
+
  112
+  context 'simple setup' do
  113
+    setup do
  114
+      @io = StringIO.new
  115
+      logger = Logger.new(@io)
  116
+      logger.level = Logger::INFO
  117
+      @http = PersistentHTTP.new(:host => 'example.com', :name => 'TestNetHTTPPersistent', :logger => logger)
  118
+      @http.headers['user-agent'] = 'test ua'
  119
+    end
  120
+
  121
+    should 'have options set' do
  122
+      assert_equal @http.proxy_uri, nil
  123
+      assert_equal 'TestNetHTTPPersistent', @http.name
  124
+    end
  125
+
  126
+    should 'handle escape' do
  127
+      assert_equal nil,  @http.escape(nil)
  128
+      assert_equal '%20', @http.escape(' ')
  129
+    end
  130
+
  131
+    should 'handle error' do
  132
+      req = get_request CMD_EOF_ERROR, PASS, PASS, PASS, PASS, FAIL, PASS, PASS
  133
+      6.times do
  134
+        @http.request(req)
  135
+      end
  136
+      assert_match "after 4 requests on", @io.string
  137
+    end
  138
+
  139
+    should 'handle finish' do
  140
+      c = Object.new
  141
+      def c.finish; @finished = true end
  142
+      def c.finished?; @finished end
  143
+      def c.start; @started = true end
  144
+      def c.started?; @started end
  145
+
  146
+      @http.finish c
  147
+
  148
+      assert !c.started?
  149
+      assert c.finished?
  150
+    end
  151
+
  152
+    should 'handle finish io error' do
  153
+      c = Object.new
  154
+      def c.finish; @finished = true; raise IOError end
  155
+      def c.finished?; @finished end
  156
+      def c.start; @started = true end
  157
+      def c.started?; @started end
  158
+
  159
+      @http.finish c
  160
+
  161
+      assert !c.started?
  162
+      assert c.finished?
  163
+    end
  164
+
  165
+    should 'fill in http version' do
  166
+      assert_nil @http.http_version
  167
+      @http.request(get_request(CMD_SUCCESS))
  168
+      assert_equal '1.1', @http.http_version
  169
+    end
  170
+
  171
+    should 'handle idempotent' do
  172
+      assert @http.idempotent? Net::HTTP::Delete.new '/'
  173
+      assert @http.idempotent? Net::HTTP::Get.new '/'
  174
+      assert @http.idempotent? Net::HTTP::Head.new '/'
  175
+      assert @http.idempotent? Net::HTTP::Options.new '/'
  176
+      assert @http.idempotent? Net::HTTP::Put.new '/'
  177
+      assert @http.idempotent? Net::HTTP::Trace.new '/'
  178
+
  179
+      assert !@http.idempotent?(Net::HTTP::Post.new '/')
  180
+    end
  181
+
  182
+    should 'handle normalize_uri' do
  183
+      assert_equal 'http://example',  @http.normalize_uri('example')
  184
+      assert_equal 'http://example',  @http.normalize_uri('http://example')
  185
+      assert_equal 'https://example', @http.normalize_uri('https://example')
  186
+    end
  187
+
  188
+    should 'handle simple request' do
  189
+      req = get_request(CMD_SUCCESS)
  190
+      res = @http.request(req)
  191
+    
  192
+      assert_kind_of Net::HTTPResponse, res
  193
+    
  194
+      assert_kind_of Net::HTTP::Get, req
  195
+      assert_equal uri_for(CMD_SUCCESS), req.path
  196
+      assert_equal 'keep-alive',         req['connection']
  197
+      assert_equal '30',                 req['keep-alive']
  198
+      assert_match %r%test ua%,          req['user-agent']
  199
+    end
  200
+
  201
+    should 'handle request with block' do
  202
+      body = nil
  203
+      
  204
+      req = get_request(CMD_SUCCESS)
  205
+      res = @http.request(req) do |r|
  206
+        body = r.read_body
  207
+      end
  208
+    
  209
+      assert_kind_of Net::HTTPResponse, res
  210
+      assert !body.nil?
  211
+    
  212
+      assert_kind_of Net::HTTP::Get, req
  213
+      assert_equal uri_for(CMD_SUCCESS), req.path
  214
+      assert_equal 'keep-alive',         req['connection']
  215
+      assert_equal '30',                 req['keep-alive']
  216
+      assert_match %r%test ua%,          req['user-agent']
  217
+    end
  218
+    
  219
+    should 'handle bad response' do
  220
+      req = get_request(CMD_BAD_RESPONSE, FAIL, FAIL)
  221
+      e = assert_raises PersistentHTTP::Error do
  222
+        @http.request req 
  223
+      end
  224
+      assert_match %r%too many bad responses%, e.message
  225
+      assert_match %r%Renewing connection because of bad response%, @io.string
  226
+      assert_match %r%Removing connection because of too many bad responses%, @io.string
  227
+  
  228
+      res = @http.request(get_request(CMD_SUCCESS))
  229
+      assert_kind_of Net::HTTPResponse, res
  230
+    end
  231
+
  232
+    should 'handle connection reset' do
  233
+      req = get_request(CMD_CONNRESET, FAIL, FAIL)
  234
+      e = assert_raises PersistentHTTP::Error do
  235
+        @http.request req 
  236
+      end
  237
+    
  238
+      assert_match %r%too many connection resets%, e.message
  239
+      assert_match %r%Renewing connection %, @io.string
  240
+      assert_match %r%Removing connection %, @io.string
  241
+  
  242
+      res = @http.request(get_request(CMD_SUCCESS))
  243
+      assert_kind_of Net::HTTPResponse, res
  244
+    end
  245
+
  246
+    should 'retry on bad response' do
  247
+      res = @http.request(get_request(CMD_BAD_RESPONSE, FAIL, PASS))
  248
+      assert_match %r%Renewing connection because of bad response%, @io.string
  249
+      assert_kind_of Net::HTTPResponse, res
  250
+    end
  251
+
  252
+    should 'retry on connection reset' do
  253
+      res = @http.request(get_request(CMD_CONNRESET, FAIL, PASS))
  254
+      assert_match %r%Renewing connection %, @io.string
  255
+      assert_kind_of Net::HTTPResponse, res
  256
+    end
  257
+    
  258
+    should 'not retry on bad response from post' do
  259
+      post = post_request(CMD_BAD_RESPONSE, FAIL, PASS)
  260
+      e = assert_raises PersistentHTTP::Error do
  261
+        @http.request(post)
  262
+      end
  263
+      assert_match %r%too many bad responses%, e.message
  264
+      assert_match %r%Removing connection because of too many bad responses%, @io.string
  265
+  
  266
+      res = @http.request(get_request(CMD_SUCCESS))
  267
+      assert_kind_of Net::HTTPResponse, res
  268
+    end
  269
+
  270
+    should 'not retry on connection reset from post' do
  271
+      post = post_request(CMD_CONNRESET, FAIL, PASS)
  272
+      e = assert_raises PersistentHTTP::Error do
  273
+        @http.request(post)
  274
+      end
  275
+      assert_match %r%too many connection resets%, e.message
  276
+      assert_match %r%Removing connection %, @io.string
  277
+  
  278
+      res = @http.request(get_request(CMD_SUCCESS))
  279
+      assert_kind_of Net::HTTPResponse, res
  280
+    end
  281
+    
  282
+    should 'retry on bad response from post when force_retry set' do
  283
+      @http.force_retry = true
  284
+      post = post_request(CMD_BAD_RESPONSE, FAIL, PASS)
  285
+      res = @http.request post
  286
+      assert_match %r%Renewing connection because of bad response%, @io.string
  287
+      assert_kind_of Net::HTTPResponse, res
  288
+    end
  289
+
  290
+    should 'retry on connection reset from post when force_retry set' do
  291
+      @http.force_retry = true
  292
+      post = post_request(CMD_CONNRESET, FAIL, PASS)
  293
+      res = @http.request post
  294
+      assert_match %r%Renewing connection %, @io.string
  295
+      assert_kind_of Net::HTTPResponse, res
  296
+    end
  297
+
  298
+    should 'allow post' do
  299
+      post = Net::HTTP::Post.new(uri_for CMD_ECHO)
  300
+      post.body = 'hello PersistentHTTP'
  301
+      res = @http.request(post)
  302
+      assert_kind_of Net::HTTPResponse, res
  303
+      assert_equal post.body, res.body
  304
+    end
  305
+
  306
+    should 'allow ssl' do
  307
+      @http.verify_callback = :callback
  308
+      c = Net::HTTP.new('localhost', 80)
  309
+    
  310
+      @http.ssl c
  311
+    
  312
+      assert c.use_ssl?
  313
+      assert_equal OpenSSL::SSL::VERIFY_NONE, c.verify_mode
  314
+      assert_nil c.verify_callback
  315
+    end
  316
+    
  317
+    should 'allow ssl ca_file' do
  318
+      @http.ca_file = 'ca_file'
  319
+      @http.verify_callback = :callback
  320
+      c = Net::HTTP.new('localhost', 80)
  321
+    
  322
+      @http.ssl c
  323
+    
  324
+      assert c.use_ssl?
  325
+      assert_equal OpenSSL::SSL::VERIFY_PEER, c.verify_mode
  326
+      assert_equal :callback, c.verify_callback
  327
+    end
  328
+    
  329
+    should 'allow ssl certificate' do
  330
+      @http.certificate = :cert
  331
+      @http.private_key = :key
  332
+      c = Net::HTTP.new('localhost', 80)
  333
+    
  334
+      @http.ssl c
  335
+    
  336
+      assert c.use_ssl?
  337
+      assert_equal :cert, c.cert
  338
+      assert_equal :key,  c.key
  339
+    end
  340
+    
  341
+    should 'allow ssl verify_mode' do
  342
+      @http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  343
+      c = Net::HTTP.new('localhost', 80)
  344
+    
  345
+      @http.ssl c
  346
+    
  347
+      assert c.use_ssl?
  348
+      assert_equal OpenSSL::SSL::VERIFY_NONE, c.verify_mode
  349
+    end
  350
+  end
  351
+
  352
+  context 'initialize proxy by env' do
  353
+    setup do
  354
+      clear_proxy_env
  355
+      ENV['HTTP_PROXY'] = 'proxy.example'
  356
+      @http = PersistentHTTP.new(:host => 'foobar', :proxy => :ENV)
  357
+    end
  358
+
  359
+    should 'match HTTP_PROXY' do
  360
+      assert_equal URI.parse('http://proxy.example'), @http.proxy_uri
  361
+      assert_equal 'foobar', @http.host
  362
+    end
  363
+  end
  364
+
  365
+  context 'initialize proxy by uri' do
  366
+    setup do
  367
+      @proxy_uri          = URI.parse 'http://proxy.example'
  368
+      @proxy_uri.user     = 'johndoe'
  369
+      @proxy_uri.password = 'muffins'
  370
+      @http               = PersistentHTTP.new(:url => 'https://zulu.com/foobar', :proxy => @proxy_uri)
  371
+    end
  372
+
  373
+    should 'match proxy_uri and have proxy connection' do
  374
+      assert_equal @proxy_uri, @http.proxy_uri
  375
+      assert_equal true, @http.use_ssl
  376
+      assert_equal 'zulu.com', @http.host
  377
+      assert_equal '/foobar', @http.default_path
  378
+
  379
+      @http.pool.with_connection do |c|
  380
+        assert c.started?
  381
+        assert c.proxy?
  382
+      end
  383
+    end
  384
+  end
  385
+
  386
+  context 'initialize proxy by env' do
  387
+    setup do
  388
+      clear_proxy_env
  389
+      ENV['HTTP_PROXY']      = 'proxy.example'
  390
+      ENV['HTTP_PROXY_USER'] = 'johndoe'
  391
+      ENV['HTTP_PROXY_PASS'] = 'muffins'
  392
+      @http                  = PersistentHTTP.new(:url => 'https://zulu.com/foobar', :proxy => :ENV)
  393
+    end
  394
+
  395
+    should 'create proxy_uri from env' do
  396
+      expected          = URI.parse 'http://proxy.example'
  397
+      expected.user     = 'johndoe'
  398
+      expected.password = 'muffins'
  399
+
  400
+      assert_equal expected, @http.proxy_uri
  401
+    end
  402
+  end
  403
+
  404
+  context 'initialize proxy by env lower' do
  405
+    setup do
  406
+      clear_proxy_env
  407
+      ENV['http_proxy']      = 'proxy.example'
  408
+      ENV['http_proxy_user'] = 'johndoe'
  409
+      ENV['http_proxy_pass'] = 'muffins'
  410
+      @http                  = PersistentHTTP.new(:url => 'https://zulu.com/foobar', :proxy => :ENV)
  411
+    end
  412
+
  413
+    should 'create proxy_uri from env' do
  414
+      expected          = URI.parse 'http://proxy.example'
  415
+      expected.user     = 'johndoe'
  416
+      expected.password = 'muffins'
  417
+
  418
+      assert_equal expected, @http.proxy_uri
  419
+    end
  420
+  end
  421
+
  422
+  context 'with timeouts set' do
  423
+    setup do
  424
+      @http = PersistentHTTP.new(:url => 'http://example.com')
  425
+      @http.open_timeout = 123
  426
+      @http.read_timeout = 321
  427
+    end
  428
+
  429
+    should 'have timeouts set' do
  430
+      @http.pool.with_connection do |c|
  431
+        assert c.started?
  432
+        assert !c.proxy?
  433
+
  434
+        assert_equal 123, c.open_timeout
  435
+        assert_equal 321, c.read_timeout
  436
+
  437
+        assert_equal 'example.com', c.address
  438
+        assert_equal 80, c.port
  439
+        assert !@http.use_ssl
  440
+      end
  441
+    end
  442
+
  443
+    should 'reuse same connection' do
  444
+      c1, c2 = nil, nil
  445
+      @http.pool.with_connection do |c|
  446
+        c1 = c
  447
+        assert c.started?
  448
+      end
  449
+      @http.pool.with_connection do |c|
  450
+        c2 = c
  451
+        assert c.started?
  452
+      end
  453
+      assert_same c1,c2
  454
+    end
  455
+  end
  456
+
  457
+  context 'with debug_output' do
  458
+    setup do
  459
+      @io = StringIO.new
  460
+      @http = PersistentHTTP.new(:url => 'http://example.com', :debug_output => @io)
  461
+    end
  462
+
  463
+    should 'have debug_output set' do
  464
+      @http.pool.with_connection do |c|
  465
+        assert c.started?
  466
+        assert_equal @io, c.instance_variable_get(:@debug_output)
  467
+        assert_equal 'example.com', c.address
  468
+        assert_equal 80, c.port
  469
+      end
  470
+    end
  471
+  end
  472
+
  473
+  context 'with host down' do
  474
+    setup do 
  475
+      @http = PersistentHTTP.new(:url => 'http://example.com', :open_timeout => DUMMY_OPEN_TIMEOUT_FOR_HOSTDOWN)
  476
+    end
  477
+
  478
+    should 'assert error' do
  479
+      e = assert_raises PersistentHTTP::Error do
  480
+        @http.request(get_request(CMD_SUCCESS))
  481
+      end
  482
+      assert_match %r%host down%, e.message
  483
+    end
  484
+  end
  485
+
  486
+  context 'with connection refused' do
  487
+    setup do
  488
+      @http = PersistentHTTP.new(:url => 'http://example.com', :open_timeout => DUMMY_OPEN_TIMEOUT_FOR_CONNREFUSED)
  489
+    end
  490
+
  491
+    should 'assert error' do
  492
+      e = assert_raises PersistentHTTP::Error do
  493
+        @http.request(get_request(CMD_SUCCESS))
  494
+      end
  495
+      assert_match %r%connection refused%, e.message
  496
+    end
  497
+  end
  498
+  
  499
+  context 'with pool size of 3' do
  500
+    setup do
  501
+      @http = PersistentHTTP.new(:url => 'http://example.com', :pool_size => 3)
  502
+    end
  503
+    
  504
+    should 'only allow 3 connections checked out at a time' do
  505
+      @http.request(get_request(CMD_SUCCESS))
  506
+      pool = @http.pool
  507
+      2.times do
  508
+        conns = []
  509
+        pool.with_connection do |c1|
  510
+          pool.with_connection do |c2|
  511
+            conns << c2
  512
+            pool.with_connection do |c3|
  513
+              conns << c3
  514
+              begin
  515
+                Timeout.timeout(2) do
  516
+                  pool.with_connection { |c4| }
  517
+                  assert false, 'should NOT have been able to get 4th connection'
  518
+                end
  519
+              rescue  Timeout::Error => e
  520
+                # successfully failed to get a connection
  521
+              end
  522
+              @http.remove(c1)
  523
+              Timeout.timeout(1) do
  524
+                begin
  525
+                  pool.with_connection do |c4|
  526
+                    conns << c4
  527
+                  end
  528
+                rescue  Timeout::Error => e
  529
+                  assert false, 'should have been able to get 4th connection'
  530
+                end
  531
+              end
  532
+            end
  533
+          end