Proxy Support

[Redth]

HTTP Proxy support would be fantastic!

[weppos]

Hi Redth,

I'm somehow confused. The HTTP protocol is not used at all in the whois library since the WHOIS protocol itself is normally defined as a TCP/IP Socket connection to the port 43.

I can't really understand how an HTTP Proxy would be useful in this case.

[Redth]

I know this does sound a bit strange, but it is in fact possible to tunnel a TCP connection over a HTTP Proxy. I'm not terribly familiar with Ruby myself, so I can't give you any specific examples in ruby, but I am doing the same in C# already.

Here's an excerpt from a wikipedia page that pretty much sums up the concept:
http://en.wikipedia.org/wiki/Tunneling_protocol

Another HTTP-based tunneling method uses the HTTP CONNECT method/command. A client issues the HTTP CONNECT command to a HTTP proxy. The proxy then makes a TCP connection to a particular server:port, and relays data between that server:port and the client connection. Because this creates a security hole, CONNECT-capable HTTP proxies commonly restrict access to the CONNECT method. The proxy allows access only to a whitelist of specific authorized servers.

As the quote also states, almost always a proxy allowing use of CONNECT requires authentication, as is the case in my specific need scenario...

[weppos]

Thanks for the documentation. I inspected the Ruby Net::HTTP proxy source code and I was able to find the use of the CONNECT statement.

I must confess I don't have a deep experience with proxy connections and, also, I don't have any proxy configuration here to use to test the feature.

[Redth]

I've PM'd you some proxies you can use to test with...

[weppos]

Closing old feature request. Please feel free to provide a patch.

[troelskn]

ruby whois works fine with ruby-proxifier (https://github.com/samuelkadolph/ruby-proxifier). It might be worth putting a note about this in the documentation.

[ruben-verhagen]

@troelskn can you provide an example on how to use ruby-proxifier for ruby whois ? thanks in advance .

[ruben-verhagen]

@Redth were you somehow able to use proxies for whois ?

[troelskn]

Sine @ruben-verhagen and others have asked how to use proxifier, here's a small overview:

proxifier works by patching the core ruby tcp libraries. So all you have to do is literally to require 'proxifier/env' and then set an ENV variable, depending on the type of proxy. Most likely, you want to set ENV['http_proxy']. For example:

require 'whois'
require 'proxifier/env'

ENV['http_proxy'] = 'xxx:xxx@us-il.proxymesh.com:31280'
client = Whois::Client.new
record = client.lookup("google.com")
puts record

(Note the above won't work - you need to replace the http_proxy setting with whatever http proxy you use. I recommend proxymesh, but have no affiliation)

I ended up needing to cycle through multiple proxies, so had to write a small patch to proxifier to allow this. Code below:

# An array of proxies to cycle through
class ProxyPool < Array

  def self.instance
    @instance ||= new
  end

  def initialize
    @lock = Mutex.new
    @pos = 0
  end

  def checkout
    if any?
      @lock.synchronize do
        next_item = self[@pos % length]
        @pos += 1
        next_item
      end
    end
  end

  module Hook
    def self.included(klass)
      klass.class_eval do
        def self.environment_proxy
          ProxyPool.instance.checkout || super
        end
      end
    end
  end

end # class ProxyPool

class TCPSocket
  include ProxyPool::Hook
end

Usage:

require 'whois'
require 'proxifier/env'
require 'proxy_pool'
ProxyPool.instance << 'xxx:xxx@us-il.proxymesh.com:31280'
ProxyPool.instance << 'xxx:xxx@us-ca.proxymesh.com:31280'
client = Whois::Client.new
record = client.lookup("google.com")
puts record

You can add as many proxies as you like to the pool.

[chamnap]

@troelskn, I don't know why I always got 403 from myprivateproxy.net. Any idea? Is it because of those proxies?