HTTP::Client proxy support

[Rinkana]

The current HTTP::Client does not support the ability to use a proxy. And it would certainly be a nice feature to have.

It might be an idea to implement this alongside the HTTP/2.2 support that is planned for some time.

[chainum]

Would be great to have this in the standard library.

Here's how you can implement proxy support meanwhile it makes its way into the standard library:

## PROXY
require "openssl" ifdef !without_openssl
require "socket"
require "base64"

# Based on https://github.com/net-ssh/net-ssh/blob/master/lib/net/ssh/proxy/http.rb
class HTTPProxy

  # The hostname or IP address of the HTTP proxy.
  getter proxy_host : String

  # The port number of the proxy.
  getter proxy_port : Int32

  # The map of additional options that were given to the object at
  # initialization.
  getter options : Hash(Symbol,String)

  getter tls : OpenSSL::SSL::Context::Client?

  # Create a new socket factory that tunnels via the given host and
  # port. The +options+ parameter is a hash of additional settings that
  # can be used to tweak this proxy connection. Specifically, the following
  # options are supported:
  #
  # * :user => the user name to use when authenticating to the proxy
  # * :password => the password to use when authenticating
  def initialize(@proxy_host, @proxy_port = 80, @options = {} of Symbol => String)
  end

  # Return a new socket connected to the given host and port via the
  # proxy that was requested when the socket factory was instantiated.
  def open(host, port, tls = nil, connection_options = {} of Symbol => Float64 | Nil)
    dns_timeout           =   connection_options.fetch(:dns_timeout, nil)
    connect_timeout       =   connection_options.fetch(:connect_timeout, nil)
    read_timeout          =   connection_options.fetch(:read_timeout, nil)

    socket                =   TCPSocket.new @proxy_host, @proxy_port, dns_timeout, connect_timeout
    socket.read_timeout   =   read_timeout if read_timeout
    socket.sync           =   true

    socket << "CONNECT #{host}:#{port} HTTP/1.0\r\n"

    if options[:user]
      credentials   =   Base64.strict_encode("#{options[:user]}:#{options[:password]}")
      credentials   =   "#{credentials}\n".gsub(/\s/, "")
      socket       <<   "Proxy-Authorization: Basic #{credentials}\r\n"
    end

    socket         <<   "\r\n"

    resp            =   parse_response(socket)

    if resp[:code]? == 200
      ifdef !without_openssl
        if tls
          tls_socket = OpenSSL::SSL::Socket::Client.new(socket, context: tls, sync_close: true, hostname: host)
          socket = tls_socket
        end
      end

      return socket
    else
      socket.close
      raise IO::Error.new(resp.inspect)
    end
  end

  private def parse_response(socket)
    resp            =   {} of Symbol => Int32 | String | Hash(String, String)

    begin
      version, code, reason = socket.gets.as(String).chomp.split(/ /, 3)

      headers       =   {} of String => String

      while (line = socket.gets.as(String)) && (line.chomp != "")
        name, value = line.split(/:/, 2)
        headers[name.strip] = value.strip
      end

      resp[:version]  =   version
      resp[:code]     =   code.to_i
      resp[:reason]   =   reason
      resp[:headers]  =   headers
    rescue
    end

    return resp
  end

end


## CLIENT
require "http/client"

class HTTPClient < ::HTTP::Client

  def set_proxy(proxy : HTTPProxy)
    begin
      @socket               =   proxy.open(host: @host, port: @port, tls: @tls, connection_options: proxy_connection_options)
    rescue IO::Error
      @socket               =   nil
    end
  end

  def proxy_connection_options
    opts                    =   {} of Symbol => Float64 | Nil

    opts[:dns_timeout]      =   @dns_timeout
    opts[:connect_timeout]  =   @connect_timeout
    opts[:read_timeout]     =   @read_timeout

    return opts
  end

end


## USE CASE
proxy_host            =   "127.0.0.1"
proxy_port            =   1234

options               =   {} of Symbol => String
options[:user]        =   "usr1"
options[:password]    =   "pw1"

proxy                 =   HTTPProxy.new(proxy_host: proxy_host, proxy_port: proxy_port, options: options)
client                =   HTTPClient.new("www.google.com")
client.set_proxy(proxy)
response              =   client.get("/")

puts "Response status: #{response.try &.status_code}"

[rdp]

If desperate some shards support it: https://github.com/veelenga/awesome-crystal#http

[straight-shoota]

#13063 proposes an implementation based on an environment variable for configuring a proxy.
This may be a good solution for many programs, but for a generic library, such a global setting seems not flexible enough. It would be unnecessarily complex to have different client instances with different proxy settings simultaneously.
This feature should be exposed in API, not through the environment.

My idea would be a property HTTP::Client#proxy, maybe also a custom constructor option.

Maybe it could make sense to add a generic environment variable, but this should come as a later enhancement.

[unixfox]

#13063 proposes an implementation based on an environment variable for configuring a proxy. This may be a good solution for many programs, but for a generic library, such a global setting seems not flexible enough. It would be unnecessarily complex to have different client instances with different proxy settings simultaneously. This feature should be exposed in API, not through the environment.

Just as a remark, Google allows setting an HTTP proxy using environment variable, so it's not uncommon: https://pkg.go.dev/golang.org/x/net/http/httpproxy

[straight-shoota]

Sure. As I mentioned, this is something many programs would want to have. But for a generic HTTP client library, this is not sufficient. It needs to be configurable per client instance. That's why I think the stdlib API should provide that.
It's easy to implement an environment variable configuration option on top of that then.

That's exactly how Golang is doing it: x/net/http/httproxy is an extension library that enhances the basic implementation in net/http.

[asterite]

In which case, in a server, you would want to use two different proxies? I thought servers were behind a proxy so it kind of makes sense that this is a global configuration.

[straight-shoota]

Not sure what you're talking about? This is about HTTP::Client, not an HTTP server.
In the server context, it's usually a "reverse proxy" (which is like a proxy... just reversed 💁‍♂️), do you mean that?

[asterite]

I honestly don't know much about proxies. But if you are in a machine behind a proxy and you need to make an http request, you tell it to use that proxy. Is that more or less how it works? If so, are you usually behind one proxy, or do you need to choose one of several?

[straight-shoota]

Well yes, that's a common use case that servers outside your local network (for example those on the open internet) are only reachable via proxy. In that case you would probably use a single proxy for all (or most) HTTP requests.
But networks can be more complex than that. You could have servers in network A only reachable via proxy X, while those in B are only reachable via Y. However your application determines these relationships, the important part is that the generic HTTP::Client implementation should support this kind of customization on the scope of a single instance.
On top of that it's easy to implement the behaviour of the common use case for a global configuration per environment variable.

[wishdev]

#13063 Has been updated to work with a proxy property at the class, subclass, and instance level. It looks for the proxy in the following order

instance -> subclass (if applicable) -> class -> environment variables (http_proxy, https_proxy, and all_proxy are respected)

It should be as customizable as one would need.......