Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

fix https_proxy problem for IO::Socket::SSL and Net::SSL, includes extensive test (Part#1) #52

Merged
merged 1 commit into from

4 participants

@noxxi

Hi,
this change fixes the behavior of https_proxy.
Current behavior: send unencrypted requests with https:// URLs to specified proxy
Correct behavior: create encrypted tunnel to proxy and send requests within
This is part#1 of the change, the rest is in lwp-protocol-https

Regards,
Steffen Ullrich (maintainer IO::Socket::SSL)

Steffen Ullrich correct behavior for https_proxy,
e.g. don't send plain https:// requests to proxy, but instead establish
CONNECT tunnel and then send requests inside tunnel.
This change does together with a change in LWP::Protocol::https.
The change supports LWP::Protocol::https with the default
IO::Socket::SSL backend, but also with Net::SSL. Also:
- proxy authorization is supported (http://user:pass@host:port as proxy
  URL, Net::SSL still needs special HTTPS_PROXY_* environemt variables,
  as before)
- CONNECT request does not need to be the first request inside the
  tunnel (not with Net::SSL)
- conn_cache is read and written inside request(), instead of writing in
  request() and reading in _new_socket(). If a https tunnel is
  established the cache_key no longer depends only on proxy host,port
  but also on the tunnel endpoint
- CONNECT is a proxy request and must always use Proxy-Authorization,
  not Authorization header
cb80c2d
@dod38fr

Hello

In short, this patch works well to connect to https://github.com through a proxy with hostname verification.

In more details, here are the tests I performed. I've slightly edited the program output for brevity.

I started with the following simple program:

use LWP::UserAgent; 
my $ua = LWP::UserAgent->new(); 
my $req = HTTP::Request->new('GET','https://github.com/libwww-perl'); 
my $res = $ua->request($req); 
print $res->dump(), "\n";

Direct connection with vanilla libwww-perl fails:

500 Can't verify SSL peers without knowing which Certificate Authorities to trust
Content-Type: text/plain
Client-Date: Wed, 20 Nov 2013 19:45:09 GMT
Client-Warning: Internal response

Direct connection with patched libwww-perl and lwp-protocol-https fails the same way.

Once the ca file is specified, direct connections works in both cases:

$ PERL_LWP_SSL_CA_FILE=/etc/ssl/certs/ca-certificates.crt perl -I libwww-perl/lib/ -I lwp-protocol-https/lib/ test-proxy.pl | grep Peer
Client-Peer: 192.30.252.129:443

Now with a proxy (tinyproxy on localhost). The test program is now:

use LWP::UserAgent; 
my $ua = LWP::UserAgent->new(); 
$ua->proxy('https' , 'http://127.0.0.1:8888') ;
my $req = HTTP::Request->new('GET','https://github.com/libwww-perl'); 
my $res = $ua->request($req); 
print $res->dump(), "\n";

Vanilla libwww-perl fails (with or without ca file):

$ perl test-proxy.pl 
HTTP/1.1 301 Moved Permanently
Via: 1.1 tinyproxy (tinyproxy/1.8.3)
Location: https://github.com/libwww-perl

Patched liwwww-perl without ca file fails with the same error as above, which is normal.

Patched liwwww-perl with ca file works (YEAH :-D):

$ PERL_LWP_SSL_CA_FILE=/etc/ssl/certs/ca-certificates.crt perl -I libwww-perl/lib/ -I lwp-protocol-https/lib/ test-proxy.pl 
HTTP/1.1 200 OK
Cache-Control: private, max-age=0, must-revalidate
Connection: close
Date: Sat, 16 Nov 2013 16:27:26 GMT
ETag: "d5156fbbd63f60cce60c6b275ca54e63"
Server: GitHub.com
Vary: Accept-Encoding
Content-Length: 17355
Content-Type: text/html; charset=utf-8
Client-Date: Sat, 16 Nov 2013 16:27:26 GMT
Client-Peer: 127.0.0.1:8888
Client-Response-Num: 1

Now using https_proxy env variable:

use LWP::UserAgent; 
my $ua = LWP::UserAgent->new(); 
$ua->env_proxy;
my $req = HTTP::Request->new('GET','https://github.com/libwww-perl'); 
my $res = $ua->request($req); 
print $res->dump(), "\n";

The proxy connection works ( :-D ):

$ https_proxy=http://localhost:8888 PERL_LWP_SSL_CA_FILE=/etc/ssl/certs/ca-certificates.crt perl -I libwww-perl/lib/ -I lwp-protocol-https/lib/ test-proxy.pl 
HTTP/1.1 200 OK
Cache-Control: private, max-age=0, must-revalidate
Connection: close
Date: Sat, 16 Nov 2013 16:35:16 GMT
ETag: "f6d5aeb24e6f297c0506a6373bf014ea"
Server: GitHub.com
Vary: Accept-Encoding
Content-Length: 17355
Content-Type: text/html; charset=utf-8
Client-Date: Sat, 16 Nov 2013 16:35:16 GMT
Client-Peer: 127.0.0.1:8888
Client-Response-Num: 1

Many thanks to @noxxi for these patches. I hope they'll soon be merged.

Hope this helps

@noxxi
@dod38fr

@noxxi : yes. You're right. Thanks for the clarification.

@dod38fr

Following @noxxi 's comments, I've updated the test report. @gisle , this patch looks good to me. What do you think ?

All the best

@gisle gisle merged commit cb80c2d into libwww-perl:master
@gisle
Owner

Seems sensible to me. I've merged the patch.

@dod38fr

@gisle PR libwww-perl/lwp-protocol-https#7 also needs to be merged for this fix to be complete

@goneri goneri referenced this pull request from a commit in fusioninventory/fusioninventory-agent
@goneri goneri test: fix the HTTPS over proxy test
The test was failing when libwww-perl really support HTTPS over proxy.
See: libwww-perl/libwww-perl#52
fe68a18
@goneri goneri referenced this pull request from a commit in fusioninventory/fusioninventory-agent
@goneri goneri test: fix the HTTPS over proxy test
The test was failing when libwww-perl really support HTTPS over
proxy.
See: libwww-perl/libwww-perl#52
591645c
@goneri goneri referenced this pull request from a commit in fusioninventory/fusioninventory-agent
@goneri goneri test: fix the HTTPS over proxy test
The test was failing when libwww-perl really support HTTPS over
proxy.
See: libwww-perl/libwww-perl#52
1e9e1d5
@thoke

Any chance this should just be '$response->is_success or die $response;'? That way the proper HTTP status can be propagated through to the callers. In my case, I'm looking for the 407 so proxy authentication can get triggered.

@thoke
Collaborator

Now that this is in v6.06, I'm doing some testing with it. I'm finding issues with two things:
1. Some of the $response->is_success or die situations are resulting in propagation of status 500 plus the underlying real HTTP status (407 proxy auth in my case).
2. This will only work with basic authentication to the proxy and only if it's specified on the proxy url.

Do you have any desire to extend this to work with provided credentials? e.g. $ua->credentials(...) Or can share any pointers so I can look into it further?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Nov 9, 2013
  1. correct behavior for https_proxy,

    Steffen Ullrich authored
    e.g. don't send plain https:// requests to proxy, but instead establish
    CONNECT tunnel and then send requests inside tunnel.
    This change does together with a change in LWP::Protocol::https.
    The change supports LWP::Protocol::https with the default
    IO::Socket::SSL backend, but also with Net::SSL. Also:
    - proxy authorization is supported (http://user:pass@host:port as proxy
      URL, Net::SSL still needs special HTTPS_PROXY_* environemt variables,
      as before)
    - CONNECT request does not need to be the first request inside the
      tunnel (not with Net::SSL)
    - conn_cache is read and written inside request(), instead of writing in
      request() and reading in _new_socket(). If a https tunnel is
      established the cache_key no longer depends only on proxy host,port
      but also on the tunnel endpoint
    - CONNECT is a proxy request and must always use Proxy-Authorization,
      not Authorization header
This page is out of date. Refresh to see the latest.
Showing with 79 additions and 33 deletions.
  1. +77 −32 lib/LWP/Protocol/http.pm
  2. +2 −1  lib/LWP/UserAgent.pm
View
109 lib/LWP/Protocol/http.pm
@@ -16,16 +16,6 @@ my $CRLF = "\015\012";
sub _new_socket
{
my($self, $host, $port, $timeout) = @_;
- my $conn_cache = $self->{ua}{conn_cache};
- if ($conn_cache) {
- if (my $sock = $conn_cache->withdraw($self->socket_type, "$host:$port")) {
- return $sock if $sock && !$sock->can_read(0);
- # if the socket is readable, then either the peer has closed the
- # connection or there are some garbage bytes on it. In either
- # case we abandon it.
- $sock->close;
- }
- }
local($^W) = 0; # IO::Socket::INET can be noisy
my $sock = $self->socket_class->new(PeerAddr => $host,
@@ -33,7 +23,7 @@ sub _new_socket
LocalAddr => $self->{ua}{local_address},
Proto => 'tcp',
Timeout => $timeout,
- KeepAlive => !!$conn_cache,
+ KeepAlive => !!$self->{ua}{conn_cache},
SendTE => 1,
$self->_extra_sock_opts($host, $port),
);
@@ -104,9 +94,10 @@ sub _fixup_header
}
$h->init_header('Host' => $hhost);
- if ($proxy) {
+ if ($proxy && $url->scheme ne 'https') {
# Check the proxy URI's userinfo() for proxy credentials
- # export http_proxy="http://proxyuser:proxypass@proxyhost:port"
+ # export http_proxy="http://proxyuser:proxypass@proxyhost:port".
+ # For https only the initial CONNECT requests needs authorization.
my $p_auth = $proxy->userinfo();
if(defined $p_auth) {
require URI::Escape;
@@ -140,26 +131,80 @@ sub request
}
my $url = $request->uri;
- my($host, $port, $fullpath);
-
- # Check if we're proxy'ing
- if (defined $proxy) {
- # $proxy is a URL to an HTTP server which will proxy this request
- $host = $proxy->host;
- $port = $proxy->port;
- $fullpath = $method eq "CONNECT" ?
- ($url->host . ":" . $url->port) :
- $url->as_string;
+
+ # Proxying SSL with a http proxy needs issues a CONNECT request to build a
+ # tunnel and then upgrades the tunnel to SSL. But when doing keep-alive the
+ # https request does not need to be the first request in the connection, so
+ # we need to distinguish between
+ # - not yet connected (create socket and ssl upgrade)
+ # - connected but not inside ssl tunnel (ssl upgrade)
+ # - inside ssl tunnel to the target - once we are in the tunnel to the
+ # target we cannot only reuse the tunnel for more https requests with the
+ # same target
+
+ my $ssl_tunnel = $proxy && $url->scheme eq 'https'
+ && $url->host.":".$url->port;
+
+ my ($host,$port) = $proxy
+ ? ($proxy->host,$proxy->port)
+ : ($url->host,$url->port);
+ my $fullpath =
+ $method eq 'CONNECT' ? $url->host . ":" . $url->port :
+ $proxy && ! $ssl_tunnel ? $url->as_string :
+ do {
+ my $path = $url->path_query;
+ $path = "/$path" if $path !~m{^/};
+ $path
+ };
+
+ my $socket;
+ my $conn_cache = $self->{ua}{conn_cache};
+ my $cache_key;
+ if ( $conn_cache ) {
+ $cache_key = "$host:$port";
+ # For https we reuse the socket immediatly only if it has an established
+ # tunnel to the target. Otherwise a CONNECT request followed by an SSL
+ # upgrade need to be done first. The request itself might reuse an
+ # existing non-ssl connection to the proxy
+ $cache_key .= "!".$ssl_tunnel if $ssl_tunnel;
+ if ( $socket = $conn_cache->withdraw($self->socket_type,$cache_key)) {
+ if ($socket->can_read(0)) {
+ # if the socket is readable, then either the peer has closed the
+ # connection or there are some garbage bytes on it. In either
+ # case we abandon it.
+ $socket->close;
+ $socket = undef;
+ } # else use $socket
+ }
}
- else {
- $host = $url->host;
- $port = $url->port;
- $fullpath = $url->path_query;
- $fullpath = "/$fullpath" unless $fullpath =~ m,^/,;
+
+ if ( ! $socket && $ssl_tunnel ) {
+ my $proto_https = LWP::Protocol::create('https',$self->{ua})
+ or die "no support for scheme https found";
+
+ # only if ssl socket class is IO::Socket::SSL we can upgrade
+ # a plain socket to SSL. In case of Net::SSL we fall back to
+ # the old version
+ if ( my $upgrade_sub = $proto_https->can('_upgrade_sock')) {
+ my $response = $self->request(
+ HTTP::Request->new('CONNECT',"http://$ssl_tunnel"),
+ $proxy,
+ undef,$size,$timeout
+ );
+ $response->is_success or die
+ "establishing SSL tunnel failed: ".$response->status_line;
+ $socket = $upgrade_sub->($proto_https,
+ $response->{client_socket},$url)
+ or die "SSL upgrade failed: $@";
+ } else {
+ $socket = $proto_https->_new_socket($url->host,$url->port,$timeout);
+ }
}
- # connect to remote site
- my $socket = $self->_new_socket($host, $port, $timeout);
+ if ( ! $socket ) {
+ # connect to remote site w/o reusing established socket
+ $socket = $self->_new_socket($host, $port, $timeout );
+ }
my $http_version = "";
if (my $proto = $request->protocol) {
@@ -428,13 +473,13 @@ sub request
# keep-alive support
unless ($drop_connection) {
- if (my $conn_cache = $self->{ua}{conn_cache}) {
+ if ($cache_key) {
my %connection = map { (lc($_) => 1) }
split(/\s*,\s*/, ($response->header("Connection") || ""));
if (($peer_http_version eq "1.1" && !$connection{close}) ||
$connection{"keep-alive"})
{
- $conn_cache->deposit($self->socket_type, "$host:$port", $socket);
+ $conn_cache->deposit($self->socket_type, $cache_key, $socket);
}
}
}
View
3  lib/LWP/UserAgent.pm
@@ -346,7 +346,8 @@ sub request
)
{
my $proxy = ($code == &HTTP::Status::RC_PROXY_AUTHENTICATION_REQUIRED);
- my $ch_header = $proxy ? "Proxy-Authenticate" : "WWW-Authenticate";
+ my $ch_header = $proxy || $request->method eq 'CONNECT'
+ ? "Proxy-Authenticate" : "WWW-Authenticate";
my @challenge = $response->header($ch_header);
unless (@challenge) {
$response->header("Client-Warning" =>
Something went wrong with that request. Please try again.