Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mojo::UserAgent is unable to make HTTPS requests via a proxy #468

Closed
evoyy opened this issue Mar 17, 2013 · 11 comments
Closed

Mojo::UserAgent is unable to make HTTPS requests via a proxy #468

evoyy opened this issue Mar 17, 2013 · 11 comments

Comments

@evoyy
Copy link

evoyy commented Mar 17, 2013

Printing the request methods inside a start event, Mojo::UserAgent appears to make 2 requests: first a GET, and then a CONNECT. However, only the CONNECT is reported when debugging output is enabled.

I can make HTTPS requests through the proxy using Firefox or LWP::UserAgent, which makes me think there is something wrong with Mojo::UserAgent.

Test code using Mojo::UserAgent:

use Mojo::UserAgent;

my $ua = Mojo::UserAgent->new;

$ua->on(
    start => sub {
        my ($ua, $tx) = @_;
        my $method = $tx->req->method;
        my $url    = $tx->req->url->to_abs;
        print "$method $url\n";
    }
);

$ua->https_proxy("http://$user:$pass\@$host:$port");

print $ua->get("https://github.com")->error;

Output:

GET https://github.com
CONNECT https://github.com
Proxy connection failed

The equivalent using LWP::UserAgent:

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

$ua->add_handler(
    request_send => sub {
        my ($request) = @_;
        my $method = $request->method;
        my $url    = $request->uri;
        print "$method $url\n";
        return;
    }
);

$ua->max_redirect(1);
$ua->proxy('https', "http://$user:$pass\@$host:$port");

print $ua->get("https://github.com")->content;

Output:

GET https://github.com
<!DOCTYPE html>
<html>
    <head .......
    etc...
@kraih
Copy link
Member

kraih commented Mar 17, 2013

This is not a bug, you can't perform HTTPS requests through an HTTP proxy without CONNECT.

@kraih kraih closed this as completed Mar 17, 2013
@evoyy
Copy link
Author

evoyy commented Mar 17, 2013

If so, why does it work with LWP::UserAgent? I use the same HTTP proxy in that test case.

In the Mojo::UA debugging error stream I can see Squid complaining that CONNECT is not permitted.

@kraih
Copy link
Member

kraih commented Mar 17, 2013

Only you can find that out, perhaps it's using a SOCKS proxy instead.

@evoyy
Copy link
Author

evoyy commented Mar 17, 2013

Ok thanks.

@evoyy
Copy link
Author

evoyy commented Mar 17, 2013

After some further investigation, I think there is an issue here.

Firstly, I made a mistake: my proxy server does support CONNECT. Sorry for the confusion.

These are the request/response headers sent/received between Mojo::UA and the proxy:.

CONNECT www.google.co.uk:443 HTTP/1.1
User-Agent: Mojolicious (Perl)
Connection: keep-alive
Proxy-Authorization: Basic b2xldmk6WnRXVDZod0s=
Accept-Encoding: gzip
Content-Length: 0
Host: 50.117.76.215:29842

HTTP/1.1 409 Conflict
Server: squid/3.2.0.13
Mime-Version: 1.0
Date: Sun, 17 Mar 2013 11:53:07 GMT
Content-Type: text/html
Content-Length: 4085
X-Squid-Error: ERR_INVALID_REQ 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from arpagic
Connection: keep-alive

Note that the Host header is set to the proxy IP, instead of the remote host. I think this is not right.

If I force the Host header like this:

use Mojo::UserAgent;
use Data::Dump;

my $ua = Mojo::UserAgent->new;

$ua->on(start => sub {
    my ($ua, $tx) = @_;
    $tx->req->headers->remove('Host')->add(Host => 'www.google.co.uk');
});

$ua->https_proxy('http://xxxxxx:xxxxxx@xx.xx.xx.xx:xxxx');

dd $ua->get('https://www.google.co.uk/')->error;
dd $ua->get('https://www.google.co.uk/')->res->body;

The connection is successful:

CONNECT www.google.co.uk:443 HTTP/1.1
User-Agent: Mojolicious (Perl)
Connection: keep-alive
Proxy-Authorization: Basic b2xldmk6WnRXVDZod0s=
Accept-Encoding: gzip
Content-Length: 0
Host: www.google.co.uk

HTTP/1.1 200 Connection established

However, no data is sent or received over then connection. I think the TLS handshake does not complete.

For reference, this is how Firefox does it:

CONNECT www.google.co.uk:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:19.0) Gecko/20100101 Firefox/19.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: www.google.co.uk
Proxy-Authorization: Basic b2xldmk6WnRXVDZod0s=

HTTP/1.1 200 Connection established

and this is how LWP::UserAgent does it (note that LWP::UA does not perform end-to-end TLS using CONNECT):

GET https://www.google.co.uk/ HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: www.google.co.uk
Proxy-Authorization: Basic b2xldmk6WnRXVDZod0s=
User-Agent: libwww-perl/6.04

HTTP/1.1 200 OK

@dougwilson
Copy link
Contributor

Yes, it looks like it may be a problem in Mojolicious to me. The Host header should be set to match the host in CONNECT. In fact, what you are seeing here is squid's header forgery detection which will return 409 Conflict when the Host header does not match. I'm sure other proxies happily pass the Host header along, though that would just make Mojolicious broken when making request over a proxy to a site that uses Host for virtual servers anyhow. This may be why it wasn't caught before (because other proxies will still pass the request on).

@kraih kraih reopened this Mar 17, 2013
@kraih kraih closed this as completed in 99dff6e Mar 17, 2013
@kraih
Copy link
Member

kraih commented Mar 17, 2013

Thanks, should be fixed.

@evoyy
Copy link
Author

evoyy commented Mar 17, 2013

The CONNECT is now being established. But then the client immediately closes the connection.

use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
$ua->https_proxy($http_proxy);
print $ua->get('https://www.google.co.uk/')->res->body;

Running with MOJO_USERAGENT_DEBUG=1 produces this:

-- Blocking request (https://www.google.co.uk/)
-- Connect (https:www.google.co.uk:443)
-- Client >>> Server (https://www.google.co.uk/)
CONNECT www.google.co.uk:443 HTTP/1.1
User-Agent: Mojolicious (Perl)
Connection: keep-alive
Proxy-Authorization: Basic b2xldmk6WnRXVDZod0s=
Accept-Encoding: gzip
Content-Length: 0
Host: www.google.co.uk


-- Client <<< Server (https://www.google.co.uk/)
HTTP/1.1 200 Connection established

In Wireshark I can see the next packet sent is a FIN from the client, instead of inializing SSL.

I have the latest IO::Socket::SSL (1.84).
I have tried this on 2 servers running Ubuntu 12.04.1, one 32-bit, one 64-bit.
I have tried using several different HTTP proxies, with and without authentication.

If you would like to try, here is a fast public HTTP proxy that I have been using for tests. I can't give my private proxy here but I can share over email.

http://173.193.200.199:8080

How can I debug this further? Thanks.

@kraih kraih reopened this Mar 17, 2013
@kraih kraih closed this as completed in 6f002f6 Mar 17, 2013
@kraih
Copy link
Member

kraih commented Mar 17, 2013

Thanks, this was a rather odd bug, it should be fixed now, but i'm not sure yet how to test it.

@evoyy
Copy link
Author

evoyy commented Mar 17, 2013

I can confirm it is fixed for me. Mojolicious support is truly awesome. Thank you!

@kraih
Copy link
Member

kraih commented Mar 17, 2013

And it's also tested now, so this shouldn't happen again. 0a591b3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kraih @dougwilson @evoyy