Support HTTPS proxying #34

Closed
robfletcher opened this Issue Sep 7, 2011 · 17 comments

Projects

None yet

4 participants

@robfletcher

No description provided.

@robfletcher robfletcher pushed a commit that referenced this issue Oct 7, 2011
Rob Fletcher #34: failing test for intercepting HTTPS 24247a8
@robfletcher robfletcher was assigned Oct 7, 2011
@lhotari

This will be a nice feature. Looking forward to using this in testing Salesforce REST API interactions.

@robfletcher

I need to figure out how to pass the SSL credentials from the Betamax proxy to the target host. I have a test working with the client connecting to the proxy via HTTPS but in that case I'm just faking the certificate & skipping authentication.

@lhotari

OWASP webscarab-ng source code for proxying HTTPS : https://code.google.com/p/webscarab-ng/source/browse/trunk/src/main/java/org/owasp/webscarab/plugins/proxy/Proxy.java (Warning: GNU GPL v2 source code)

@robfletcher

You sure about the license? According to this page the project is CC licensed

@jwagenleitner

The content of that media wiki page is CC licensed, the code looks to be licensed under gpl v2.

@lhotari

License of Webscarab-ng is mentioned here: https://code.google.com/p/webscarab-ng/ and also in the header of http://owasp.cvs.sourceforge.net/viewvc/owasp/webscarab/src/org/owasp/webscarab/plugin/proxy/ConnectionHandler.java?revision=HEAD&view=markup .

They have a licensing issue themselves as Apache HttpClient code is used in GPL code. I believe in "pure" GPL v2, all "linked" code has to be GPL .

@robfletcher

I've been doing some experimenting using Jetty's ProxyServlet overridden to simply set a Via header on request & response (see ad5d60c) but I can't get further than the proxy server rejecting the HTTPS request as it thinks it's plaintext. Not sure if my test is too naive but the same thing happens if I configure a browser to use the proxy & try to connect to an HTTPS URL.

@lhotari

https proxy protocol is decribed in this "ietf internet draft" document: Tunneling TCP based protocols through Web proxy servers .

Quote from document describing the data pipelining (example of proxy authentication):

 CLIENT -> SERVER                        SERVER -> CLIENT
 --------------------------------------  -----------------------------------
 CONNECT home.netscape.com:443 HTTP/1.0
 User-agent: Mozilla/4.0
 <<< empty line >>>
                                         HTTP/1.0 407 Proxy auth required
                                         Proxy-agent: Netscape-Proxy/1.1
                                         Proxy-authenticate: ...
                                         <<< empty line >>>
 CONNECT home.netscape.com:443 HTTP/1.0
 User-agent: Mozilla/4.0
 Proxy-authorization: ...
 <<< empty line >>>
                                         HTTP/1.0 200 Connection established
                                         Proxy-agent: Netscape-Proxy/1.1
                                         <<< empty line >>>
              <<< data tunneling to both directions begins >>>

RFC2817 mentions:
"The CONNECT method was originally described in a Work in Progress
titled, "Tunneling TCP based protocols through Web proxy servers",
[8] by Ari Luotonen of Netscape Communications Corporation. It was
widely implemented by HTTP proxies, but was never made a part of any
IETF Standards Track document. The method name CONNECT was reserved,
but not defined in [1]."

"Since HTTP/1.1 [1] defines Upgrade as a hop-by-hop mechanism, this
memo also documents the HTTP CONNECT method for establishing end-to-
end tunnels across HTTP proxies."


Webscarab (links to source code in my previous comments) contains a working implementation of a https MITM proxying, only problem is that the code is GPL licensed.

@robfletcher

Yeah, I was hoping to be able to implement this without rewriting the whole proxy layer. It's currently based on Jetty but it seems like the Jetty ProxyServlet may not handle CONNECT requests (several people seem to have got stuck at the same point as me).

If I do change the implementation to use a bespoke proxy I don't want to be too dependent on using Webscarab as an example because of the licensing issue.

@robfletcher

Note to self: Peter Ledbrook has some example code attached to this bug report that might help ensure I'm doing the client connection correctly in the test: http://jira.codehaus.org/browse/GMOD-266

@ikikko

Betamax is a great product!

Supporting https handling becomes better than now.
I'm expecting implementing this feature.

Thanks.

@lhotari

I got https support working: lhotari@aee5d23
It's still a proof-of-concept that was forked off the https-proxy-servlet branch.

Adding support was actually quite easy since Jetty contained a non-blocking handler for the CONNECT method.
I just overrided the ConnectHandler to always connect to the proxy server listening with ssl-support (port 5001).

The CONNECT method "wiring" was actually the only missing piece in the solution. Other parts of the solution are just by-passing SSL certificate checks (and host name checks). By-passing SSL checks is activated by a new boolean field in Recorder called sslSupport.

I won't have time to continue working on this before August... You are free to pick the changes you want from what I have made. I didn't have time to clean up the commit.

@robfletcher

That's absolutely fantastic. I've been working on other things since gr8conf and haven't had a chance to start on this myself yet. I'll definitely pick up your branch & integrate it. I really appreciate the effort, thanks.

@lhotari

btw. An alternative approach to by-pass SSL certificate checks would be to generate a certificate and sign it on-demand for each host / domain (wildcard cert). This could be done in the CONNECT method's handler and startup a SSL listener on a new port for each cert. In this approach it would only require to add Betamax's CA to trusted CAs (using a custom trustStore by setting the system property javax.net.ssl.trustStore).

A commercial product called Bluecoat uses this type of approach (dynamicly generated & signed certs) to filter SSL traffic in corporate networks (it's a transparent proxy).

@robfletcher

I've merged this in to a new branch based on master bdbb47310277c0ead77d354cc62c42a9d6fd5ff2 (i.e. not using a servlet implementation) and it's not quite working. I'll investigate tomorrow.

From what you've said I don't think there's any reason it should need the servlet, a plain handler should be adequate.

@robfletcher robfletcher pushed a commit that closed this issue Aug 8, 2012
@lhotari lhotari HTTPS support. Fixes #34 021d82f
@robfletcher

My mistake I missed one simple thing in the merge. HTTPS support is now in master!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment