Add back Man in the Middle Support

[oxtoacart]

Support for man in the middle (MITM) on https connections was lost as part of the 1.0 refactor. We should add this back in.

There are two levels to which we can take it:

• Rudimentary (as before). Proxy simply uses its own self-signed certificate, which does not match the domain of the server that the client requested. Typical clients will not accept this certificate.
• Full. Proxy actually acts as a certificate authority and issues certificates on the fly. As long as the client trusts LittleProxy's cert, the client will then implicitly trust the issued certs. See here for an explanation of how this is done by mitmproxy.

[codersbrew]

What was the main reason for it being removed? Did it not work appropriately?

[robfletcher]

This would be tremendously useful to me. I have a library called Betamax that is for unit testing components that make HTTP connections. I'm in the process of porting it from running on Jetty to Netty. LittleProxy looks like it could save me an awful lot of effort as I wouldn't have to implement my own proxy just my own implementation of HttpFilters.

My existing Jetty-based implementation can do MITM on HTTPS connections (via self-signed cert so doesn't work with clients that validate the cert chain). I think this issue would be the only blocker to me achieving feature parity using LittleProxy.

[oxtoacart]

@codersbrew The internals of LittleProxy have changed a fair amount and we decided to stabilize and release without MITM in the interest of time. Getting back at least the rudimentary MITM should in theory not be too hard, as the proxy server already has TLS support for proxy chaining purposes.

@robfletcher Betamax looks really cool!

[oxtoacart]

@robfletcher @codersbrew

I've added back MITM and it's available on the mitm branch.

I anticipate that it'll be included in the next release

There's a new interface, MitmManager, whose use is demonstrated in MitmProxyTest. For basic MITM support, you can use the SelfSignedMitmManager in org.littleshoot.proxy.extras.

I've structured MitmManager such that if and when someone is ready to implement impersonation using dynamically generated certificates, that should be doable by implementing an appropriate MitmManager.

[codersbrew]

Thanks @oxtoacart . I may have a need for dynamically generated client certs, so if I get something working I'll push it back to the branch.

[robfletcher]

That's awesome. I'll try to give it a try over the next couple of days. I have Betamax running using LittleProxy for HTTP traffic already.

[oxtoacart]

@robfletcher Glad to hear you've got it running already

[robfletcher]

@oxtoacart the callbacks in HttpFilters give me exactly what I need. I don't have everything perfect but it's a good enough proof-of-concept that if HTTPS works too I'll definitely port everything over to LittleProxy

[oxtoacart]

@codersbrew I'm looking forward to your pull request :) Here's a project that does impersonation but with blocking sockets:

http://crypto.stanford.edu/ssl-mitm/

That may give you some good ideas.

For unit testing, I highly recommend extending BaseProxyTest similarly to how MitmProxyTest does. This will give pretty good coverage of some common scenarios.

[codersbrew]

Hmm am I doing something wrong.. I started up a basic MITM proxy server and then proceeded to use Firefox or what have you to browse https websites and get the following error:

sun.security.validator.ValidatorException: No trusted certificate found

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

[oxtoacart]

@codersbrew Sorry about that. It looked good in unit test land but not the real world. Problem was that it didn't trust the server. If you pull the latest from that branch, it should work for you now.

[robfletcher]

A few things I've found whilst trying to use this for Betamax:

• The tunnelled request passes through requestPre and requestPost on a filter but not responsePre and responsePost.
• The CONNECT request passes through all the filter methods but obviously then I can't see the detail of the tunnelled request (the body of a POST request for example).
• The tunnelled request only has a relative URL so I have to reconstruct the full URL using the Host header.
• One of my tests tries to hit https://httpbin.org/get but the CONNECT fails with a HTTP 502. This was a test I'd developed for my Jetty-based MITM functionality where it did work.

Is this behavior what you'd expect?

P.S. I don't mean to hijack this issue – happy to move this discussion elsewhere if it would be more appropriate.

[oxtoacart]

@robfletcher thanks for taking a look and for the detailed feedback.

• The fact that the tunneled request only includes the resource portion of the url and not the host is expected. I don't know why exactly that logic is in LittleProxy, but I assume there's a reason for it so I'm leaving it alone for now.
• The issue with httpbin is covered under bug MITM proxying to hosts whose names begin with "http" is broken #92 and is fixed.
• I don't see any problem filtering. To convince myself that it's working correctly, I updated the MitmProxyTest with conditions that check request and response pre and post filters. Keep in mind that unless you override getMaximumRequestBufferSizeInBytes() and/or getMaximumResponseBufferSizeInBytes() on your HttpFilters implementation, requests and responses will be streamed through the filters as they are chunked by the browser, Netty and the server. This means that often you'll see an HttpRequest with headers and no content followed by one or more HttpContent objects with the actual content. Likewise for responses you may see an HttpResponse with headers and no content, followed by one or more HttpContent objects.

[robfletcher]

If I'm reading your test right there it looks like you're just testing if the request goes via requestPre and requestPost. What I'm seeing is this flow:

• CONNECT goes thru requestPre
• CONNECT goes thru requestPost
• tunnelled GET goes thru requestPre
• tunnelled GET goes thru requestPost
• CONNECT goes thru responsePre
• CONNECT goes thru responsePost

Each of these steps is happening multiple times as the chunked content goes through (which I'm handling). But the tunnelled response never goes through responsePre and responsePost.

Maybe that's correct. If so I will need to come up with a way to reconcile the tunnelled request against the CONNECT response which may be tricky as it's not the same filter instance.

[oxtoacart]

Ahh, I see the problem. There is no CONNECT response from the server. The CONNECT terminates at the proxy, which responds to the client with a 200. The HttpResponse that we see in responsePre and responsePost is in fact to the GET request. Unfortunately, the filters instance is still the one from the original CONNECT. I'll take a look.

[robfletcher]

Yes, that makes sense. For my purposes I really need to be able to tie the request to the response. I want to just ignore the CONNECT and deal with the tunneled request.

On Tue, Sep 24, 2013 at 5:19 PM, oxtoacart notifications@github.com
wrote:

Ahh, I see the problem. There is no CONNECT response from the server. The CONNECT terminates at the proxy, which responds to the client with a 200. The HttpResponse that we see in responsePre and responsePost is in fact to the GET request. Unfortunately, the filters instance is still the one from the original CONNECT. I'll take a look.

Reply to this email directly or view it on GitHub:
#79 (comment)

[oxtoacart]

@robfletcher Okay, fixed under #93. Thanks for finding this!

[robfletcher]

@oxtoacart This is working really well for me now. Thanks for all your hard work getting this all working.

[oxtoacart]

@robfletcher Really glad to hear that it's working for you now! Let us know when you've completed the switch to LittleProxy and we'll happily announce it. I need to get some sort of list of "projects that use LittleProxy" going.