Join GitHub today
Added forward proxy support and included an example #1893
Overview of changes:
If there are any desired changes or updates to this PR, feel free to let me know. I won't take any offense.
Thanks, it's good to get the generic nature of the code available this way!
Ideally the example would be better off working with both HTTP and HTTPS, so it could truly be used like a general purpose forward proxy. This would require having a listener that can handle both normal and TLS encrypted TCP connections. I don't believe Vibe.d currently supports this (let me know if I'm wrong), but I have some ideas on how to add support for this.
The way to achieve this is to call
listenHTTP twice with the same request handler, but with two different
HTTPServerSettings, once with and once without TLS enabled. But shouldn't this be more or less irrelevant for the forward proxy, since HTTPS requests are usually forwarded using CONNECT requests tunneled through whichever protocol the proxy exposes?
I was under the impression that there was a high level test in the "tests" directory, but obviously it was just the TCP proxy test that I had in mind. I would approach this in a similar way, setting up one normal server, one forward/reverse proxy and then firing requests at the proxy and the normal server to compare the results (that would also test the proxy support in the HTTP client). Would be great to have a test, but since none existed up to now, I wouldn't see that as mandatory for this PR.
So I took another pass at this and finished the changes you requested I believe. I've never worked on a library before, only applications, so the API backwards compatibility shamefully never crossed my mind, but that should all be taken care of now.
I liked the ProxyMode idea, so I've implemented that. I also generalized two of the core functions but added compatibility aliases for backwards compatibility, so let me know what you think. To me, it makes more sense now that a forward proxy mode is an option.
I'm working on writing some unit tests, and I also want to do more on supporting HTTPS, and unit tests will probably help speed up and verify that. I'm not an expert by any means, but typically proxies like Charles or Burp Suite run on a single port (8080) and are capable of forwarding both HTTP and HTTPS traffic through the same port simultaneously. I don't believe this is the same as HTTP CONNECT, although that can be achieved via HTTP CONNECT. However, in my tests, enabling HTTP CONNECT did not solve that problem because my browser was trying to establish a HTTPS connection with the proxy which was failing because it was just an HTTP listener without TLS enabled. My understanding is that proxies like Charles or Burp Suite, will effectively run something like a multiplexer listener, that will detect whether a TLS connection is attempting to be established, and then dispatch the connection to either an SSL/TLS handler or just normal HTTP handler.
Firefox and most other browsers only allow configuring a single HTTP proxy to use, and don't allow specifying a proxy for TLS and a proxy for Non-TLS traffic. Perhaps I misunderstood your suggestion about calling
No you are right, that won't work on the same port without additional effort currently. I didn't check the traffic for proxy requests for quite a while, so maybe things have indeed changed. However, the question would be how a HTTPS request that simply goes through the proxy would specify its destination to the proxy and at the same time would remain securely encrypted end-to-end. What the mentioned proxies usually have to do is to generate a dummy certificate for the HTTPS site and basically act as a man in the middle. I'd still assume that they simply intercept CONNECT requests that way, but I'll double check that.
The changes look good now. The CI tests just currently fail due to the three trailing spaces in lines 59 to 61 and due to an unrelated error obviously caused by some changes in the Travis-CI setup. From my side we could merge this as a first step and add tackle the HTTPS and testing issues separately.