Upstream HTTPS

[JakeAustwick]

I also posted this on Discourse, but I'm posting it here because I believe it to be a bug.

I have my script changing the upstream proxy for HTTP connections, but it seems to fail for HTTPS connections. Here's a basic stripped down example.

https://gist.github.com/JakeAustwick/c3116d86f4483d1acede4420f02efb72

The script works fine for http://api.ipify.org/, but fails for https://api.ipify.org/. With HTTPS I see the following, which shows the upstream proxy wasn't changed.

Server connection to default_proxy:8888 failed: Error connecting to "default_proxy": [Errno 8] nodename nor servname provided, or not known

Running Command:
mitmdump -s utilities/proxy_redirects.py -U http://default_proxy:8888

Platform / Library Version ( I also tested on Ubuntu):

mitmdump --version
Mitmproxy version: 2.0.1 (release version) 
Python version: 3.6.1
Platform: Darwin-14.5.0-x86_64-i386-64bit
SSL version: OpenSSL 1.1.0e 16 Feb 2017
Mac version: 10.10.5 ('', '', '') x86_64

[Kriechi]

The error messages indicates that it can't resolve default_proxy. Is this a valid DNS name in your environment? Can you try and directly use the IP, e.g., -U http://192.168.1.1:8888?

[JakeAustwick]

@Kriechi Sorry, maybe I didn't explain well enough. The issue is that it shouldn't be using default_proxy at all. Look at the code, the proxy should always get changed to a different one. So the one I'm passing on the command line should be irrelevant.

[mhils]

the proxy should always get changed to a different one.

This is not true for the initial CONNECT request: https://gist.github.com/JakeAustwick/c3116d86f4483d1acede4420f02efb72#file-proxy-py-L21-L22

[JakeAustwick]

@mhils I tried it without that line too; same issue.

I actually only have those lines because they were from an online example. Is there some kind of technical reason why I need those two lines, and the CONNECT can't go through one of my upstream proxies?

[mhils]

The example has a comment explaining this: https://github.com/mitmproxy/mitmproxy/blob/master/examples/complex/change_upstream_proxy.py

[JakeAustwick]

@mhils The comment suggests that doing this is only required if I need access to the request data to make the proxy choice, right?

I have the same issue with the following code though, with the CONNECT line removed (because I'm just using a random proxy, no request data needed).

https://gist.github.com/JakeAustwick/af0bc5a5052b2ff8ec423a9c6ca49b97

[mhils]

@mhils The comment suggests that doing this is only required if I need access to the request data to make the proxy choice, right?

If you don't redirect before the CONNECT request, said request will be sent to the default server. Whatsoever, looking at the code, this won't fix your problem. We always establish a connection with the default host here. I'm not exactly sure why that is the case, just calling self.set_server(address) might enough at this point. Not sure, maybe that breaks some hidden invariants. Feel free to try that out and submit a PR if it works for you.