net/http, x/net/http/httpproxy: http_proxy is being used for https requests

[tkopecek]

Go handles http_proxy/https_proxy/no_proxy in non-standard way. According to source comment http_proxy is used even for https urls. This is counterintuitive and not-working if it is not overriden.

My usecase is that I've local squid running with http_proxy exported. Nevertheless, squid is configured to handle also https but it is not propagated because it is using untrusted self-signed certificate. Go tries to connect to https via the proxy and fails with the reasonable certificate signed by unknown authority message. But at first place it shouldn't have used that proxy at all.

Code failing on this is referenced here

[gopherbot]

Change https://golang.org/cl/249440 mentions this issue: http/httpproxy: match http scheme when selecting http_proxy

[bradfitz]

That CL's code seems fine, @fraenkel, but it's a behavior change away from the documented behavior (and changes the documented behavior), so the decision on whether to do this should be made intentionally.

I'm pretty sure the old behavior (of HTTP_PROXY also applying to "https" scheme URLs when HTTPS_PROXY was not present) was intentional but I don't have the time to go digging through git history to figure out whose behavior we were copying at the time, but I thought we were.

/cc @rsc who might also remember and should decide who makes this decision.

One can tunnel any protocol through an HTTP proxy: https://wiki.squid-cache.org/Features/HTTPS#CONNECT_tunnel

[fraenkel]

While CONNECT is the mechanism used, this is about the environment variables. all_proxy was meant to be the catch all but that is not implemented.

[bradfitz]

@neild, this should probably go in now-ish with release notes for Go 1.16 so we can see if anybody is surprised during the rcs/betas.

[neild]

Googling for http_proxy and https_proxy turns up mostly references to curl and wget in the first page of results, and I've confirmed that curl/wget don't use the value of the http_proxy environment variable for HTTPS connections. So while on one hand this is a behavioral change, it appears to be a change that brings http/httpproxy's behavior in line with the most common definition of these environment variables.

Tentatively SGTM.

[dmitshur]

Closing as fixed. (We still need to document this change in Go 1.16 release notes, but that's tracked separately.)

The code change has happened in x/net in CL 249440, which got pulled into the standard library in CL 266898.