Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: provide better error message when proxy is misconfigured #29809

Open
jtfogarty opened this issue Jan 18, 2019 · 13 comments

Comments

Projects
None yet
5 participants
@jtfogarty
Copy link

commented Jan 18, 2019

What version of Go are you using (go version)?

$ go version go1.11 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build200009662=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Doing a go get fails behind a corporate proxy if environment variables are set as below; HTTP_PROXY=http://proxyIp:proxyPort
it succeeds if set as
HTTP_PROXY=proxyIp:proxyPort
this is true for HTTP or HTTPS variables This is also true for any go project using net/http
I'm seeing this issue with Docker versions greater than 17.09, Jenkins-x, goBuffalo
Older versions of go (specifically 1.8.3 - the only version I tested) do not behave this way
Docker version 17.09 and lower do not behave this way.

What did you expect to see?

I expect it to pull through the proxy server with the below environment variables set;
HTTP_PROXY=http://proxyIp:proxyPort
HTTPS_PROXY=https://proxyIp:proxyPort

What did you see instead?

Process fails to get through the proxy.

@bradfitz bradfitz changed the title net/http Proxy Failure cmd/go: http proxy failure Jan 18, 2019

@bradfitz bradfitz added this to the Go1.13 milestone Jan 18, 2019

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

/cc @bcmills for cmd/go.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

What did you do?

Doing a go get fails […]

More detail, please.

  • Fails in what way? (Does the error message indicate what went wrong?)

  • You appear to be using an insecure (HTTP-only) proxy. Did you pass the -insecure flag to go get?

@jtfogarty

This comment has been minimized.

Copy link
Author

commented Jan 22, 2019

With variables as below;

http_proxy=http://proxyIp:proxyPort
https_proxy=https://proxyIp:proxyPort

The error is;

go get golang.org/x/blog/content
package golang.org/x/blog/content: unrecognized import path "golang.org/x/blog/content" (https fetch: Get https://golang.org/x/blog/content?go-get=1: proxyconnect tcp: EOF)

With variables as;

http_proxy=proxyIp:proxyPort
https_proxy=proxyIp:proxyPort

The below succeeds
go get golang.org/x/blog/content

Not sure about your point on HTTP only. The go get command does not specify either.

@bcmills bcmills removed the WaitingForInfo label Jan 22, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

What happens if you run curl https://golang.org/x/blog/content?go-get=1 in the same environment?

@jtfogarty

This comment has been minimized.

Copy link
Author

commented Jan 22, 2019

Works fine when variables are set to;

http_proxy=http://proxyIp:proxyPort
https_proxy=https://proxyIp:proxyPort
curl https://golang.org/x/blog/content?go-get=1
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="golang.org/x/blog git https://go.googlesource.com/blog">
<meta name="go-source" content="golang.org/x/blog https://github.com/golang/blog/ https://github.com/golang/blog/tree/master{/dir} https://github.com/golang/blog/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://godoc.org/golang.org/x/blog/content">
</head>
<body>
Nothing to see here; <a href="https://godoc.org/golang.org/x/blog/content">move along</a>.
</body>
</html>

@bcmills bcmills self-assigned this Feb 12, 2019

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

@jtfogarty, sorry for the delay here. I was actively ignoring cmd/go issues, the cmd/go owners deal with them, but since you asked me about this at GCP Next I'm taking a look.

There's a difference between:

https_proxy=https://proxyIp:proxyPort

and

https_proxy=proxyIp:proxyPort

The former proxies https requests (e.g. those from cmd/go get) using CONNECT requests to an https proxy server at the given ip:port.

The latter proxies those https requests still using CONNECT requests but to an http proxy server at the given ip:port.

The notable difference is that you almost certainly don't have a valid https certificate for your proxy server if you're connecting via an IP address and not via a hostname.

There are a few rare exceptions (like https://1.1.1.1/) but generally IP addresses don't get TLS certs, and only hostnames do.

Admittedly the error message above kinda sucks:

(https fetch: Get https://golang.org/x/blog/content?go-get=1: proxyconnect tcp: EOF)

It looks like your proxy is immediately hanging up on you (cmd/go) when it's not presented any SNI value (a hostname) in the TLS ClientHello.

I get a slightly better error if I use a dummy IP address for a proxy:

bradfitz@go:~$ GOROOT=$HOME/go GODEBUG=http2debug GO111MODULE=off HTTPS_PROXY=https://212.47.227.56:443 go get -u -v bradfitz.com/foo
Fetching https://bradfitz.com/foo?go-get=1
https fetch failed: Get https://bradfitz.com/foo?go-get=1: proxyconnect tcp: remote error: tls: internal error
package bradfitz.com/foo: unrecognized import path "bradfitz.com/foo" (https fetch: Get https://bradfitz.com/foo?go-get=1: proxyconnect tcp: remote error: tls: internal error)

There at least I see (proxyconnect tcp: remote error: tls: internal error)

But perhaps we could detect this case (an https proxy without a hostname) and upgrade the error message to something more helpful with a hint ("you need a hostname probably") along with the original error.

Can you confirm that's your issue?

@bradfitz bradfitz changed the title cmd/go: http proxy failure cmd/go: bad error message when using https proxy by IP instead of hostname Apr 29, 2019

@jtfogarty

This comment has been minimized.

Copy link
Author

commented May 4, 2019

Hi @bradfitz We are connecting via a hostname. Sorry for the confusion.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 7, 2019

Maybe your proxy doesn't listen on https.

What does curl say when you do curl -I https://ip:port ?

@jtfogarty

This comment has been minimized.

Copy link
Author

commented May 7, 2019

This is only an issue for apps written in go. I was installing a server with helm today. I was able to execute the following https_proxy=https://ServerName:Port wget https://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz .
and it downloaded fine. I attempted to use install a helm chart and I ran into the same issue. The fix was to execute https_proxy=ServerName:Port helm fetch https://charts.rook.io/release

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 7, 2019

What does curl say when you do curl -I https://ip:port ?

@jtfogarty

This comment has been minimized.

Copy link
Author

commented May 7, 2019

Not sure what ip:port you want me to curl to but if I curl to github.com with the https_proxy variable set, I get the following;

curl -I https://www.github.com
HTTP/1.1 200 Connection established

HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://github.com/

Using the ip for github

curl -I https://192.30.253.112
HTTP/1.1 403 Forbidden
Mime-Version: 1.0
Date: Tue, 07 May 2019 12:14:14 CDT
Via: 1.1 InternalServerName:Port (Cisco-WSA/10.5.2-061)
Content-Type: text/html
Connection: keep-alive
Proxy-Connection: keep-alive
Content-Length: 2844

curl: (56) Received HTTP code 403 from proxy after CONNECT
@bradfitz

This comment has been minimized.

Copy link
Member

commented May 7, 2019

So I bet the issue here is that Go just can't validate your InternalServerName:Port's TLS cert.

I'm afraid I can't really debug further by talking in the abstract. I'd need more verbose logs that you'd inevitably need to redact too much to be useful.

If you want to privately email me the full details, I'm bradfitz at golang.org.

@bradfitz bradfitz removed the GoCommand label May 8, 2019

@bradfitz bradfitz changed the title cmd/go: bad error message when using https proxy by IP instead of hostname net/http: provide better error message when proxy is misconfigured May 8, 2019

@bradfitz bradfitz modified the milestones: Go1.13, Unplanned May 8, 2019

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 8, 2019

This ended up being an environment configuration problem.

User had:

https_proxy=https://foobar:1234
http_proxy=http://foobar:1234

Where foobar:1234 was only an HTTP server, not https. (It could proxy an https request via CONNECT, but it did not speak https itself)

openssl s_client and curl confirmed that foobar:1234 only spoke http.

Go could perhaps provide a better error message than EOF when this occurs, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.