net/http: ProxyFromEnvironment no_proxy=localhost doesn't match empty Host

[skiptomyliu]

What version of Go are you using (go version)?

go version go1.16.6 darwin/amd64
go version go1.16.6 linux/amd64
go version go1.17.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes, also tested go 1.17.1

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/Users/dliu/src/go/bin/"
GOCACHE="/Users/dliu/Library/Caches/go-build"
GOENV="/Users/dliu/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/dliu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB="github.com/lyft/*,github.lyft.net/*"
GOOS="darwin"
GOPATH="/Users/dliu/go"
GOPRIVATE=""
GOROOT="/usr/local/Cellar/go/1.16.6/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.16.6/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/dliu/Desktop/gotest2/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/hz/phr9x2hd2z16zyn6gcs8l86m0000gp/T/go-build4278321027=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ cat proxytest.go
package main

import (
        "log"
        "google.golang.org/grpc"
)

func main() {
        _, err := grpc.Dial(":9901", grpc.WithInsecure(), grpc.WithBlock())
        if err != nil {
                log.Fatal(err)
        }
}


$ GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99   no_proxy=localhost https_proxy=localhost:10001 /usr/local/go/bin/go run proxytest.go
INFO: 2021/09/10 10:36:42 [core] parsed scheme: ""
INFO: 2021/09/10 10:36:42 [core] scheme "" not registered, fallback to default scheme
INFO: 2021/09/10 10:36:42 [core] ccResolverWrapper: sending update to cc: {[{:9901  <nil> 0 <nil>}] <nil> <nil>}
INFO: 2021/09/10 10:36:42 [core] ClientConn switching balancer to "pick_first"
INFO: 2021/09/10 10:36:42 [core] Channel switches to new LB policy "pick_first"
INFO: 2021/09/10 10:36:42 [core] Subchannel Connectivity change to CONNECTING
INFO: 2021/09/10 10:36:42 [core] Subchannel picks a new address ":9901" to connect
INFO: 2021/09/10 10:36:42 [core] pickfirstBalancer: UpdateSubConnState: 0xc000191ac0, {CONNECTING <nil>}
INFO: 2021/09/10 10:36:42 [core] Channel Connectivity change to CONNECTING
WARNING: 2021/09/10 10:36:42 [core] grpc: addrConn.createTransport failed to connect to {:9901 localhost:9901 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:10001: connect: connection refused". Reconnecting...

What did you expect to see?

Was expecting that because no_proxy=localhost is set, that the connection would obey and thus not CONNECT tunnel to the proxy. When a blank hostname is set, Go implicitly uses localhost. However, it seems I'm not able to no_proxy localhost if the hostname is set to blank. Explicitly setting to localhost:9901 resolves the issue.

What did you see instead?

The connection is attempting to use the proxy even though no_proxy=localhost.

Originally filed with grpc-go grpc/grpc-go#4719 (comment) but was directed here.

[thanm]

@neild

[1000Delta]

Hello, I read some source code about your problem, I think this problem is caused by how grpc call the golang http.ProxyFromEnvironment.

For golang httpproxy.proxyForURL, you can see that if the argument reqURL.Scheme is "https", you will get the cfg.httpsProxy.

And in grpc, if your target is :9901, grpc will call the internal function transport.mapAddress, this function will create a struct http.Request and set inner url.URL.Scheme to "https", then will call the transport.httpProxyFromEnvironment, which is a variable of golang exported function http.ProxyFromEnvironment.

More about golang httpproxy.proxyForURL, it will call the cfg.useProxy, this method will check your host and ip, this also answer the question of why explicitly setting the grpc.Dial target to localhost:9901 can resolve this issue, it matches localhost before the :. However, only setting the port like :9901 will not let cfg.useProxy return false. So you will get your httpsProxy.

You can debug and set a breakpoint to mapAddress function, you will see the call will get your https_proxy environment.

I hope this is helpful.