net: cross compiled Go has lengthy timeouts when resolving names if a DNS server is invalid

[geofffranks]

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/gfrau/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/gfrau/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/jh/z59wc4wx5cz823x6l5f81qn463wn7j/T/go-build408544351=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

We cross compiled an application on a mac, with GOOS=linux as the target. When run on target systems where there are two DNS resolvers listed in /etc/resolv.conf, but one of them is unreachable, the application takes quite a while to resolve dns (15s?). Changing the DNS resolution mechanism with GO_DEBUG doesn't appear to work, the error occurs with either the go or cgo DNS resolvers selected. As soon as we comment out the DNS server that is unreachable, the app works fine. It does not matter which one of the DNS servers is unreachable (first one listed, or second). The app is slow, and strace shows it connecting to both of them no matter what.

What did you expect to see?

I was expecting Go to use the primary DNS server first, with a 2 second timeout before using the next in the resolver list, returning the response of the first successful server.

What did you see instead?

Timeouts, and slow applications when the app was built on darwin/amd64 for a target of linux/amd64.

This problem goes away if we do a go build directly on linux/amd64 and do not cross-compile (both using CGO_ENABLED=0 and CGO_ENABLED=1). However, if the primary resolver listed is unreachable for this test case, the timeouts take over 10s for DNS resolution, rather than the expected 2s. If the secondary server is unreachable, it is never hit, as the primary resolved the name already, and no timeouts are noticed

[subbu05]

On the host box, can you try resolving the dns query using dig command? Also check if server is configured with ipv6 addresses.

[odeke-em]

Thank you filing this issue @geofffranks!

Kindly paging some net gurus @bradfitz @ianlancetaylor @mikioh

[geofffranks]

The server is not configured with ipv6 addresses. Dig works fine when the first server is available and the second is unavailable. When I reverse the order of servers in /etc/resolv.conf, dig times out for 2 seconds, then queries the next server and resolves the record

[bradfitz]

It's GODEBUG, not GO_DEBUG, and that mechanism only lets you pick between cgo and non-cgo if the cgo version is even compiled in to the binary, which it isn't by default if you cross-compiled from a Mac without a Linux toolchain on the Mac.

I think the cross-compiling part of this is a red herring. You can probably reproduce the same on Linux only setting GODEBUG. Could you post the output of your test setting first GODEBUG=netdns=cgo+1 and then GODEBUG=netdns=go+1?

[geofffranks]

I think GO_DEBUG was a typo when I created the issue. Just ran through the tests again:

cross-compiled with GODEBUG=netdns=cgo+1 - says it's using Go's DNS resolver, and we see the issue
cross-compiled with GODEBUG=netdns=go+1 - says it's using Go's DNS resolver, and we see the issue

statically compiled from linux with GODEBUG=netdns=cgo+1 - says it's using Go's DNS resolver, and is fast
statically compiled from linux with GODEBUG=netdns=go+1 - says it's using Go's DNS resolver, and is fast

dynamically compiled from linux with GODEBUG=netdns=cgo+1 - says it's using cgo DNS resolver, and is fast
dynamically compiled from linux with GODEBUG=netdns=go+1 - says it's using Go's DNS resolver, and is fast

[seankhliao]

I think there've been enough changes that if someone else sees this, they should file a new issue.