net: pure Go DNS resolver hangs with ipv6 link-local address on macOS

[mattrobenolt]

What version of Go are you using (go version)?

$ go version
go version go1.18 darwin/amd64

This is also reproduced against go1.17.8, so doesn't appear to be new to go1.18.

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

macOS 12.3, Intel (amd64)

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/xxx/Library/Caches/go-build"
GOENV="/Users/xxx/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/xxx/go/pkg/mod"
GOOS="darwin"
GOPATH="/Users/xxx/go"
GOPROXY="https://proxy.golang.org/,direct"
GOROOT="/usr/local/Cellar/go/1.18/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.18/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pb/fk1_tmbn1nq32bgpd6njflcw0000gn/T/go-build3259567292=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Making a DNS lookup against an ipv6 link-local address hangs and fails on macOS when using thenative (non-CGO) resolver.

This appears to manifest within the non-CGO based DNS resolver, but I believe it's more generally applicable to the network stack itself, it's just easier to reproduce through DNS.

The simplest reproduction we could come up with was this compiled with CGO_ENABLED=0:

package main

import (
	"context"
	"fmt"
	"net"
	"os"
	"time"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	ips, err := net.DefaultResolver.LookupIP(ctx, "ip", "google.com")
	if err != nil {
		fmt.Fprintf(os.Stderr, "Could not get IPs: %v\n", err)
		os.Exit(1)
	}
	for _, ip := range ips {
		fmt.Printf("google.com. IN A %s\n", ip.String())
	}
}

Paired with an /etc/resolv.conf of:

nameserver fe80::a04e:cfff:fe2f:ad64%en0
nameserver 10.0.0.1

I've deduced this down to explicitly any link-local address (fe80::*) and not generically to ipv6. Other ipv6 addresses, both public and private network appear fine.

This behavior is especially concerning because it seems to cascade into larger macOS failures. All DNS seems to start failing on the whole system requiring ultimately a reboot to restore stability.

Given this, I believe this is also a bug within macOS that Go happens to be exploiting somehow.

I personally don't have a macOS machine to iterate on this, but we've deduced this both with customers running our CLI (https://github.com/planetscale/cli) as well as employees internally.

After a bunch of deducing, we've narrowed it down to the link-local address.

Again, when compiled with CGO_ENABLED=1, everything works correctly and is fine. The issue only occurs when compiled with CGO_ENABLED=0, triggering the pure Go DNS resolution path.

We originally thought this was related to arm64 (M1 Macs), but was able to finally reproduce it on Intel.

What did you expect to see?

Fast DNS resolution

What did you see instead?

Program hangs, eventually times out, in extreme cases, causing entire OS to become unstable.

For additional context: planetscale/discussion#181

[heschi]

cc @ianlancetaylor @neild

[dannyfallon]

We reproduced this with Go 1.18.4 and Go 1.16.5 on M1 and Intel machines running OS X 12.3-12.5 with two different Go applications. All affected users had the fe80::* IPv6 DNS address in their nameservers. The behaviour was always what the OP described as the extreme circumstances - hung process that ignores signals, eventual network disconnection and then a frozen machine needing forced reboots.

Changing DNS servers on the network connection to drop the fe80::* address resolved it. As a mitigation we also started to build OS X binaries on OS X machines with Cgo enabled to force the use of the Cgo resolver. We lost several days debugging this ourselves and when I came to report this I found this issue already. Is there any update or ETA on when investigation might take place? This seems fairly critical.

[apparentlymart]

One possible way to confound a reproduction on darwin_amd64 is if a program is being built for both darwin_amd64 and darwin_arm64 on a build worker whose native toolchain is the darwin_arm64 one. In that situation the default for CGO_ENABLED is different between the two targets: darwin_arm64 is considered to be a cross-compile and therefore CGO_ENABLED defaults to off.

I wrote some more about this in a comment on the general issue about the pure Go resolver on macOS. The short version is that if you want to reproduce it reliably it's important to explicitly set CGO_ENABLED=0 or CGO_ENABLED=1 consistently across builds for both architectures, and not rely on the default. That will then ensure that both of your builds are using the same resolver implementation.

[silvestriluca]

I'm experiencing this issue too using Terraform (which is based on GoLang).
Very very bad bug. All my network stack crashes and I'm forced to:

• force close the apps that are using nework (slack, browser, terminal...)
• Reboot mac

Check hashicorp/terraform#31467 (comment)

It is worth to get in touch with macOS network devs, because even if there is some issue in go implementation, network stack shouldn't crash so dramatically, to the point that reboot is the only option.

[heschi]

cc @golang/darwin

[pacorreia]

I'm using WSL and faced issues with this, so no restricted to Macs :(

[apparentlymart]

Hi @pacorreia,

Given that everything so far has suggested a bad interaction between the pure Go resolver and macOS, I think it would help if you could reproduce the bug template in a comment here, with similar content to the original issue comment but describing the experience on your own system, to help recognize whether you've found another instance of the same problem or if you've found a different problem with similar symptoms.

[pacorreia]

@apparentlymart Thanks for your suggestion, since this is being experienced through Terraform I will open an issue with them following the guidelines of Go project, then they will be in touch with you if necessary.

What I've found if of interested is that building a binary with CGO_ENABLED=0 go build . works, setting CGO_ENABLED=1 will produce a binary unable to solve DNS, actually forcing it to go always to IPv6 no matter if the machine has it enabled/disabled.