net: Dial does not respond to quickly-broken IPv6 connections by falling back to IPv4

[oakad]

Go version

go version go1.22.4 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOOS='darwin'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.4/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.4/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.4'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/q9/qcgtwgsj0y72gr01_djqgmyw0000gq/T/go-build1826860007=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Trying to fetch a random module (all break the same):

% go get nhooyr.io/websocket
go package net: confVal.netCgo = false netGo = false
go package net: using cgo DNS resolver
go package net: hostLookupOrder(proxy.golang.org) = cgo
go: module nhooyr.io/websocket: Get "https://proxy.golang.org/nhooyr.io/websocket/@v/list": write tcp [fe80::bed0:74ff:fe64:598e%utun4]:56330->[2a00:1450:4003:80c::2011]:443: write: socket is not connected

Machine has IPv6 disabled:

% dig proxy.golang.org
; <<>> DiG 9.10.6 <<>> proxy.golang.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53713
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;proxy.golang.org. IN A

;; ANSWER SECTION:
proxy.golang.org. 46 IN A 142.250.184.177

;; Query time: 366 msec
;; SERVER: 10.20.141.5#53(10.20.141.5)
;; WHEN: Fri Jun 28 21:47:33 AEST 2024
;; MSG SIZE rcvd: 61

What did you see happen?

Go get is unable to fetch a module because it's using a wrong proxy address.

What did you expect to see?

Go get should be able to fetch a module.

[gabyhelp]

Related Issues

• net: PreferGo DNS resolver failure #56107 (closed)
• net: pure Go DNS resolver hangs with ipv6 link-local address on macOS #52839
• net: Error when running concurrent DNS queries using cgo on MacOS #30521 (closed)
• Golang net resolver error messages are incorrect. #43702 (closed)
• cmd/go: GOPROXY should not override GONOPROXY/GOPRIVATE #56477 (closed)
• net: resolver error messages indicate wrong dns server when using custom resolver #43703
• net: "127.0.0.1:" should be an invalid tcp4 address for ResolveTCPAddr #56144 (closed)
• x/net: MacOS lookupIP resolves domains different if CGO_ENABLED flag used. #51753 (closed)
• cmd/go: get not using GOPROXY and bypassing operation to git #39835 (closed)
• net: DNS Lookups Very Slow (netgo) #49253 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

That's not proof that IPv6 is disabled, only that dig defaults to an A (IPv4) query.

[oakad]

It is, I assure you.
However, there's a caveat: we have a Cisco VPN which insists on advertising an additional resolver; the said resolver is able to resolve AAAA record ("Request A records, Request AAAA records"). Basically, I've got this config:

DNS configuration

resolver #1
search domain[0] : heh
nameserver[0] : heh
nameserver[1] : heh
flags : Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
order : 1

DNS configuration (for scoped queries)
resolver #1
nameserver[0] : heh
nameserver[1] : heh
if_index : 15 (en0)
flags : Scoped, Request A records
reach : 0x00000002 (Reachable)

resolver #2
search domain[0] : heh
nameserver[0] : heh
nameserver[1] : heh
if_index : 23 (utun4)
flags : Scoped, Request A records, Request AAAA records
reach : 0x00000002 (Reachable)
order : 1

Still, go should not pick the AAAA address. Or, at least, it should not do so unconditionally, because I don't think our setup is uniquely broken. :-)

[mateusz834]

From the output it is clear that the cgo resolver is being used, so out of our scope.

[oakad]

https://danp.net/posts/macos-dns-change-in-go-1-20/

This had started happening relatively recently and I believe it is caused by changes above.

[mateusz834]

Can you try forcing the go resolver and see if it helps in your case? GODEBUG=netdns=go

[oakad]

How do I enable both this feature and dns debug so we can see it is used for real?

[mateusz834]

GODEBUG=netdns=go+2

[oakad]

Tough luck:

% go get nhooyr.io/websocket
go package net: confVal.netCgo = false netGo = true
go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(proxy.golang.org) = files,dns
go: module nhooyr.io/websocket: Get "https://proxy.golang.org/nhooyr.io/websocket/@v/list": write tcp [fe80::bed0:74ff:fe64:598e%utun4]:57052->[2a00:1450:4003:80c::2011]:443: write: socket is not connected

[oakad]

For reference, curl does this:

% curl -v https://proxy.golang.org/nhooyr.io/websocket/@v/list

• Host proxy.golang.org:443 was resolved.
• IPv6: 2a00:1450:4003:80c::2011
• IPv4: 142.250.184.177
• Trying 142.250.184.177:443...
• Trying [2a00:1450:4003:80c::2011]:443...
• Connected to proxy.golang.org (142.250.184.177) port 443
• ALPN: curl offers h2,http/1.1

[seankhliao]

What if you pass --ipv6 to curl?

In theory go's network stack should also be doing fast fallback / dual stack ipv4 and ipv6

[mateusz834]

So the tittle is incorrect, it resolves correctly, but it fails to connect to the server when ipv6 is unavail, right?

[oakad]

curl gets stuck when forced to use ipv6. It may be that despite underlying adapter has ipv6 disabled, the Cisco vpn client pretends it's got an ipv6 address on the utun interface. Yet it causes no issues anywhere, everything works fine apart from go.

% curl -v --ipv6 https://proxy.golang.org/nhooyr.io/websocket/@v/list

• Host proxy.golang.org:443 was resolved.
• IPv6: 2a00:1450:4003:80c::2011
• IPv4: (none)
• Trying [2a00:1450:4003:80c::2011]:443...
  ... waits for timeout