Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: Support the /etc/resolver DNS resolution configuration hierarchy on OS X when cgo is disabled #12524

Closed
Rotonen opened this issue Sep 6, 2015 · 86 comments
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Milestone

Comments

@Rotonen
Copy link

Rotonen commented Sep 6, 2015

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/resolver.5.html

OS X allows you to add TLD specific resolver configurations. Quite popular ones are /etc/resolver/vm for local virtual machines and /etc/resolver/dev for local development purposes.

https://golang.org/src/net/dnsclient_unix.go#L231

Go seems to be hardcoded to only take /etc/resolv.conf into account on Unix platforms.

@nodirt
Copy link
Contributor

nodirt commented Sep 6, 2015

I don't think Go-native DNS resolving mechanism is used on Mac.
https://golang.org/src/net/dnsclient_unix.go#L231 is not executed if I run

addrs, err := net.LookupHost("google.com")

on my Mac.

If I enable debugging (GODEBUG=netdns=2 go run test.go), the following is printed:

go package net: using cgo DNS resolver
go package net: hostLookupOrder(google.com) = cgo

which means that OS-native DNS resolving is used.

Can you supply an exact configuration file, Go code, actual and expected output?

@titanous
Copy link
Member

titanous commented Sep 6, 2015

@nodirt This is for a binary with cgo off.

@davecheney
Copy link
Contributor

davecheney commented Sep 6, 2015

If cgo is disabled then the pure go DNS resolver will be used. If you want
to use the Mac DNS resolver, plese build with cgo.

On Mon, 7 Sep 2015 07:47 Jonathan Rudenberg notifications@github.com
wrote:

@nodirt https://github.com/nodirt This is for a binary with cgo off.


Reply to this email directly or view it on GitHub
#12524 (comment).

@nodirt
Copy link
Contributor

nodirt commented Sep 6, 2015

Shouldn't be a problem since this is needed only on a dev machine.

On Sun, Sep 6, 2015 at 4:06 PM Dave Cheney notifications@github.com wrote:

If cgo is disabled then the pure go DNS resolver will be used. If you want
to use the Mac DNS resolver, plese build with cgo.

On Mon, 7 Sep 2015 07:47 Jonathan Rudenberg notifications@github.com
wrote:

@nodirt https://github.com/nodirt This is for a binary with cgo off.


Reply to this email directly or view it on GitHub
#12524 (comment).


Reply to this email directly or view it on GitHub
#12524 (comment).

@titanous
Copy link
Member

titanous commented Sep 6, 2015

In this specific case, @Rotonen was using the Flynn binary that we distribute as a compiled artifact, it is compiled without cgo to ease cross-compilation. Just because the user is a developer doesn't mean that they are a Go developer or want to compile the binary for themselves. The only question here is if this feature is out of scope for the pure-Go resolver.

@minux
Copy link
Member

minux commented Sep 7, 2015

@ianlancetaylor ianlancetaylor changed the title Support the /etc/resolver DNS resolution configuration hierarchy on OS X net: Support the /etc/resolver DNS resolution configuration hierarchy on OS X Sep 8, 2015
@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Sep 8, 2015

I don't see anything wrong with supporting the OS X /etc/resolver directory. That said, my understanding is that the Go DNS resolver does not work well on most OS X machines. That is why it is disabled by default.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Sep 8, 2015
@mterron
Copy link

mterron commented Jun 1, 2016

This would be great in all platforms anyway. Is there any disadvantage from supporting this behaviour? It seems that it'd neatly resolve the need to install and configure dnsmasq to provide the simple function of having different resolvers for different TLDs.

@jason-riddle

This comment was marked as spam.

@ghost

This comment was marked as spam.

@bradfitz
Copy link
Contributor

bradfitz commented Jun 23, 2017

Any updates would be posted here. No updates have been posted here.

@bradfitz bradfitz added help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 23, 2017
@bitglue
Copy link

bitglue commented Apr 20, 2018

See resolver(5). Just reading the files out of /etc/resolver/* will miss out on other mechanisms for configuring the same thing, for example configuration profiles or IKE attributes.

@flyinprogrammer
Copy link

flyinprogrammer commented Sep 25, 2018

Just stumbled upon this today while attempting to use coredns as a dns proxy for local development. It's a real bummer to discover how naive our support for os x is.

@bradfitz
Copy link
Contributor

bradfitz commented Sep 25, 2018

We've generally assumed people use cgo on Darwin, so this bug has never been a priority.

I do admit that practically means that Darwin binaries need to be built on Darwin, which is difficult for people wanting to cross-compile for a dozen platforms as part of their release process.

Perhaps on Darwin without cgo we could just shell out to a program to do DNS resolution (e.g. host, dig, nslookup?). At least nslookup has an interactive mode that would permit re-using a child process for multiple lookups, if that proves necessary for performance.

@timfallmk
Copy link

timfallmk commented Nov 1, 2022

Seven years, but we got there. Thanks for all the hard work everyone!

Edit: Math is hard

@AaronFriel
Copy link

AaronFriel commented Nov 3, 2022

Thank you @rsc, @ianlancetaylor, @bradfitz for closing this issue!

I am overjoyed to deliver the good news to our team and to share it with other maintainers of open source tools. This was the most significant obstacle to building tools that work for every user Go, I can't wait for the release that lands this CL!

@AaronFriel
Copy link

AaronFriel commented Nov 3, 2022

@rsc Will the next release to include this be 1.20 or 1.19.4?

@seankhliao
Copy link
Member

seankhliao commented Nov 3, 2022

1.20

romaindoumenc pushed a commit to TroutSoftware/go that referenced this issue Nov 3, 2022
Change the macOS implementation to use libc calls.
Using libc calls directly is what we do for all the runtime and os syscalls.
Doing so here as well improves consistency and also makes it possible
to cross-compile (from non-Mac systems) macOS binaries that use the
native name resolver.

Fixes golang#12524.

Change-Id: I011f4fcc5c50fbb5396e494889765dcbb9342336
Reviewed-on: https://go-review.googlesource.com/c/go/+/446178
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
@AaronFriel
Copy link

AaronFriel commented Nov 3, 2022

Would it be possible to consider this for cherry-pick or an env var to enable during compilation or runtime in 1.19.x?

I believe the 1.20 release would ordinarily occur in February to March of next year (given the 6 month release cycle). If an env var enabled this behavior sooner, I believe the Pulumi CLI and provider binary ecosystem would be early adopters.

@seankhliao
Copy link
Member

seankhliao commented Nov 3, 2022

I don't think this meets the bar for backports https://github.com/golang/go/wiki/MinorReleases.

Use tip or cherry pick onto your own fork if you need it earlier.

@AaronFriel
Copy link

AaronFriel commented Nov 4, 2022

Reading the docs, it appears that interested parties may make a suggestion to backport an issue when a workaround is untenable. For Pulumi, we produce many dozens of binaries and imposing cross-compilation costs (& other cgo behavior changes) is not viable. In addition, we produce guidance for our customers & users to produce their own provider binaries in Go and it imposes a cost on them. (And not every CI tool is as easy to produce CGO binaries on macOS as GitHub Actions.)

I don't want to overstep however, it looks like most uses of the gopherbot to create the backport issue have been by Go maintainers & per the note you shared, "only the authors of the original CL have the ability to create the cherry-pick."

@rsc Would it be possible to consider this for backport given the aforementioned reason?

Though if the go 1.20 beta 1 is anticipated soon (last year the 1.18 beta was available in December), we may consider using that and recommending it in the interim. The goal is to mitigate issues like pulumi/pulumi-aws#2185 and I believe that this CL may fix other macOS name resolution bugs such as #52839.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Nov 4, 2022

This is a risky change that's going to need some significant testing and soak time. It's not appropriate for a backport to a minor release. I'm sorry.

We expect the first release candidate for 1.10 to be available in early December.

@AaronFriel
Copy link

AaronFriel commented Nov 4, 2022

Appreciate that, I totally understand the need for this to be well vetted.

@bradfitz
Copy link
Contributor

bradfitz commented Nov 5, 2022

@AaronFriel, Pulumni can make its own fork of Go with this change cherry-picked in to build your CLI binaries.

We maintain our own fork of Go fo @tailscale at https://github.com/tailscale/go which over the past 2.5 years has fluctuated from a few to a few dozen patches, as needed. It lets us move at our own pace, without worrying about Go release cycles.

Feel free to copy our infrastructure for doing so. Or contact me directly if you'd like.

@danp
Copy link
Contributor

danp commented Nov 6, 2022

Great to see this fixed, thanks!

I think the build tags need a slight adjustment. When testing it out by building for darwin/arm64 on linux/amd64 I was still getting go package net: built with netgo build tag; using Go's DNS resolver from GODEBUG=netdns=1 running this program.

I think this diff is needed:

diff --git a/src/net/cgo_unix_syscall.go b/src/net/cgo_unix_syscall.go
index 7170f14c46..6b146e58b1 100644
--- a/src/net/cgo_unix_syscall.go
+++ b/src/net/cgo_unix_syscall.go
@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build cgo && !netgo && darwin
+//go:build !cgo && !netgo && darwin
 
 package net
 
diff --git a/src/net/netgo.go b/src/net/netgo.go
index 75baa88035..3bfb2ee86c 100644
--- a/src/net/netgo.go
+++ b/src/net/netgo.go
@@ -4,9 +4,9 @@
 
 // Default netGo to true if the netgo build tag is being used, or the
 // C library DNS routines are not available. Note that the C library
-// routines are always available on Windows.
+// routines are always available on Windows and Darwin.
 
-//go:build netgo || (!cgo && !windows)
+//go:build netgo || (!cgo && !windows && !darwin)
 
 package net

With that, I was able to build for darwin/arm64 on linux/amd64 and get go package net: using cgo DNS resolver. I was also able to build directly on my darwin/arm64 system with CGO_ENABLED=0 and still have it use the cgo resolver.

Happy to open a CL if that sounds right.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Nov 6, 2022

There may be something to do here but I don't see how change could be correct. If the cgo build tag is not enabled, we will build net/cgo_stub.go, and the direct calls to getaddrinfo won't occur. Or so it seems to me.

@danp
Copy link
Contributor

danp commented Nov 6, 2022

You are quite right! The debug info seemed correct but it was hitting the stub cgoLookupHost and falling back to the Go resolver.

Perhaps this is getting closer. With it, along with some print debugging in cgo_unix.go's cgoLookupHost, I can build on my darwin/arm64 system with CGO_ENABLED=0 and 1 and see that it's getting results there. I'm also able to build in both CGO_ENABLED modes on a linux/amd64 system and have things still build and seem to work.

diff --git a/src/net/cgo_bsd.go b/src/net/cgo_bsd.go
index 1456289b06..082e91faa8 100644
--- a/src/net/cgo_bsd.go
+++ b/src/net/cgo_bsd.go
@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build cgo && !netgo && (darwin || dragonfly || freebsd)
+//go:build cgo && !netgo && (dragonfly || freebsd)
 
 package net
 
diff --git a/src/net/cgo_darwin.go b/src/net/cgo_darwin.go
new file mode 100644
index 0000000000..af0217b6b4
--- /dev/null
+++ b/src/net/cgo_darwin.go
@@ -0,0 +1,5 @@
+package net
+
+import "internal/syscall/unix"
+
+const cgoAddrInfoFlags = (unix.AI_CANONNAME | unix.AI_V4MAPPED | unix.AI_ALL) & unix.AI_MASK
diff --git a/src/net/cgo_stub.go b/src/net/cgo_stub.go
index 298d829f6f..c901d4bb80 100644
--- a/src/net/cgo_stub.go
+++ b/src/net/cgo_stub.go
@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build !cgo || netgo
+//go:build (!cgo && !darwin) || netgo
 
 package net
 
diff --git a/src/net/cgo_unix.go b/src/net/cgo_unix.go
index a944727338..415a2b1904 100644
--- a/src/net/cgo_unix.go
+++ b/src/net/cgo_unix.go
@@ -7,7 +7,7 @@
 // Instead of C.foo it uses _C_foo, which is defined in either
 // cgo_unix_cgo.go or cgo_unix_syscall.go
 
-//go:build cgo && !netgo && unix
+//go:build !netgo && (cgo || darwin)
 
 package net
 
diff --git a/src/net/cgo_unix_syscall.go b/src/net/cgo_unix_syscall.go
index 7170f14c46..c5c27967b1 100644
--- a/src/net/cgo_unix_syscall.go
+++ b/src/net/cgo_unix_syscall.go
@@ -2,7 +2,7 @@
 // Use of this source code is governed by a BSD-style
 // license that can be found in the LICENSE file.
 
-//go:build cgo && !netgo && darwin
+//go:build !netgo && darwin
 
 package net
 
diff --git a/src/net/netgo.go b/src/net/netgo.go
index 75baa88035..3bfb2ee86c 100644
--- a/src/net/netgo.go
+++ b/src/net/netgo.go
@@ -4,9 +4,9 @@
 
 // Default netGo to true if the netgo build tag is being used, or the
 // C library DNS routines are not available. Note that the C library
-// routines are always available on Windows.
+// routines are always available on Windows and Darwin.
 
-//go:build netgo || (!cgo && !windows)
+//go:build netgo || (!cgo && !windows && !darwin)
 
 package net
 

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Nov 7, 2022

Probably best to discuss further on a patch submission. Thanks.

@gopherbot
Copy link

gopherbot commented Nov 7, 2022

Change https://go.dev/cl/448020 mentions this issue: net: adjust build tags for darwin libc calls

gopherbot pushed a commit that referenced this issue Nov 7, 2022
Support for direct libc calls was added in CL 446178 but the build
tags weren't quite activating it when cgo was not enabled. Adjust them
and add a new supporting file for darwin.

This should use the new direct libc calls with both CGO_ENABLED=0 and
CGO_ENABLED=1 when building for darwin.

Updates #12524

Change-Id: Ieee4b298dee13f389ed3a63c0a4a3a18c9180163
Reviewed-on: https://go-review.googlesource.com/c/go/+/448020
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Dan Peterson <danp@danp.net>
Auto-Submit: Ian Lance Taylor <iant@google.com>
@AaronFriel
Copy link

AaronFriel commented Nov 10, 2022

@bradfitz I may reach out, thank you! I'm not sure if we're ready to take on maintaining a fork, but you've piqued my interest.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 14, 2022
@dmitshur dmitshur modified the milestones: Backlog, Go1.20 Nov 14, 2022
ashley-cui added a commit to ashley-cui/podman that referenced this issue Nov 30, 2022
Fix a regression caused by not setting CGO correctly in the makefile while building podman-remote on OSX. See golang/go#12524 for details.

Signed-off-by: Ashley Cui <acui@redhat.com>
ashley-cui added a commit to ashley-cui/podman that referenced this issue Nov 30, 2022
Fix a regression caused by not setting CGO correctly in the makefile while building podman-remote on OSX. See golang/go#12524 for details.

Signed-off-by: Ashley Cui <acui@redhat.com>
ashley-cui added a commit to ashley-cui/podman that referenced this issue Nov 30, 2022
Fix a regression caused by not setting CGO correctly in the makefile while building podman-remote on OSX. See golang/go#12524 for details.

Signed-off-by: Ashley Cui <acui@redhat.com>
ashley-cui added a commit to ashley-cui/podman that referenced this issue Nov 30, 2022
Fix a regression caused by not setting CGO correctly in the makefile while building podman-remote on OSX. See golang/go#12524 for details.

Signed-off-by: Ashley Cui <acui@redhat.com>
ashley-cui added a commit to ashley-cui/podman that referenced this issue Nov 30, 2022
Fix a regression caused by not setting CGO correctly in the makefile while building podman-remote on OSX. See golang/go#12524 for details.

Signed-off-by: Ashley Cui <acui@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-Darwin
Projects
None yet
Development

Successfully merging a pull request may close this issue.