Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: Support the /etc/resolver DNS resolution configuration hierarchy on OS X #12524

Open
Rotonen opened this issue Sep 6, 2015 · 61 comments
Open

Comments

@Rotonen
Copy link

@Rotonen Rotonen commented Sep 6, 2015

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/resolver.5.html

OS X allows you to add TLD specific resolver configurations. Quite popular ones are /etc/resolver/vm for local virtual machines and /etc/resolver/dev for local development purposes.

https://golang.org/src/net/dnsclient_unix.go#L231

Go seems to be hardcoded to only take /etc/resolv.conf into account on Unix platforms.

@nodirt
Copy link
Contributor

@nodirt nodirt commented Sep 6, 2015

I don't think Go-native DNS resolving mechanism is used on Mac.
https://golang.org/src/net/dnsclient_unix.go#L231 is not executed if I run

addrs, err := net.LookupHost("google.com")

on my Mac.

If I enable debugging (GODEBUG=netdns=2 go run test.go), the following is printed:

go package net: using cgo DNS resolver
go package net: hostLookupOrder(google.com) = cgo

which means that OS-native DNS resolving is used.

Can you supply an exact configuration file, Go code, actual and expected output?

@titanous
Copy link
Member

@titanous titanous commented Sep 6, 2015

@nodirt This is for a binary with cgo off.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Sep 6, 2015

If cgo is disabled then the pure go DNS resolver will be used. If you want
to use the Mac DNS resolver, plese build with cgo.

On Mon, 7 Sep 2015 07:47 Jonathan Rudenberg notifications@github.com
wrote:

@nodirt https://github.com/nodirt This is for a binary with cgo off.


Reply to this email directly or view it on GitHub
#12524 (comment).

@nodirt
Copy link
Contributor

@nodirt nodirt commented Sep 6, 2015

Shouldn't be a problem since this is needed only on a dev machine.

On Sun, Sep 6, 2015 at 4:06 PM Dave Cheney notifications@github.com wrote:

If cgo is disabled then the pure go DNS resolver will be used. If you want
to use the Mac DNS resolver, plese build with cgo.

On Mon, 7 Sep 2015 07:47 Jonathan Rudenberg notifications@github.com
wrote:

@nodirt https://github.com/nodirt This is for a binary with cgo off.


Reply to this email directly or view it on GitHub
#12524 (comment).


Reply to this email directly or view it on GitHub
#12524 (comment).

@titanous
Copy link
Member

@titanous titanous commented Sep 6, 2015

In this specific case, @Rotonen was using the Flynn binary that we distribute as a compiled artifact, it is compiled without cgo to ease cross-compilation. Just because the user is a developer doesn't mean that they are a Go developer or want to compile the binary for themselves. The only question here is if this feature is out of scope for the pure-Go resolver.

@minux
Copy link
Member

@minux minux commented Sep 7, 2015

@ianlancetaylor ianlancetaylor changed the title Support the /etc/resolver DNS resolution configuration hierarchy on OS X net: Support the /etc/resolver DNS resolution configuration hierarchy on OS X Sep 8, 2015
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 8, 2015

I don't see anything wrong with supporting the OS X /etc/resolver directory. That said, my understanding is that the Go DNS resolver does not work well on most OS X machines. That is why it is disabled by default.

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Sep 8, 2015
@mterron
Copy link

@mterron mterron commented Jun 1, 2016

This would be great in all platforms anyway. Is there any disadvantage from supporting this behaviour? It seems that it'd neatly resolve the need to install and configure dnsmasq to provide the simple function of having different resolvers for different TLDs.

@jason-riddle
Copy link

@jason-riddle jason-riddle commented Feb 16, 2017

i know this issue is quite old but has there been any traction on this?

@ghost
Copy link

@ghost ghost commented Jun 23, 2017

any resolution?

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jun 23, 2017

Any updates would be posted here. No updates have been posted here.

@bitglue
Copy link

@bitglue bitglue commented Apr 20, 2018

See resolver(5). Just reading the files out of /etc/resolver/* will miss out on other mechanisms for configuring the same thing, for example configuration profiles or IKE attributes.

@flyinprogrammer
Copy link

@flyinprogrammer flyinprogrammer commented Sep 25, 2018

Just stumbled upon this today while attempting to use coredns as a dns proxy for local development. It's a real bummer to discover how naive our support for os x is.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Sep 25, 2018

We've generally assumed people use cgo on Darwin, so this bug has never been a priority.

I do admit that practically means that Darwin binaries need to be built on Darwin, which is difficult for people wanting to cross-compile for a dozen platforms as part of their release process.

Perhaps on Darwin without cgo we could just shell out to a program to do DNS resolution (e.g. host, dig, nslookup?). At least nslookup has an interactive mode that would permit re-using a child process for multiple lookups, if that proves necessary for performance.

mmrobins added a commit to mmrobins/homebrew-core that referenced this issue Nov 18, 2020
This resolves an issue where golang doesn't respect the way MacOS does
DNS resolution via `/etc/resolv.conf`

hashicorp/terraform#26532 (comment)
golang/go#12524

Without this setting whenever I try to `terraform init` to download
provider plugins while on my company's VPN, I get errors like the
following

```
Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
terraform-providers/heroku: no available releases match the given
constraints
```

```
Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
terraform-providers/fastly: could not connect to registry.terraform.io:
Failed
to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http:
request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting
headers)
```
BrewTestBot added a commit to Homebrew/homebrew-core that referenced this issue Nov 19, 2020
Set CGO_ENABLED=1 for terraform build

This resolves an issue where golang doesn't respect the way MacOS does
DNS resolution via `/etc/resolv.conf`

hashicorp/terraform#26532 (comment)
golang/go#12524

Without this setting whenever I try to `terraform init` to download
provider plugins while on my company's VPN, I get errors like the
following

```
Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
terraform-providers/heroku: no available releases match the given
constraints
```

```
Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider
terraform-providers/fastly: could not connect to registry.terraform.io:
Failed
to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": net/http:
request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting
headers)
```

Closes #65148.

Signed-off-by: chenrui <chenrui333@gmail.com>
Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
@jfesler
Copy link

@jfesler jfesler commented Jun 3, 2021

I build Darwin binaries from Linux based CI/CD pipelines.

In the apps I produce for my customers, I have had to resort to unspeakable hacks involving replacing the default resolver, with something that "speaks DNS" (since the resolver has no interface{}); and to follow what scutil shows (much like /etc/resolv.conf points people to). @JohnStarich 's post above is a similar module. This method is a terrible hack, and it only works in apps that explicitly load the module. (No offense to John or his code!)

Resorting to making releases from a laptop is pretty undesirable. Despite this, for some tools, this has had to happen anyways - because home-brew binaries of important projects don't work in the VPN case. kubectl is a great example.

I'd like to see us get to the point where Go binaries "just work", even when cross compiled, without having to incorporate resolver hacks like John's into all the Go apps our customers need on their laptops. Whether that means resolving (hah) the issues with the system resolver, or better understanding of the system's resolver configuration (including routing of queries by domain to alternate server(s)).

As a developer shipping binaries to customers, this is my single biggest Go pain point today.

@bitglue
Copy link

@bitglue bitglue commented Jun 3, 2021

I don't know if by "home-brew" you meant https://brew.sh/, but my experience has been packages built that way work as long as cgo hasn't been explicitly disabled, which was in the past, in the case of kubectl. I don't know what the current state is, but they claim to have fixed it without any explanation of where.

Of course this is of no help for cross-compiled binaries. I agree that it's a Big Problem, and it seems pretty clear just expecting every person shipping Go binaries to "just deal with it" isn't working.

@bhcleek
Copy link
Contributor

@bhcleek bhcleek commented Jun 3, 2021

I have had to resort to unspeakable hacks involving replacing the default resolver

I've been able to avoid the unspeakable hacks when cross-compiling by using xgo from its docker image, but xgo hasn't been updated in a long while now and seems to be unmaintained.

@zbintliff
Copy link

@zbintliff zbintliff commented Jun 3, 2021

@bitglue if you compile locally using brew, yes. But if the provider wants to avoid any go mod issues and provide the binary by building on circle ci or something, you can't do because of this. The resulting binary will not respect OSX resolvers, especially on corporate VPNs.

FlorianLoch added a commit to FlorianLoch/homebrew-core that referenced this issue Jun 15, 2021
It is required to build binaries for macOS using `CGO_ENABLED=1` in order to use macOS's native DNS/hostname resolver. See golang/go#12524. This is of special importance for corporate networks with GitLab instances running in a VPN.
BrewTestBot added a commit to Homebrew/homebrew-core that referenced this issue Jun 16, 2021
It is required to build binaries for macOS using `CGO_ENABLED=1` in order to use macOS's native DNS/hostname resolver. See golang/go#12524. This is of special importance for corporate networks with GitLab instances running in a VPN.

Closes #79324.

Signed-off-by: Carlo Cabrera <30379873+carlocab@users.noreply.github.com>
Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
smacfarlane added a commit to hashicorp/terraform-provider-scaffolding that referenced this issue Jun 17, 2021
In order for DNS resolution to work on MacOS in some environments, go
projects must be built with on a MacOS host with `-tags=netcgo` and
CGO_ENABLED not explicity set to either 0 or 1.   This is the default
behavor when running `go build` on MacOS.

Explicitly enabling CGO with `CGO_ENABLED=1` causes issues when
cross-compiling between GOARCHs on MacOS, related to the version of the
MacOS SDK that is selected. This may not be an issue in future MacOS
releases. Disabling CGO with `CGO_ENABLED=0` causes `-tags=netcgo` to be
ignored, preventing DNS from functioning.

This change seperates the `GOOS=darwin` from the remainder of the
supported `GOOS` targets behind different goreleaser build ids. This
allows us to build the MacOS releases on MacOS hosts in CI, and the
remainder can be cross-compiled from a Linux host.

Running goreleaser outside of CI will remain unaffected by this change
with the exception of building for MacOS from non-MacOS hosts, in which
case the resulting binaries may exhibit DNS issues but should otherwise
function. Users can skip building for MacOS in this case by running
`goreleaser build --id=default`.

For reference, the upstream issue with more details about DNS resolution
on MacOS can be found in this GitHub issue: golang/go#12524

Signed-off-by: Scott Macfarlane <smacfarlane@hashicorp.com>
@apparentlymart
Copy link

@apparentlymart apparentlymart commented Aug 26, 2021

Earlier on I described how the Terraform team at HashiCorp uses cross-compilation to build for multiple platforms, and thus our darwin_amd64 releases end up not having cgo enabled.

In the interests of keeping things current, I just wanted to note that in conjunction with reworking our build process to support other macOS-specific chores such as notarization and building for the new darwin_arm64 platform we have now abandoned cross-compilation as our build strategy for the official Terraform releases, and thus this problem is no longer such a high pressure for Terraform users who choose to use the binaries we produce.

However, there are still various people who for one reason or another build Terraform from source themselves, and they can unknowingly end up producing a build that has significantly different DNS resolution behavior than the official builds depending on what strategy they take to do so. Also, the Terraform artifacts we produce are used in conjunction with a variety of other plugin executables, many of which are built by teams at other companies who have devised their own release strategies which may or may not be using cross compilation or enabling cgo, and so this situation does still remain somewhat problematic for some smaller cohorts in our community.

Overall it feels unfortunate to me that the DNS resolution behavior of Go programs can vary so drastically depending on how they are built, whereas in most other regards the Go toolchain does a great job of ensuring consistency regardless of strategy. I think a number of Go-based codebases have gone through this sequence of being naive to that possibility, and then learning too late that this difference exists as a result of a likely-frustrating debugging session where the root cause exists outside of the developers' typical field of view.

With that said, I do understand that there isn't a super clear answer here, since DNS resolution behavior for Unix-alike systems has traditionally been the responsibility of a system's C library. If there isn't a viable way for Go to exhibit the correct DNS resolution behavior without interacting with libc using cgo then I hope instead for some inspiration on how to make this quirk more visible to developers, so they aren't given false optimism by how well the Go toolchain typically handles cross-compilation.

@blalor
Copy link

@blalor blalor commented Aug 26, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.