New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: revisit unconditional use of cgo lookups for darwin #16345

Open
danp opened this Issue Jul 13, 2016 · 21 comments

Comments

Projects
None yet
9 participants
@danp
Contributor

danp commented Jul 13, 2016

https://golang.org/cl/8945 changed to using the native Go stub resolver for most systems. Darwin was excluded, here, due to firewall warnings.

Is this still an issue?

On my 10.11 system with the firewall enabled this test program doesn't produce any warnings, even with cgo disabled:

package main

import (
    "fmt"
    "net"
)

func main() {
    ns, err := net.LookupHost("www.google.com")
    if err != nil {
        panic(err)
    }
    fmt.Println(ns)
}
% CGO_ENABLED=0 GODEBUG=netdns=2 go run main.go
go package net: built with netgo build tag; using Go's DNS resolver
go package net: hostLookupOrder(www.google.com) = files,dns
[216.58.192.164 2607:f8b0:4009:80e::2004]

(nothing pops up when running)

There are probably other reasons to conditionally exclude Darwin for now, such as if /etc/resolver config is used (#12524), but perhaps removing this blanket condition could be a start.

cc @mdempsky and @bradfitz since you authored that change.

@ianlancetaylor ianlancetaylor added this to the Go1.8 milestone Jul 13, 2016

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 16, 2016

The Darwin exclusion predates https://golang.org/cl/8945 I thought. Can you do some digging and investigate its history?

What do we gain by using Go's resolver by default on Mac? It seems like we'd need to do #12524 first as you mentioned otherwise we risk doing more harm than good.

@danp

This comment has been minimized.

Contributor

danp commented Jul 19, 2016

I did some digging!

Before https://golang.org/cl/8945 the only way to use the Go's resolver (for most platforms, not just darwin) was to build with netgo (and/or without cgo?). This caused the stubs to return false for completed and triggered use of the Go resolver.

I discovered along the way that android is also currently summarily excluded from using Go's resolver due to #10714, though it's done per request instead of per-conf-setup like darwin. In the CL for that, @minux called for the same to be done for iOS and then subsequently found the exclusion for darwin.

All unix platforms except for darwin and android now use Go's resolver unless weirdness is detected along here or here. I think we could gain more consistency across platforms if darwin were to join in with necessary safety checks.

Re #12524, that could be something we detect and fall back to cgo for initially. Once there is support for it in the Go resolver the check could be removed and Go's resolver could be used in that case.

At the very least if we decide not to pursue this it would be good to update the comment around the darwin exclusion to explain it's due to more than possible firewall warnings.

@minux

This comment has been minimized.

Member

minux commented Jul 20, 2016

@danp

This comment has been minimized.

Contributor

danp commented Jul 20, 2016

I only have immediate access to 10.11 and 10.10 systems, neither of which pop up anything when trying the test program in the description (which I've updated to show exactly how I ran it). Trying to see if any community members have access to older systems for testing. Worth noting that 10.10 would have been current at the time of https://golang.org/cl/8945.

I'm also unable to find anything when searching for this issue happening generally. @bradfitz, did you experience this firsthand? Is it possible it was this standard firewall warning that pops up when a process listens for outside connections, and maybe caused by something else?

@groob

This comment has been minimized.

Contributor

groob commented Jul 20, 2016

@dpiddy @bradfitz I don't think this a bug. I tested on 10.11 and the application firewall is not triggered either.

Having a "server" triggers the firewall on OS X, since that would offer an incoming connection. But DNS resolution is outgoing, so the firewall would ignore it.

@danp

This comment has been minimized.

Contributor

danp commented Aug 16, 2016

Any further thoughts on this?

I can try and put together ways to detect when we should not use Go's resolver if we want to move forward with removing the blanket condition.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 9, 2016

@dpiddy, I can test on OS X 10.8, now that we have VM-based builders and ancient versions available.

@danp

This comment has been minimized.

Contributor

danp commented Sep 9, 2016

Great! Let me know what I can do to help.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 9, 2016

I tried on OS X 10.8 with the firewall on in its most restrictive mode (block all incoming connections), and I saw no pop-up dialog using Go's DNS resolver. I set GODEBUG=netdns=go+2 and saw it use Go's DNS resolver and get the right answers.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 9, 2016

I can try and put together ways to detect when we should not use Go's resolver if we want to move forward with removing the blanket condition.

Propose away.

@danp

This comment has been minimized.

Contributor

danp commented Sep 21, 2016

Hope to spend some good time on this soon. So far existence of /etc/resolvers is the first obvious cgo fallback condition.

What I'm more unsure of is this bit in the darwin resolver(5) man page:

However, client configurations are not limited to file storage. The implementation of the DNS multi-client search strategy may also locate client configuratins in other data sources, such as the System Configuration Database. Users of the DNS system should make no assumptions about the source of the configuration data.

I'm not familiar with what those other data sources might be or if we can detect their use.

Issues like docker/for-mac#19 suggest scutil can be used to discover DNS config in the System Configuration Database. Would that give enough hints on when cgo should be preferred? Would it work for all Go-supported OS X versions?

There might also be special names that should prefer cgo.

Any available insight on these things would be greatly appreciated!

And if it would help get things started to open a CL removing the blanket darwin exclusion but falling back to cgo if /etc/resolvers exists I can certainly do that.

@quentinmit

This comment has been minimized.

Contributor

quentinmit commented Oct 7, 2016

scutil is a wrapper around the SystemConfiguration framework. scutil --dns prints the current resolver configuration. The default configuration is DNS from DHCP/manual, followed by mdns for a number of specific domains ("local" and the PTR domains).

It looks like Chromium uses the unexported symbol dns_configuration_copy to get the configuration: https://chromium.googlesource.com/chromium/src/+/b4aadc32a7fd6d42ac3cc9adbb20a8ee4a267572/net/dns/dns_config_watcher_mac.cc

@quentinmit quentinmit added the NeedsFix label Oct 7, 2016

@quentinmit

This comment has been minimized.

Contributor

quentinmit commented Oct 7, 2016

Also, another link. Here is the implementation of dns_configuration_copy (though I think we should use it from libSystem and/or SystemConfiguration.framework, not copy it into Go):

http://publicsource.apple.com/source/configd/configd-802.40.13/dnsinfo/dnsinfo_copy.c

@danp

This comment has been minimized.

Contributor

danp commented Oct 7, 2016

Thanks for the tips!

though I think we should use it from libSystem and/or SystemConfiguration.framework, not copy it into Go

Any pointers on what that might look like?

Couple other notes from digging:

  • resolver(5) describes support for custom ports both in nameserver lines with 1.2.3.4.55 for port 55 and via the port directive. Use of port would trigger setting unknownOpt on the config but the nameserver form will need consideration.
  • like OpenBSD, I don't think OS X has any notion of nsswitch.conf so the condition there should probably be expanded or reworked
@quentinmit

This comment has been minimized.

Contributor

quentinmit commented Oct 7, 2016

OS X's version of nsswitch.conf is the data in the SystemConfiguration framework.

Look at the Chromium source code; also look at our Keychain code in crypto/x509. I imagine we'd do something similar where we weakly link the symbols and call them from cgo.

Though an open question is whether it's actually worth the complexity to use the native resolvers for some but not all queries, I think.

@danp

This comment has been minimized.

Contributor

danp commented Oct 7, 2016

The crypto/x509 tip helped me get an idea for what would be involved, thanks.

Though an open question is whether it's actually worth the complexity to use the native resolvers for some but not all queries, I think.

If the result of this is deciding it's too complex to use Go's resolver when the native resolver (via cgo) is available, that's fine with me. At least we know and have info to consider for the future.

There might still be room for improving the experience when the native resolver is unavailable, such as with support for /etc/resolver.

@rsc rsc added NeedsDecision and removed NeedsFix labels Oct 20, 2016

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 27, 2016

On my OS X 10.11.6 system, if I go to Security & Privacy and then Firewall and Turn Firewall On and then Firewall Options... and then check "Block all incoming connections", then GODEBUG=netdns=go+2 go run lookup.go hangs, but GODEBUG=netdns=cgo+2 go run lookup.go keeps working.

It is true that without "Block all incoming connections", the cgo mode does not get popup dialogs like it used to long ago. But I think we probably still can't turn on Go resolution by default.

For the record, the original CL adding cgo support for resolving host names, specifically for OS X, was golang.org/cl/4437053 aka c9164a5.

I agree it would be nice if we could do what we do on Linux etc where we look at resolv.conf and nsswitch.conf and decide if it's OK to use the pure Go resolver. The reason we look at nsswitch.conf is to see if there are any non-DNS lookup methods configured, and if so we delegate to the C library. On the Mac there is no nsswitch.conf but as I understand it there's effectively always a non-DNS lookup method configured (Bonjour). So if there were an accurate nsswitch.conf we'd never use the Go resolver by default.

To summarize:

  1. The most restrictive OS X firewall setting makes the Go resolver hang.
  2. OS X has no nsswitch.conf but if it had an accurate one we'd see non-DNS resolution methods listed and would choose to use the cgo resolver.

For both these reasons, I think we should leave the default on OS X where it is, namely using the cgo resolver.

Note that people who want to use the pure Go resolver need not recompile their programs, as in @danp's example (the CGO_ENABLED=0 is causing package net to be rebuilt entirely). It suffices to set GODEBUG=netdns=go, as mentioned in the net package doc.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Nov 1, 2016

On the Mac there is no nsswitch.conf but as I understand it there's effectively always a non-DNS lookup method configured (Bonjour).

That's only for *.local names, or things without a dot. Even on Linux when Avahi/mDNS stuff is listed in nslookup.conf, we only do cgo if we see *.local or no dot (if search domains are listed), iirc. We could probably do the same on Darwin.

I'm going to kick this to Unplanned for now. If somebody wants to own this and do the pure Go thing when it's really safe on macOS and won't be annoying for the user, feel free to research and post your plan. It could go in Go 1.9 if there's a plan that addresses @rsc's concerns.

@danp

This comment has been minimized.

Contributor

danp commented Feb 22, 2017

Just tried the test on my 10.12.3 system, with "Block all incoming connections" enabled, and it worked both with GODEBUG=netdns=go+2 and GODEBUG=netdns=cgo+2. Can others confirm?

@shawnps

This comment has been minimized.

Member

shawnps commented Feb 25, 2017

@danp same result for me

@bitglue

This comment has been minimized.

bitglue commented May 1, 2018

VPN is a common use case where using the Go native resolver with current behavior would break many things. For example, an IKEv2 VPN which does not do full tunneling will not install the DNS server provided in the IKE configuration as the system default, and thus will not end up in /etc/resolv.conf, thus will not be used by the native Go resolver. It is possible to push additional IKE configuration attributes which specify the VPN-provided DNS server should be used for some domains, or even all domains (less than ideal, because privacy), but in no case is this exposed through /etc/resolv.conf: one must query SystemConfiguration to get it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment