New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: allow custom Resolver method implementation(s) #12503

Open
sajal opened this Issue Sep 4, 2015 · 30 comments

Comments

Projects
None yet
@sajal

sajal commented Sep 4, 2015

I mentioned in #12476 that I would like to detect the time it took for DNS resolution phase of the Dial process in Dialer.Dial . The solution posted there was very hacky and adds unnecessarily to the API.

@bradfitz suggested

Perhaps net.Dialer could have an optional Resolver, akin to how http.Client has an optional Transport, or http.Server has an optional ErrorLog, etc.

This seems like an excellent idea. Here is how I propose we go about it by adding minimal complexity and preserving code compatibility.

I propose net package adds a new Resolver interface.

type Resolver interface {
    Resolve(host string, deadline time.Time) (addrs []IPAddr, err error)
}

The signature of Resolver.Resolve is same as lookupIPDeadline which Dial eventually uses. Dialer gets an optional field CustomResolver of type Resolver.

The Resolver object (or nil) gets passed around thru the resolution process.

Dialer.Dial -> resolveAddrList -> internetAddrList .

internetAddrList currently always uses lookupIPDeadline, it would need to be changed such that if the passed custom resolver is not nil then use it, otherwise use lookupIPDeadline.

Other functions calling resolveAddrList or internetAddrList would need to be modified to add an extra nil argument . This does not break code compatibility because they are unexported functions.

Benefits of allowing a custom Resolver

  • Allowing me to collect timing information as mentioned in #12476
  • Allowing users to implement their own DNS logic. Failovers, etc.
  • Mocking for tests.
  • Client side caching, pre-fetching, etc.
  • In time, other packages ( like the superb github.com/miekg/dns ) could provide their own Resolver implementations.
  • Great for people like me who rely on Go to write network debugging tools.

@sajal sajal changed the title from proposal: Allow passing of custom Resolver to Dialer.dial in package net to proposal: Allow passing of custom Resolver to Dialer.Dial in package net Sep 4, 2015

@sajal sajal changed the title from proposal: Allow passing of custom Resolver to Dialer.Dial in package net to proposal: Allow passing of custom Resolver to net.Dialer Sep 4, 2015

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Sep 4, 2015

@sajal

This comment has been minimized.

Show comment
Hide comment
@sajal

sajal Sep 9, 2015

Should I implement and submit the change for codereview? Or wait for some comments here?

sajal commented Sep 9, 2015

Should I implement and submit the change for codereview? Or wait for some comments here?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Sep 9, 2015

Member

No need to prototype it yet. The code will be relatively easy compared to getting the design right.

I suspect that signature isn't general enough. Maybe it's good enough for a dialer, but perhaps it needs a different name.

I bet we don't want to define an interface in the net package. If anything, it could just be an optional func type on the Dialer, similar to funcs on http://golang.org/pkg/net/http/#Transport

Member

bradfitz commented Sep 9, 2015

No need to prototype it yet. The code will be relatively easy compared to getting the design right.

I suspect that signature isn't general enough. Maybe it's good enough for a dialer, but perhaps it needs a different name.

I bet we don't want to define an interface in the net package. If anything, it could just be an optional func type on the Dialer, similar to funcs on http://golang.org/pkg/net/http/#Transport

@sajal

This comment has been minimized.

Show comment
Hide comment
@sajal

sajal Sep 9, 2015

Perhaps call it Lookupfunc(or better name) and deadline-ing is handled inside net package. It might mirror signature of net.LookupIP which is used by default if Lookupfunc is nil.

Anything that does a lookup could ask for optional field for Lookupfunc to allow user to provide their own implementation.

sajal commented Sep 9, 2015

Perhaps call it Lookupfunc(or better name) and deadline-ing is handled inside net package. It might mirror signature of net.LookupIP which is used by default if Lookupfunc is nil.

Anything that does a lookup could ask for optional field for Lookupfunc to allow user to provide their own implementation.

@adg adg added Proposal and removed Proposal labels Sep 25, 2015

@rsc rsc modified the milestones: Proposal, Unplanned Oct 24, 2015

@rsc rsc changed the title from proposal: Allow passing of custom Resolver to net.Dialer to proposal: allow net.Dialer to use custom resolver Oct 24, 2015

@rsc rsc changed the title from proposal: allow net.Dialer to use custom resolver to proposal: net: allow Dialer to use custom resolver Oct 24, 2015

@benburkert

This comment has been minimized.

Show comment
Hide comment
@benburkert

benburkert Nov 5, 2015

Contributor

I would also like to see a Resolver interface but with multiple methods that match the net.Lookup* funcs.

type Resolver interface {
  LookupAddr(addr string) (names []string, err error)
  LookupCNAME(name string) (cname string, err error)
  LookupHost(host string) (addrs []string, err error)
  LookupIP(host string) (ips []IP, err error)
  LookupMX(name string) (mxs []*MX, err error)
  LookupNS(name string) (nss []*NS, err error)
  LookupPort(network, service string) (port int, err error)
  LookupSRV(service, proto, name string) (cname string, addrs []*SRV, err error)
  LookupTXT(name string) (txts []string, err error)
}

The timeout & deadline functionality could be configured when the resolver is created:

func NewResolver(options ResolverOption...) (Resolver, error)

type ResolverOption func(*resolver) error

func ResolverTimeout(duration time.Duration) ResolverOption
func ResolverDeadline(deadline time.Time) ResolverOption
Contributor

benburkert commented Nov 5, 2015

I would also like to see a Resolver interface but with multiple methods that match the net.Lookup* funcs.

type Resolver interface {
  LookupAddr(addr string) (names []string, err error)
  LookupCNAME(name string) (cname string, err error)
  LookupHost(host string) (addrs []string, err error)
  LookupIP(host string) (ips []IP, err error)
  LookupMX(name string) (mxs []*MX, err error)
  LookupNS(name string) (nss []*NS, err error)
  LookupPort(network, service string) (port int, err error)
  LookupSRV(service, proto, name string) (cname string, addrs []*SRV, err error)
  LookupTXT(name string) (txts []string, err error)
}

The timeout & deadline functionality could be configured when the resolver is created:

func NewResolver(options ResolverOption...) (Resolver, error)

type ResolverOption func(*resolver) error

func ResolverTimeout(duration time.Duration) ResolverOption
func ResolverDeadline(deadline time.Time) ResolverOption
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Nov 6, 2015

Member

@benburkert, that is not a Go-style (small) interface. Once you have 9 methods, surely somebody would want to add a tenth later, but they can't for compatibility reasons. 9 methods is also hard to implement. We'd probably have to add some sort of EmptyResolver type that people could embed which just returned errors for everything.

I'd start with looking at which interfaces are actually needed by the things this bug is about. Maybe you'd have 9 interfaces instead (maybe starting with 3?) and combine them as needed like io.ReadWriteCloser? I don't know. I haven't given this much thought yet.

Member

bradfitz commented Nov 6, 2015

@benburkert, that is not a Go-style (small) interface. Once you have 9 methods, surely somebody would want to add a tenth later, but they can't for compatibility reasons. 9 methods is also hard to implement. We'd probably have to add some sort of EmptyResolver type that people could embed which just returned errors for everything.

I'd start with looking at which interfaces are actually needed by the things this bug is about. Maybe you'd have 9 interfaces instead (maybe starting with 3?) and combine them as needed like io.ReadWriteCloser? I don't know. I haven't given this much thought yet.

@davecheney

This comment has been minimized.

Show comment
Hide comment
@davecheney

davecheney Nov 6, 2015

Contributor

What about Lookup(recordtype, query string) ...

It's similar to our Dial(network, address string) function, and would permit wildcard, ANY, and lookups for types not yet added to the dns spec.

Just spitballing...

Contributor

davecheney commented Nov 6, 2015

What about Lookup(recordtype, query string) ...

It's similar to our Dial(network, address string) function, and would permit wildcard, ANY, and lookups for types not yet added to the dns spec.

Just spitballing...

@theckman

This comment has been minimized.

Show comment
Hide comment
@theckman

theckman Jan 14, 2016

Contributor

I just ran in to this issue myself, except a little bit abstracted away from net.Dialer. My use-case may be a little weird, but this would come in extremely handy for me if it was also exposed within the net/http package.

I'm writing a utility that's going to talk over TLS to backend systems (HTTP + JSON) and I'm using Consul to discover the individual backend nodes. The biggest issue is that I don't have all of my system's DNS requests being serviced by Consul, so pulling a configuration from /etc/resolv.conf won't really work. I plan on using their port 8600 DNS interface.

So I'll end up needing to first obtain a list of IP addresses from the Consul DNS endpoint and then use that IP address in the URL. Following that, I'll need to set the Host field on the request so that the TLS validation works. The only downside here is that I end up having to do a network operation at the creation time of the http.Request struct instead of when actually invoking the request.

If the http.Transport struct was modified to support a custom DNS resolver code path, it would make the code cleaner and avoid the upfront network call.

Contributor

theckman commented Jan 14, 2016

I just ran in to this issue myself, except a little bit abstracted away from net.Dialer. My use-case may be a little weird, but this would come in extremely handy for me if it was also exposed within the net/http package.

I'm writing a utility that's going to talk over TLS to backend systems (HTTP + JSON) and I'm using Consul to discover the individual backend nodes. The biggest issue is that I don't have all of my system's DNS requests being serviced by Consul, so pulling a configuration from /etc/resolv.conf won't really work. I plan on using their port 8600 DNS interface.

So I'll end up needing to first obtain a list of IP addresses from the Consul DNS endpoint and then use that IP address in the URL. Following that, I'll need to set the Host field on the request so that the TLS validation works. The only downside here is that I end up having to do a network operation at the creation time of the http.Request struct instead of when actually invoking the request.

If the http.Transport struct was modified to support a custom DNS resolver code path, it would make the code cleaner and avoid the upfront network call.

@mikioh

This comment has been minimized.

Show comment
Hide comment
@mikioh

mikioh Mar 11, 2016

Contributor

At the moment, Dial runs the following processes serially for simplicity:

  1. multiple host and service discovery racers
  2. making a short list of target addresses
  3. multiple connection setup racers, and picking a single winner

In near future, when we want more performance on some circumstances, we probably run:

  1. multiple host/service discovery+connection setup racers
    • per address family, per service {name,port}, etc
  2. picking a single winner

For both cases, the Resolver interface needs to take information for host and service filters. Moreover, it would probably need certificates for supporting upcoming DNS over TLS and DANE.

Looks like this proposal makes it possible to place complicated DNS-related packages at x/net repository. I'm happy if we have fancy name/service discovery functionality without replumbing of packages in standard library.

Contributor

mikioh commented Mar 11, 2016

At the moment, Dial runs the following processes serially for simplicity:

  1. multiple host and service discovery racers
  2. making a short list of target addresses
  3. multiple connection setup racers, and picking a single winner

In near future, when we want more performance on some circumstances, we probably run:

  1. multiple host/service discovery+connection setup racers
    • per address family, per service {name,port}, etc
  2. picking a single winner

For both cases, the Resolver interface needs to take information for host and service filters. Moreover, it would probably need certificates for supporting upcoming DNS over TLS and DANE.

Looks like this proposal makes it possible to place complicated DNS-related packages at x/net repository. I'm happy if we have fancy name/service discovery functionality without replumbing of packages in standard library.

@anatol

This comment has been minimized.

Show comment
Hide comment
@anatol

anatol Mar 29, 2016

I am trying to run a network application at Android arm64 system and http.Get fails because of DNS resolution failed. It turned out Android uses custom dns resolver interface. https://android.googlesource.com/platform/bionic/+/master/libc/dns/net/gethnamaddr.c#564 An application opens /dev/socket/dnsproxyd socket and uses it to resolve names. I tried GODEBUG=netdns=cgo and for some reason it does not work on Android.

It would be nice if I can implement a custom dns resolver and tell my application to use it. Here is similar issue from another project [1].

[1] syncthing/syncthing-android#412 (comment)

anatol commented Mar 29, 2016

I am trying to run a network application at Android arm64 system and http.Get fails because of DNS resolution failed. It turned out Android uses custom dns resolver interface. https://android.googlesource.com/platform/bionic/+/master/libc/dns/net/gethnamaddr.c#564 An application opens /dev/socket/dnsproxyd socket and uses it to resolve names. I tried GODEBUG=netdns=cgo and for some reason it does not work on Android.

It would be nice if I can implement a custom dns resolver and tell my application to use it. Here is similar issue from another project [1].

[1] syncthing/syncthing-android#412 (comment)

@sajal

This comment has been minimized.

Show comment
Hide comment
@sajal

sajal Mar 30, 2016

Valid use-case for custom resolver, but have you tried using the Android NDK to build your binary?

sajal commented Mar 30, 2016

Valid use-case for custom resolver, but have you tried using the Android NDK to build your binary?

@jabley

This comment has been minimized.

Show comment
Hide comment
@jabley

jabley May 11, 2016

I'd similarly be interested in having timing information available, similar to time_* variables in curl. A monitoring tool that can periodically probe networks would be very handy.

Happy to open up a separate proposal if it's felt to be off-topic for this one?

jabley commented May 11, 2016

I'd similarly be interested in having timing information available, similar to time_* variables in curl. A monitoring tool that can periodically probe networks would be very handy.

Happy to open up a separate proposal if it's felt to be off-topic for this one?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz May 11, 2016

Member

@jabley, that already happened for Go 1.7. See #12580

Member

bradfitz commented May 11, 2016

@jabley, that already happened for Go 1.7. See #12580

@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Jul 19, 2016

Contributor

This needs a proper proposal document to move forward.

Contributor

adg commented Jul 19, 2016

This needs a proper proposal document to move forward.

@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Oct 3, 2016

Contributor

An extension to the work done in #16672

Contributor

adg commented Oct 3, 2016

An extension to the work done in #16672

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Oct 3, 2016

Member

In particular, this got submitted: https://go-review.googlesource.com/29440

Member

bradfitz commented Oct 3, 2016

In particular, this got submitted: https://go-review.googlesource.com/29440

@adg adg modified the milestones: Go1.9, Proposal Oct 31, 2016

@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Oct 31, 2016

Contributor

Need design work, but should be good for Go 1.9.

Contributor

adg commented Oct 31, 2016

Need design work, but should be good for Go 1.9.

@bradfitz bradfitz changed the title from proposal: net: allow Dialer to use custom resolver to net: allow custom Resolver method implementation(s) Oct 31, 2016

@polomsky

This comment has been minimized.

Show comment
Hide comment
@polomsky

polomsky Dec 5, 2016

I have another use case for which this feature would be usable. I need to make some app firewall. I have proxy which uses http.Request. Urls are specified by user and I want to block access to several ip addresses, e.g. localhost. Currently I do not know any way how to do it except usage of Host header. But it is not sufficient, because I can not specify multiple ip addresses for request (in case when DNS server returns more addresses for same name). So custom dns resolver (with removing wrong ips) would be very great.

polomsky commented Dec 5, 2016

I have another use case for which this feature would be usable. I need to make some app firewall. I have proxy which uses http.Request. Urls are specified by user and I want to block access to several ip addresses, e.g. localhost. Currently I do not know any way how to do it except usage of Host header. But it is not sufficient, because I can not specify multiple ip addresses for request (in case when DNS server returns more addresses for same name). So custom dns resolver (with removing wrong ips) would be very great.

@sajal

This comment has been minimized.

Show comment
Hide comment
@sajal

sajal Dec 5, 2016

@polomsky I am doing something similar. In my Transport I am using my own dialer which is basically clone of the default dialer but applies the IP policy before making the connection usable. The drawback is a TCP connection was made but the client cant do anything with it if islocalip returns true.

sajal commented Dec 5, 2016

@polomsky I am doing something similar. In my Transport I am using my own dialer which is basically clone of the default dialer but applies the IP policy before making the connection usable. The drawback is a TCP connection was made but the client cant do anything with it if islocalip returns true.

@sajal

This comment has been minimized.

Show comment
Hide comment
@sajal

sajal Dec 16, 2016

So, currently net.Dialer accepts a net.Resolver which is a struct and not an interface. As I understand it, there is no way to override the Lookup* methods. Are there any plans to make it an interface? Just like how http.Client uses http.RoundTripper interface instead of explicit http.Transport struct.

sajal commented Dec 16, 2016

So, currently net.Dialer accepts a net.Resolver which is a struct and not an interface. As I understand it, there is no way to override the Lookup* methods. Are there any plans to make it an interface? Just like how http.Client uses http.RoundTripper interface instead of explicit http.Transport struct.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Dec 16, 2016

Member

@sajal, yes, we left room for in the design for that:

// TODO(bradfitz): optional interface impl override hook

Member

bradfitz commented Dec 16, 2016

@sajal, yes, we left room for in the design for that:

// TODO(bradfitz): optional interface impl override hook

@sheerun

This comment has been minimized.

Show comment
Hide comment
@sheerun

sheerun May 26, 2017

@bradfitz I'd like to point out that it would be nice if LookupPort could accept a hostname to connect to as well. For now it seems you expect port number as service argument, as shown for example by ReverseProxy implementation.

Use Case: Resolving to custom port with ReverseProxy (e.g. example.com should go to 127.0.0.1:3456)

sheerun commented May 26, 2017

@bradfitz I'd like to point out that it would be nice if LookupPort could accept a hostname to connect to as well. For now it seems you expect port number as service argument, as shown for example by ReverseProxy implementation.

Use Case: Resolving to custom port with ReverseProxy (e.g. example.com should go to 127.0.0.1:3456)

@mikegleasonjr

This comment has been minimized.

Show comment
Hide comment
@mikegleasonjr

mikegleasonjr May 31, 2017

Hi guys, just wondering if go 1.9 beta drops tomorrow would it means this feature won't make its way into go 1.9?

mikegleasonjr commented May 31, 2017

Hi guys, just wondering if go 1.9 beta drops tomorrow would it means this feature won't make its way into go 1.9?

@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor May 31, 2017

Contributor

@mikegleasonjr As far as I know nobody has written the code for this. At this point I think it is unlikely to get into Go 1.9, but it is still possible if somebody sends in a patch very soon.

Contributor

ianlancetaylor commented May 31, 2017

@mikegleasonjr As far as I know nobody has written the code for this. At this point I think it is unlikely to get into Go 1.9, but it is still possible if somebody sends in a patch very soon.

@excavador

This comment has been minimized.

Show comment
Hide comment
@excavador

excavador Sep 1, 2017

I definitely net some simple way to hook LookupIP / LookupHost / LookupAddr for unit-testing purposes

excavador commented Sep 1, 2017

I definitely net some simple way to hook LookupIP / LookupHost / LookupAddr for unit-testing purposes

@ianlancetaylor ianlancetaylor modified the milestones: Go1.10, Unplanned Dec 8, 2017

@mauricio

This comment has been minimized.

Show comment
Hide comment
@mauricio

mauricio Jan 19, 2018

So is there something being planned for this? With net.Resolver being a struct it's still impossible to collect any data about what is going on internally on the DNS layer.

mauricio commented Jan 19, 2018

So is there something being planned for this? With net.Resolver being a struct it's still impossible to collect any data about what is going on internally on the DNS layer.

@benburkert

This comment has been minimized.

Show comment
Hide comment
@benburkert

benburkert Jan 19, 2018

Contributor

It's possible to intercept DNS requests generated by the net package by modifying the Dial func of DefaultResolver. For example, this package automatically adds cacheing of DNS queries: https://godoc.org/github.com/benburkert/dns/init

Contributor

benburkert commented Jan 19, 2018

It's possible to intercept DNS requests generated by the net package by modifying the Dial func of DefaultResolver. For example, this package automatically adds cacheing of DNS queries: https://godoc.org/github.com/benburkert/dns/init

@polomsky

This comment has been minimized.

Show comment
Hide comment
@polomsky

polomsky Jan 22, 2018

It's possible to intercept DNS requests generated by the net package by modifying the Dial func of DefaultResolver

@benburkert it works only for platforms where build in go resolver is present. E.g. unusable for windows.

polomsky commented Jan 22, 2018

It's possible to intercept DNS requests generated by the net package by modifying the Dial func of DefaultResolver

@benburkert it works only for platforms where build in go resolver is present. E.g. unusable for windows.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jun 1, 2018

Change https://golang.org/cl/115855 mentions this issue: net: Implement hooks to override Resolver's lookup methods

gopherbot commented Jun 1, 2018

Change https://golang.org/cl/115855 mentions this issue: net: Implement hooks to override Resolver's lookup methods

@jfesler

This comment has been minimized.

Show comment
Hide comment
@jfesler

jfesler Jun 22, 2018

Override functions as in https://golang.org/cl/115855 would be "ok". Doesn't seem consistent with other things that use interfaces; but it would allow for shenanigans (dns lookup metrics; alternatives to DNS; caching; and so forth). This would also allow people to avoid custom dialers; instead using the battle tested dual stack happy eyeball code.

jfesler commented Jun 22, 2018

Override functions as in https://golang.org/cl/115855 would be "ok". Doesn't seem consistent with other things that use interfaces; but it would allow for shenanigans (dns lookup metrics; alternatives to DNS; caching; and so forth). This would also allow people to avoid custom dialers; instead using the battle tested dual stack happy eyeball code.

@mwiora

This comment has been minimized.

Show comment
Hide comment
@mwiora

mwiora Aug 20, 2018

hi all,

this topic seems to be implemented and still open so I'd like to try it ;)
As described on stackoverflow I would like to directly use the new resolver configuration options.

The only option I miss is to avoid the fallback to another DNS - I thought about a switch.

mwiora commented Aug 20, 2018

hi all,

this topic seems to be implemented and still open so I'd like to try it ;)
As described on stackoverflow I would like to directly use the new resolver configuration options.

The only option I miss is to avoid the fallback to another DNS - I thought about a switch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment