Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: DNS timeouts can alter the IPv4/IPv6-ity of LookupIP #17448

Closed
pmarks-net opened this issue Oct 15, 2016 · 19 comments

Comments

Projects
None yet
6 participants
@pmarks-net
Copy link
Contributor

commented Oct 15, 2016

When using the non-cgo DNS resolver, the following program:

addrs, err := net.LookupIP("kernel.org.")
fmt.Printf("addrs=%v\nerr=%v\n", addrs, err)

Outputs kernel.org's IPv4/IPv6 addresses, as expected:

addrs=[2620:3:c000:a:0:1991:8:25 2001:4f8:1:10:0:1991:8:25 199.204.44.194 149.20.4.69 198.145.20.140]
err=<nil>

However, if you blackhole AAAA responses, the output becomes IPv4-only:

$ sudo iptables -I INPUT -p udp -m string --algo kmp --hex-string '|06 6b 65 72 6e 65 6c 03 6f 72 67 00 00 1c 00 01|' -j DROP
$ go run test.go
addrs=[149.20.4.69 198.145.20.140 199.204.44.194]
err=<nil>

And vice versa for A responses:

$ sudo iptables -I INPUT -p udp -m string --algo kmp --hex-string '|06 6b 65 72 6e 65 6c 03 6f 72 67 00 00 01 00 01|' -j DROP
$ go run test.go
addrs=[2620:3:c000:a:0:1991:8:25 2001:4f8:1:10:0:1991:8:25]
err=<nil>

If you replace INPUT with OUTPUT, the timeouts become socket errors, and the same thing happens, but more quickly.

Essentially, the problem is that LookupIP presents the pair of queries as a single transaction, but internally, failures are non-atomic, so a flaky network can cause the function to report success, but only provide half of the addresses. The fix is to return a failure instead of a half-success.

At Google, there have been cases involving other resolvers (b/31811300) where network flakiness causes a client to make an incorrect address selection decision, leading to minor outages, confused engineers, and/or ugly workarounds. I haven't directly observed problems with Go's resolver in particular, but I'd like to preemptively fix this while it's still theoretical.

The relevant dnsclient_unix.go changes would be:

  • In tryOneName, distinguish socket errors from DNS protocol errors. My best idea is to set IsTimeout=true for all socket errors, because IsTemporary is already used to indicate SERVFAIL.
  • In goLookupIPOrder, cause any Timeout() error to abort the nameList loop, discard all addresses, and fail immediately.

(Interestingly, getaddrinfo also exhibits this problem, but asymmetrically: dropping AAAA replies causes a name to become IPv4-only, but dropping A replies returns an error.)

@bradfitz

This comment has been minimized.

Copy link
Member

commented Oct 15, 2016

My best idea is to set IsTimeout=true for all socket errors, because IsTemporary is already used to indicate SERVFAIL

I don't remember the code at the moment, but that sounds perhaps a bit hacky. I'd rather not overload the meanings of IsTimeout and IsTemporary to mean other things.

In any case, the Go 1.8 freeze is fast approaching. Is this something you want to work on in the next couple weeks? I think @mdempsky and I at least are racing to finish up other stuff.

/cc @mikioh too.

@bradfitz bradfitz added this to the Go1.8Maybe milestone Oct 15, 2016

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Oct 15, 2016

There are environments where real queries fail like this. I don't think
it's wise to prevent A lookups from succeeding if AAAA lookups fail. What
is the actual harm if the non-preferred address family is used?

I'd also like to see a summary of how the popular system resolvers behave.

On Oct 15, 2016 03:17, "Brad Fitzpatrick" notifications@github.com wrote:

My best idea is to set IsTimeout=true for all socket errors, because
IsTemporary is already used to indicate SERVFAIL

I don't remember the code at the moment, but that sounds perhaps a bit
hacky. I'd rather not overload the meanings of IsTimeout and IsTemporary
to mean other things.

In any case, the Go 1.8 freeze is fast approaching. Is this something you
want to work on in the next couple weeks? I think @mdempsky
https://github.com/mdempsky and I at least are racing to finish up
other stuff.

/cc @mikioh https://github.com/mikioh too.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#17448 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAHEMZrByoiL6Lo8XC61Z3OdUwAEACLkks5q0H4XgaJpZM4KXkul
.

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2016

I think it's fine to make the newly added methods of Resolver in Go 1.8 more deterministic and configurable for people who want to use Lookup APIs deterministically such as telco operators, but am not keen to change the default "non-deterministic" behavior because IP infrastructure consumers usually don't take care with details of underlying stuff, they care about only the returned error value and result.

Fortunately we've not seen reports like "exchanging multiple different RRs on multiple DNS transports is inefficient" or "exchanging multiple different RRs on a single DNS transport makes some L4-7 appliance malfunction" yet. Maybe it's a good timing to start improving the existing builtin DNS stub resolver, though I guess it requires some blueprint.

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 17, 2016

There are environments where real queries fail like this

I'm curious, do you have actual examples of this? I've heard of cases where AAAA queries return REFUSED or SERVFAIL, but no response at all is pretty harsh.

What is the actual harm if the non-preferred address family is used?

For net.Dial, it's not a big deal because the DNS and TCP connections are part of the same operation, so a flaky network could just as easily steer the TCP connection toward a non-preferred family, and DNS must be re-queried if the operation fails.

The harm arises when applications use DNS responses in less-obvious ways. For example, imagine a connection algorithm that says:

  1. Query DNS until it succeeds
  2. Make TCP connections until it succeeds

Where the operation can only be restarted by killing the process. If the client is IPv4-only or IPv6-only, then it could get stuck connecting to a non-viable address family.

Or, consider an application that generates a unique ID by querying DNS, selecting one result with a fixed IPv4 or IPv6 bias. DNS flakiness could alter the answer, when it would be more correct for an incomplete result to fail.

I'm not trying to argue that the above are ideal designs, but they work 99.9...% of the time, and the principle of least astonishment would suggest adding more nines.

I'd also like to see a summary of how the popular system resolvers behave.

Can you suggest a list? Note that only resolvers exposing "Lookup A+AAAA" as a single operation are in scope for this issue.

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2016

I'm curious, do you have actual examples of this? I've heard of cases where AAAA queries return REFUSED or SERVFAIL, but no response at all is pretty harsh.

It's prevalent enough that Ubuntu documentation talks about it: https://help.ubuntu.com/community/WebBrowsingSlowIPv6IPv4

Also lots of people online talking about it

Can you suggest a list? Note that only resolvers exposing "Lookup A+AAAA" as a single operation are in scope for this issue.

Well, the obvious starting set is Darwin, recent GNU libc, and Windows.

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 17, 2016

I mentioned this previously, but glibc's behavior appears to be:

  • AAAA timeout means "no data".
  • A timeout means "fail".

For example:

$ ldd --version
ldd (Gentoo 2.23-r2 p4) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

$ sudo iptables -I INPUT -p udp -m string --algo kmp --hex-string '|06 6b 65 72 6e 65 6c 03 6f 72 67 00 00 1c 00 01|' -j DROP
$ time getent ahosts kernel.org
149.20.4.69     STREAM kernel.org
149.20.4.69     DGRAM  
149.20.4.69     RAW    
199.204.44.194  STREAM 
199.204.44.194  DGRAM  
199.204.44.194  RAW    
198.145.20.140  STREAM 
198.145.20.140  DGRAM  
198.145.20.140  RAW    
real    0m15.013s
user    0m0.000s
sys     0m0.000s
$ sudo iptables -D INPUT -p udp -m string --algo kmp --hex-string '|06 6b 65 72 6e 65 6c 03 6f 72 67 00 00 1c 00 01|' -j DROP

$ sudo iptables -I INPUT -p udp -m string --algo kmp --hex-string '|06 6b 65 72 6e 65 6c 03 6f 72 67 00 00 01 00 01|' -j DROP
$ time getent ahosts kernel.org
real    0m15.071s
user    0m0.001s
sys     0m0.001s
$ sudo iptables -D INPUT -p udp -m string --algo kmp --hex-string '|06 6b 65 72 6e 65 6c 03 6f 72 67 00 00 01 00 01|' -j DROP
@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 19, 2016

Alright, I spun up Windows Server 2016 in a VM, deleted the "disable IPv6" cruft, and forwarded DNS through my Linux VPS. Here's how net.LookupIP behaves with A, AAAA, and both blackholed:

C:\Users\pmarks\Desktop\go>go run test.go
addrs=[2620:3:c000:a:0:1991:8:25 2001:4f8:1:10:0:1991:8:25]
err=<nil>

C:\Users\pmarks\Desktop\go>go run test.go
addrs=[199.204.44.194 198.145.20.140 149.20.4.69]
err=<nil>

C:\Users\pmarks\Desktop\go>go run test.go
addrs=[]
err=lookup kernel.org.: getaddrinfow: No such host is known.

This... isn't helping my case, is it?

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 20, 2016

I don't have a good way to test macOS, but we've established that my proposal would in fact be stricter than both the Linux/Windows system resolvers.

I see that there is a type Resolver available where we could introduce a flag to control the strictness, and paranoia would suggest that net.Dial continue to use the non-strict version (because it's relatively well-behaved when addresses go missing).

But which should be the default for net.LookupIP? Should users have to set a flag to tolerate broken DNS infrastructure, or set a flag to improve address stability?

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Oct 21, 2016

I am firmly in the camp that "tolerate broken DNS infrastructure" should remain the default. Or at least "do the same thing as the system resolver".

I have a Mac to test on but I'm not sure the best way to block the AAAA queries.

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 21, 2016

I have a Mac to test on but I'm not sure the best way to block the AAAA queries.

Here's how I managed to test Windows:

  • Get Unbound running on a Linux box.
  • Use the Linux box as your DNS resolver.
  • Use the iptables rules in my first post to filter kernel.org A/AAAA queries
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Oct 22, 2016

You can use PF on various BSD variants including macOS, OS X by default nowadays.

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2016

@pmarks-net,

Can you please elaborate on to improve address stability?

I thought that you wanted stuff like https://tools.ietf.org/html/draft-wkumari-dnsop-multiple-responses or some experimental implementation as a solution to https://tools.ietf.org/html/draft-ietf-dnsop-no-response-issue for in-house tools. But it looks like you want more. I'm still not sure whether your proposal would work well in the wild, also DNS is the best tool to address issues you are assuming.

@quentinmit,

Or at least "do the same thing as the system resolver".

I'm not sure it would be a help because A and AAAA queries handling varies in each getaddrinfo implementation. IIRC, in early 00s developers realized that following simply RFC 3484 doesn't solve IPv6-IPv4 fallback issues in the wild; see RFC 4074, 4472. As a result, some implementation checks IP routing information/connectivity before querying to reduce unnecessary message exchange, some queries A-record first, some experimental makes a timeout for AAAA-query by using previously succeeded RTT of A-query/response, and perhaps others might do something similar or more complicated things. Moreover we can see AAAA filter in the wild as the side effect of World IPv6 Day/Launch.

I feel like adding a few control knobs is fine, but am not keen on changing the default behavior.

"tolerate broken DNS infrastructure" should remain the default.

Agree.

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2016

improve address stability was simply a shorthand for the A/AAAA-only timeout problem stated previously. If a name has a fixed set of addresses in DNS, then LookupIP should either return that set, or fail.

What I'll probably do is add a StrictTimeoutsOrSomething flag to type Resolver, so that users who need stability can enable it.

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2016

If a name has a fixed set of addresses in DNS

I'm a bit confused. What does this mean?

I believe you are not talking about RRset because A and AAAA are different types. Also as defined in RFC 4472, DNS transports and DNS records are independent. I think it's impossible to determine what kind of correlation exists between different class/type RRs without retrieving extra information like I-Ds mentioned above. Am I missing something?

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2016

I'm a bit confused. What does this mean?

I'm confused by your confusion. I'm describing a completely-ordinary DNS name, with A and AAAA records, whose values are not changing:

$ host example.com
example.com has address 93.184.216.34
example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946

In this case, LookupIP would ideally either return [93.184.216.34, 2606:2800:220:1:248:1893:25c8:1946] (in some order), or fail completely. With the current implementation, mere packet loss can change the result to [93.184.216.34], which is suboptimal.

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2016

Well, a fixed set of addresses in DNS doesn't mean addresses exchanged in DNS semantics? All query-response exchanges and transports are independent in DNS semantics, but API is free from the semantics. And you simply want to detect a failure of DNS transport and make Lookup/Resolve APIs return early for some reasons. Is my understanding correct?

[edit]

I guess that the new behavior would be intolerant of delegation (including lame delegation) and RRs move. It that okay?

I wrote above for another project but pasted it wrongly into this issue, but it looks related to this issue by synchronisity.

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2016

Right, this change would only affect DNS transport failures, due to a timeout or socket error. Any situation where the DNS server actually responds (empty RR set, NXDOMAIN, lame delegation, SERVFAIL, REFUSED, etc.) would maintain the existing behavior.

@gopherbot

This comment has been minimized.

Copy link

commented Nov 2, 2016

CL https://golang.org/cl/32572 mentions this issue.

@pmarks-net

This comment has been minimized.

Copy link
Contributor Author

commented Dec 3, 2016

Further experience with b/31811300 has shown that (at least our own) resolvers can spuriously return SERVFAIL with low probability, and trigger the same problems as a timeout.

This implies that the strict error handler should actually be watching for Temporary (which is a superset of Timeout), and that socket-level errors can be sanely reported as Temporary. This means that socket errors and SERVFAIL would set the same flag, but that seems less hacky than reporting a socket error as a timeout.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.