Node-local DNS resolver often responds extremely slowly #1659
Description
Bug report
What is the problem?
DNS queries to the DNS resolver on the node are often taking a long while to be answered.
I am repeatedly executing the following on a client connected to the node:
dig @fd4e:f2d7:88d2:ffff::1 ralfj.de
where fd4e:f2d7:88d2:ffff::1 is the "next node" anycast IP. This command sometimes comes back immediately, sometimes it comes back after several seconds, and sometimes it times out after several seconds (all of these cases seem very roughly equally likely).
After the initial query, even when the answer takes several seconds, there is no DNS traffic on the tunnel between the node and the gateway. The link between the client and the node is a Linux bridge; both of these are just VMs on the same host and hence there should be 0 packet loss. I see the same behavior when talking to a real node over a WiFi link.
When I do nslookup ralfj.de on the node locally, the answer always comes back immediately. Also when I do dig @fd4e:f2d7:88d2:ffff::101 ralfj.de, which sends the query to one of our gateways, the answer comes back immediately -- so the network connection(s) all seem fine.
What is the expected behaviour?
The DNS reply should come back with decent latency (<50ms, say).
Gluon Version:
v2018.2
Site Configuration:
Custom patches:
None.