Environments with a high volume of DNS queries will drop them from time to time

[aemneina]

Rancher versions:
rancher/server or rancher/rancher: 1.6.21 (this impacts 2.x as well)
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)

Description of the problem:
Weave has some great research done here,
https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts

Results:
Not sure if we need to move away from kube-dns behind a service IP, but thats for architects to decide.

[Oats87]

If kube-dns were to be run as a daemonset, the kubelet could be configured to configure pods to forward DNS queries to the local node's kube-dns rather than traversing through the service IP, it should be possible to avoid having the kernel drop DNS query packets.

[superseb]

kubernetes/kubernetes#56903

[aemneina]

@superseb @alena1108, @Oats87 pointed me to Node Local DNS, which should be a solid solution. Forces dns lookups over TCP. I believe it's alpha in 1.13.x
kubernetes/kubernetes#70555

[alena1108]

@aemneina that would be the best solution given it's officially supported by k8s. And that k8s 1.13 was released today, 12/03/2018 :)