-
Notifications
You must be signed in to change notification settings - Fork 39.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DNS latency of 5s when uses iptables forward in pods network traffic #62628
Comments
/sig network |
@xiaoxubeii Could you elaborate a bit more on this behavior? Is this a bug in netfilter? Or is this a bug in kube-proxy that it doesn't follow certain standard while using netfilter? Thanks. cc @bowei |
I am by the way experiencing the same thing. With Kubernetes 1.10+CoreOS+Weave+CoreDNS/kube-dns+kube-proxy ipvs, I see constant 5s latency on DNS resolution. tcpdump shows that the first AAAA requests get lost somehow: https://hastebin.com/banulayire.swift. With |
Most of the comments relate to things which will cause intermittent packet loss, but OP seems to be talking about a consistent symptom - every time you do the request it will drop the same packet. Am I understanding the OP correctly? I can’t imagine what would cause it to drop the last packet. How would it know it’s the last one? |
@bboreham The blog post I linked above explains the issue very well. It's a race condition with conntrack/SNAT. glibc/musl are very good at triggering it when sending A/AAAA lookups in parallel. Using |
@MrHohn @bboreham @Quentin-M I think the problem that i met is about race condition on
and we send DNS requests in pods, the second DNS requests would be received while the first one is still not confirmed and both DNS requests would have an unconfirmed conntrack. So the second one would be dropped in So a simple solution is to use
It is more like a netfilter bug or flaw, but i think the kube-dns also need do something to avoid it. |
Yes, this is what's described there. AFAIK, real solution would be to patch kubelet, kube-proxy and the overlay networks (flannel, weave, calico, etc), adding But I agree that your patch, while slowing down DNS lookups a little (well it's not as bad as 5secs+!), is simple and effective. People in other threads have also been mentioning deployment initializers, using dnsPolicy=None in Kubernetes 1.10 or manually mounting /etc/resolv.conf - but I'd rather not force cluster users' to apply such workaround, whereas it's an actual infrastructure issue. Or it's a totally different issue? |
It’s great you are all agreed what is the cause, but still: a race condition would randomly cause or not cause the problem. So, was it consistent - happened every time, or occasional - sometimes happened? |
Fair point. I wonder if something like this may be happening instead? I am not that familiar with networking to be able to tell. |
@Quentin-M I am not sure which one causes this problem, @bboreham In my case, it is consistent. It always drop AAAA packet. |
The inestimable @brb has found another race condition, which will tend to cause the last packet to be dropped. weaveworks/weave#3287 (comment) |
I would just like to add here that the
I have reached out on the freenode's #musl channel, but unfortunately it does not seem like there is much desire to add support for the option:
|
/close |
I just posted a little write-up about our journey troubleshooting the issue, and how we are worked around it in production: https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/. |
@xiaoxubeii @Quentin-M what is the current favorite workaround for this? We run flannel with vxlan and have here and there bleeps in out monitoring spike dns request up to 5s. One time probably (not 100% sure) we had a production incident, because of that. Should I port the script referenced by @Quentin-M to flannel or is there already something else? |
FYI: flannel-io/flannel#1001 (comment) |
…hat the kernel race condition happens less likely. Background see https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/ kubernetes/kubernetes#62628 https://github.com/coreos/flannel/pull/1001\#issuecomment-403802099
@xiaoxubeii why is this issue closed? @brb has fixed one of the kernel races but other causes remain. |
coded a fix for musl on Alpine Linux 3.7, which removed AAAA query by default (AF_UNSPEC). see #56903 (comment) thanks, |
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
The DNS will get a 5s latency of AAAA when uses iptables forward in network traffic between pods.
What you expected to happen:
No latency.
How to reproduce it (as minimally and precisely as possible):
The cluster cidr is 172.20.80.0/24, gw is current node. Cluster, pods and nodes are in l2 network using VXLAN.
Anything else we need to know?:
If cni gw of cluster cidr is current node, the network traffic between pods and services will use iptables forward:
If enable forwarding conntrack, netfilter will drop first AAAA record packet when requests dns. It will cause dns latency of 5s.
Environment:
kubectl version
): v1.9.2uname -a
): 3.10.0-327.18.2.el7.x86_64The text was updated successfully, but these errors were encountered: