DNS latency of 5s when uses iptables forward in pods network traffic

[xiaoxubeii]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:
The DNS will get a 5s latency of AAAA when uses iptables forward in network traffic between pods.

What you expected to happen:
No latency.

How to reproduce it (as minimally and precisely as possible):

• CNI configuration

        "name": "mynet",
        "type": "macvlan",
	"master": "eth0",
        "ipam": {
                "type": "host-local",
                "subnet": "172.20.0.0/17",
		"rangeStart": "172.20.64.129",
		"rangeEnd": "172.20.64.254",
		"gateway": "172.20.127.254",
		"routes": [
			{"dst":"0.0.0.0/0"},
			{"dst":"172.20.80.0/24", "gw":"172.20.0.62"}
		]
        }
}

• Network Architecture
  The cluster cidr is 172.20.80.0/24, gw is current node. Cluster, pods and nodes are in l2 network using VXLAN.

Anything else we need to know?:
If cni gw of cluster cidr is current node, the network traffic between pods and services will use iptables forward:

-P FORWARD ACCEPT
-A FORWARD -m comment --comment "kubernetes forward rules" -j KUBE-FORWARD

-N KUBE-FORWARD
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 172.20.0.0/17 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 172.20.0.0/17 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

If enable forwarding conntrack, netfilter will drop first AAAA record packet when requests dns. It will cause dns latency of 5s.

Environment:

• Kubernetes version (use kubectl version): v1.9.2
• Cloud provider or hardware configuration: None
• OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core)
• Kernel (e.g. uname -a): 3.10.0-327.18.2.el7.x86_64
• Install tools: kubeadm
• Others:

[xiaoxubeii]

/sig network
/assign

[MrHohn]

If enable forwarding conntrack, netfilter will drop last AAAA record packet when requests dns

@xiaoxubeii Could you elaborate a bit more on this behavior? Is this a bug in netfilter? Or is this a bug in kube-proxy that it doesn't follow certain standard while using netfilter? Thanks.

cc @bowei

[Quentin-M]

I am by the way experiencing the same thing. With Kubernetes 1.10+CoreOS+Weave+CoreDNS/kube-dns+kube-proxy ipvs, I see constant 5s latency on DNS resolution. tcpdump shows that the first AAAA requests get lost somehow: https://hastebin.com/banulayire.swift. With single-request or single-request-reopen, the issue is gone.

[Quentin-M]

Actually, relevant reads:

• https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02
• DNS lookup timeouts due to races in conntrack weaveworks/weave#3287
• https://www.spinics.net/lists/netfilter-devel/msg15860.html
• https://patchwork.kernel.org/patch/9785641/

[bboreham]

Most of the comments relate to things which will cause intermittent packet loss, but OP seems to be talking about a consistent symptom - every time you do the request it will drop the same packet.

Am I understanding the OP correctly?

I can’t imagine what would cause it to drop the last packet. How would it know it’s the last one?

[Quentin-M]

@bboreham The blog post I linked above explains the issue very well. It's a race condition with conntrack/SNAT. glibc/musl are very good at triggering it when sending A/AAAA lookups in parallel. Using single-request-reopen works around the issue by making serializing the queries. One better fix (as documented) is to add --random-fully to every MASQUERADE rules (kubelet, kube-proxy, overlay).

[xiaoxubeii]

@MrHohn @bboreham @Quentin-M I think the problem that i met is about race condition on
conntrack insertions. I used the node as the gateway, so it would redirects packets from pods to services. When iptables sets:

-A KUBE-FORWARD -s 172.20.0.0/17 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 172.20.0.0/17 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

and we send DNS requests in pods, the second DNS requests would be received while the first one is still not confirmed and both DNS requests would have an unconfirmed conntrack. So the second one would be dropped in nf_conntrack_confirm, which results in an DNS timeout and retransmit.

So a simple solution is to use single-request-reopen:

single-request-reopen (since glibc 2.9)
                     Sets RES_SNGLKUPREOP in _res.options.  The resolver
                     uses the same socket for the A and AAAA requests.  Some
                     hardware mistakenly sends back only one reply.  When
                     that happens the client system will sit and wait for
                     the second reply.  Turning this option on changes this
                     behavior so that if two requests from the same port are
                     not handled correctly it will close the socket and open
                     a new one before sending the second request.

It is more like a netfilter bug or flaw, but i think the kube-dns also need do something to avoid it.