NodeLocal DNS cache breaks connection tracking, conflicts with Calico

[fasaxc]

What happened:

the node local DNS cache uses NOTRACK rules in iptables and these disable connection tracking. This is incompatible with Calico, which relies on connection tracking to implement its flow-based firewall. Since the DNS traffic gets hit with NOTRACK, the response packets are dropped because they're not part of a flow.

What you expected to happen:

DNS traffic conntracked as normal.

How to reproduce it (as minimally and precisely as possible):

This came from a user report, I think that what's needed is:

• Install a system with Calico for CNI.
• Add some network policy to a pod that allows egress DNS but disallows ingress.
• DNS requests from the pod fail.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.17.4
• Cloud provider or hardware configuration: Centos/Ubuntu
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug): Calico v3.14.1
• Others:

[k8s-ci-robot]

@fasaxc: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fasaxc]

/sig network

[thockin]

Can we get a bit more detail about which leg of DNS is being impacted? The rationale for NOTRACK is that connection-tracking + UDP + non-reused sockets == LOTS of conntrack records that serve literally no value but are all stuck waiting to expire => unhappy users.

[prameshj]

Can we get a bit more detail about which leg of DNS is being impacted? The rationale for NOTRACK is that connection-tracking + UDP + non-reused sockets == LOTS of conntrack records that serve literally no value but are all stuck waiting to expire => unhappy users.

Correct. NOTRACK was added as a feature to save on conntrack table entries and also avoid DNATs which had some race conditions leading to packet drops.
I think a similar issue with NetworkPolicy was reported with NodeLocalDNS and we documented it in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/README.md#network-policy-and-dns-connectivity

Can we get more details on which part of the traffic was being dropped? is it the nodelocaldns to client pod response?

[pacoxu]

In #3795, users use calico GlobalNetworkPolicy and only projectcalico/calico#3795 (comment) without it.

[thockin]

What do we do with this issue?

[prameshj]

We had this come up elsewhere and it was confirmed that the rule in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/README.md#network-policy-and-dns-connectivity was working.

Given that the node-local-dns behavior depends on allowing untracked connections, we would need this additional config.
Can we check if that works in your setup too @fasaxc ?
/assign @fasaxc

[thockin]

Ping @fasaxc

[fasaxc]

Sorry, I need to replicate the user's environment to get the next level of diags; haven't got around to doing that yet.

[thockin]

@prameshj Are we tracking this as a bug, as a feature request, or is it still needing triage?

[prameshj]

Actually this is is Working as Intended.. so if configuring the rule documented in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/README.md#network-policy-and-dns-connectivity works, then this can be closed, IMO.

[jayunit100]

... ok, so shall we close?

[prameshj]

ok, let's close it. Please reopen if configuring the additional rule does not work.

/close

[k8s-ci-robot]

@prameshj: Closing this issue.

In response to this:

ok, let's close it. Please reopen if configuring the additional rule does not work.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.