Repository navigation
NodeLocal DNS cache breaks connection tracking, conflicts with Calico #98758
Description
Activity
@fasaxc: This issue is currently awaiting triage.
If a SIG or subproject determines this is a relevant issue, they will accept it by applying the
triage/acceptedlabel and provide further guidance.The
triage/acceptedlabel can be added by org members by writing/triage acceptedin a comment.Details
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
/sig network
Can we get a bit more detail about which leg of DNS is being impacted? The rationale for NOTRACK is that connection-tracking + UDP + non-reused sockets == LOTS of conntrack records that serve literally no value but are all stuck waiting to expire => unhappy users.
Can we get a bit more detail about which leg of DNS is being impacted? The rationale for NOTRACK is that connection-tracking + UDP + non-reused sockets == LOTS of conntrack records that serve literally no value but are all stuck waiting to expire => unhappy users.
Correct. NOTRACK was added as a feature to save on conntrack table entries and also avoid DNATs which had some race conditions leading to packet drops.
I think a similar issue with NetworkPolicy was reported with NodeLocalDNS and we documented it in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/README.md#network-policy-and-dns-connectivityCan we get more details on which part of the traffic was being dropped? is it the nodelocaldns to client pod response?
In #3795, users use calico GlobalNetworkPolicy and only projectcalico/calico#3795 (comment) without it.
What do we do with this issue?
We had this come up elsewhere and it was confirmed that the rule in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/README.md#network-policy-and-dns-connectivity was working.
Given that the node-local-dns behavior depends on allowing untracked connections, we would need this additional config.
Can we check if that works in your setup too @fasaxc ?
/assign @fasaxc- added a commit that references this issue
on Feb 19, 2021 Sorry, I need to replicate the user's environment to get the next level of diags; haven't got around to doing that yet.
Actually this is is Working as Intended.. so if configuring the rule documented in https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/README.md#network-policy-and-dns-connectivity works, then this can be closed, IMO.
... ok, so shall we close?
ok, let's close it. Please reopen if configuring the additional rule does not work.
/close
@prameshj: Closing this issue.
Details
In response to this:
ok, let's close it. Please reopen if configuring the additional rule does not work.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
What happened:
the node local DNS cache uses NOTRACK rules in iptables and these disable connection tracking. This is incompatible with Calico, which relies on connection tracking to implement its flow-based firewall. Since the DNS traffic gets hit with NOTRACK, the response packets are dropped because they're not part of a flow.
What you expected to happen:
DNS traffic conntracked as normal.
How to reproduce it (as minimally and precisely as possible):
This came from a user report, I think that what's needed is:
Anything else we need to know?:
Environment:
kubectl version): 1.17.4cat /etc/os-release):uname -a):