Enable NodeLocal DNSCache

[sagor999]

Is your feature request related to a problem? Please describe

https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/
Why we need this?
It will greatly improve our dns performance as well as dns resiliency.
Majority of dns requests from each node will stay on the node itself by hitting local cache, greatly reducing pressure on coredns, as well udp traffic to coredns (that sometimes might get dropped, lost).
This also reduces pressure on conntrack table, since conntrack would not need to track those requests now.

Describe the behaviour you'd like

Describe alternatives you've considered

Additional context

[aledbf]

It will greatly improve our dns performance as well as dns resiliency.

We need metrics about the current state

This also reduces pressure on conntrack table, since conntrack would not need to track those requests now.

Before this route, we need to know the size of the conntrack table.

Why?

• We don't use services (ws-proxy do not use DNS but the IP address of the POD)
• Adding nodelocal DNS adds additional load to every single node in the cluster

[sagor999]

Our current conntrack table usage at peak from last two weeks is about 60%, though it is coming from meta cluster:
topk(20, node_nf_conntrack_entries/node_nf_conntrack_entries_limit)

I agree though this is a big change and probably we can hold off on this one for now.

[kylos101]

Removing this from scheduled work for now, moved to the "Cluster and Node Improvements" epic. I consider that to be a short lived backlog. @aledbf @sagor999 we things end up rotting there I'll plan to close them. But this way, they exit our inbox.

[sagor999]

Yeah, I think this one can be closed for the time being. If we will start seeing high load on coredns pods, or excessive latency for dns queries, then we can return back to exploring using nodelocal dns cache for that.
So I will go ahead and close this issue, we can always re-open it in the future.