[Known Issue] Private DNS with .local entries won't work after Kuberentes 1.18/Ubuntu 18. #2052

paulgmiller · 2021-01-13T00:31:25Z

If your vitual network has custom dns server and uses a dns record that ends with .local then it will no longer work after your nodes go up to to ubuntu 18. This a happens automatically when you upgrade to k8s 1.17.
This is because ubuntu 18 usses syste
https://askubuntu.com/questions/917784/systemd-resolved-does-not-query-dns-server-for-local-domain
https://www.man7.org/linux/man-pages/man8/systemd-resolved.service.8.html

lookups for domains with the ".local" suffix are
not routed to DNS servers, unless the domain is specified
explicitly as routing or search domain for the DNS server and
interface. This means that on networks where the ".local"
domain is defined in a site-specific DNS server, explicit
search or routing domains need to be configured to make
lookups work within this DNS domain. Note that these days,
it's generally recommended to avoid defining ".local" in a
DNS server, as RFC6762[2] reserves this domain for exclusive
MulticastDNS use.

Temporary mitigations:
You can leave one agentpool behind on 1.17 but eventually you will need to use a different record in your private dns server.
Changing the /etc/resolve.conf symlink to point at the static /run/systemd/resolve/resolv.conf file may also work though not tested yet and we don't have a daemonset to do this for you.

We are working on fixing this.

paulgmiller · 2021-01-29T17:20:26Z

Relevant disucssion of this in systemd issue: systemd/systemd#13763
Someone else trying to use a daemonset to bypass systemd-resolve as mentioned above https://gist.github.com/levimm/c27a8940479e4f17e53da0d9d477defe#file-dnsresolv-yaml
Not vetted nor tested by AKS so use entirely at own risk.

paulgmiller · 2021-01-29T18:01:05Z

Also curious if setting ResolveUnicastSingleLabel=yes would unblock .local without completely disabling systemd. @xuto2

joaguas · 2021-02-04T18:52:51Z

Seems that pods are not affected by this issue, even when using hostNetwork:

After digging for a while it seems it's --resolv-conf=/run/systemd/resolve/resolv.conf on kubelet holding it together meaning coredns or other hostnetworked will actually get the upstream resolvers effectively bypassing the nss-resolve/systemd-resolved resolver.

Makes sense since otherwise the loop plugin on the coredns cm would halt coredns due to the loopback circular reference.

As discussed with @paulgmiller this will still impact nodes (ie: pulling images from a .local registry).

@xuto2

xuto2 · 2021-02-05T21:12:28Z

@joaguas thanks a lot for sharing. It's true this doesn't affect pod traffic as we understand as well. The daemonset approach could be a temp mitigation while we're evaluating a permanent solution from aks node side.

bbgobie · 2021-02-08T14:27:24Z

Also curious if setting ResolveUnicastSingleLabel=yes would unblock .local without completely disabling systemd. @xuto2

I'm not sure if it would, but a fix needs to encompass more than just .local
As the systemd-resolv change seems to make our nodes not use the dns servers from the network we can also not resolve internal resources like private end points

bbgobie · 2021-03-16T20:32:10Z

1.17 End of Support is approaching pretty quick, is there official word from AKS on what they want us to do with this? We all run an unsupported daemonset to fix it?

xuto2 · 2021-03-16T22:15:31Z

we're disabling resolved on all new 1804 VMs, the change is in release and expected to be done by next week. AKS release notes https://github.com/Azure/AKS/releases/tag/2021-03-08 also mentioned it.

ghost · 2021-04-16T01:01:13Z

Action required from @Azure/aks-pm

ghost · 2021-05-01T06:02:12Z