coredns forward . /etc/resolv.conf not working

[dee-kryvenko]

Environmental Info:
K3s Version:

k3s version v1.21.4+k3s1 (3e250fdb)
go version go1.16.6

Node(s) CPU architecture, OS, and Version:

Linux lima-default 5.11.0-36-generic #40-Ubuntu SMP Fri Sep 17 18:15:22 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

curl -sfL https://get.k3s.io | sh - from a Lima default Ubuntu VM.

Describe the bug:

I have a private DNS I need to use, and on my Ubuntu host I've configured it with resolvectl dns eth0 x.x.x.x y.y.y.y. My pods in k3s however doesn't seem to get to these servers and they are using something else. I verified my Corefile is:

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    hosts /etc/coredns/NodeHosts {
      ttl 60
      reload 15s
      fallthrough
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

So if I read it correctly - forward . /etc/resolv.conf line means coredns should use resolver from the host. My /etc/resolv.conf on the host is pointing to the systemd-resolved and if I take a nameserver from there and try to dig/nslookup using that server both on the host and inside a pod - it resolves correctly. My suspicion that k3s actually sits in between coredns and the host, or maybe I need to use other than eth0 interface to set DNS? cni0 interface doesn't work either.

Steps To Reproduce:

• Installed K3s: curl -sfL https://get.k3s.io | sh -
• Configure systemd-resolved to use a private DNS
• Run a pod and try to resolve a private address

Expected behavior:

Pod should be able to resolve private address.

Actual behavior:

Not resolving.

Additional context / logs:

Backporting

[brandond]

K3s validates the host's resolv.conf to ensure it doesn't contain any multicast or ipv6 upstream DNS servers. If it does, it will generate a replacement resolv.conf that points at 8.8.8.8. If you want to provide your own resolv.conf, you can point K3s at one with the --resolv-conf flag.

[dee-kryvenko]

Ah I see - I thought I am going crazy... Indeed I double checked that the nameserver IP is not reachable from pods.

So there is no way to make coredns to use systemd-resolved from the host?

Where do I put --resolv-conf if I installed k3s with a curl -sfL https://get.k3s.io | sh -?

[brandond]

You can re-run the install script with that flag added as shown here: https://rancher.com/docs/k3s/latest/en/installation/install-options/how-to-flags/#example-b-install-k3s-exec

Or simply edit the systemd unit and then reload/restart k3s.

[dee-kryvenko]

Thank you!

I've found related issue coredns/coredns#2087 and seems like the answer is - no, coredns does not support systemd-resolved. But - there is a pre-existing /run/systemd/resolve/resolv.conf that actually seems to reflect systemd-resolved config, I used your example with url -sfL https://get.k3s.io | sh -s - --resolv-conf /run/systemd/resolve/resolv.conf and that seems to work.