Pod DNS resolution broken

[kmatasfp]

Initially Flux setup failed because I was getting

2023-02-15T06:05:02.397Z error GitRepository/home-kubernetes.flux-system - Reconciler error failed to checkout and determine revision: unable to clone 'https://github.com/kmatasfp/k8s-external': Get "https://github.com/kmatasfp/k8s-external/info/refs?service=git-upload-pack": dial tcp: lookup github.com on 10.43.0.10:53: read udp 10.42.140.4:48046->10.43.0.10:53: read: connection refused

confirmed that it was dns issue by running dig and nslookup from within a Pod I deployed for test purposes.

Even joined Discord to see if any information there and found this thread: https://discord.com/channels/673534664354430999/1074532630395105341

From there I got a link to recently merged PR #627, If I removed the changes made there DNS resolution started to work again from the Pods.

Host Details:
Ubuntu 22.04

resolvectl status

Current DNS Server: 1.1.1.1
DNS Servers: 1.1.1.1
DNS Domain: local.${SECRET_DOMAIN}

So the change might work for Fedora hosts, but not Ubuntu. Imho need to revisit this or provide a doc or have step in ansible playbook that during install is Ubuntu specific and makes sure that DNS works

[onedr0p]

Thanks for reporting this issue, I do have a feeling this needs to be documented more or maybe other people can share their experiences and what's causing the issue and how they fixed it.

There's not only a difference in ubuntu or fedora but also in the way people handle DNS at home. It's all over the place so it's hard to have one fix for all situations. Search domains in /etc/resolv.conf are notorious for being a big pain in the ass for most people since DHCP servers hand out their domain and systemd-resolved forces into resolv.conf

I'll go ahead and convert this into a GH discussion.

I suggest people post here their problems and what they did to mitigate DNS issues.

[Tyler-Cash]

Just thought I'd document the problem as I saw it. I don't think my solution is really mergeable in the current state as it feels too jank.

Cause

As you mentioned, systemd-resolved is ultimately the root cause of this. It'll override /etc/resolv.conf with nameserver 127.0.0.1:53. From a container that'll obviously not resolve the way the systemd-resolved service is expecting so you get DNS issues depending on the containers DNS implementation. Some containers seem to implement their own DNS resolution logic, so some will work others won't. Specifically the onedr0p containers rely on your hosts DNS to work correctly so will fail.

Janky Solution

cluster-installation.yaml

    - name: Fix DNS for ubuntu
      block:
        - name: Stop and disable service which messes with DNS
          service:
            name: "systemd-resolved"
            state: stopped
            enabled: false
        - name: Delete link to the systemd-resolved resolv.conf
          ansible.builtin.file:
            path: /etc/resolv.conf
            state: absent
        - name: Create empty resolv.conf
          ansible.builtin.file:
            path: /etc/resolv.conf
            state: touch
        - name: Setup nameserver
          blockinfile: |
            dest=/etc/resolv.conf
            content="nameserver 10.0.90.121"
      when: ansible_facts['distribution'] == 'Ubuntu'
      

Make this change, run your Ansible playbook and kill your kube-system/coredns pod, DNS should start working for you. (If it doesn't see k3s.yaml below)

Basically, we're just killing the service which messes with our resolv.conf and updating the file to have a known good DNS to handle queries for us, this could be changed to an internal DNS if you wanted to as well.

For some odd reason, it didn't apply for me, so I needed to add this into the group vars as well. This should be the implied value, I don't really understand why I need to do this...
k3s.yaml

  kubelet-arg:
    # Don't pull /etc/resolv.conf from host
    - "resolv-conf=/etc/resolv.conf"

For a non jank solution, I think we would try to see if DNS resolution works in a known broken pod, then do this fix if needed and restart the CoreDNS deployment so that no human intervention is required and it's idempotent. I think I'll leave it here for now.

[sarcasticgoat]

DNS search domains in /etc/resolv.conf on my nodes were causing issues with external DNS resolution (thanks pfSense). I was able to fix it using @Tyler-Cash's addition to cluster-installation.yaml.

- name: Fix DNS for nodes
      block:
        - name: Stop and disable service which messes with DNS
          service:
            name: "systemd-resolved"
            state: stopped
            enabled: false
        - name: Setup nameserver
          blockinfile:
            dest: /etc/resolv.conf
            block: |
              nameserver $dnsserver

There's not really a non-jank solution for those of us experiencing this issue since it's an upstream issue from DHCP like @onedr0p mentioned.

[DevSecNinja]

Just implemented your / @Tyler-Cash's code block on Ubuntu 22.04.2. Looks good so far, thanks!

For reference: https://github.com/DevSecNinja/k3s-home/blob/main/ansible/playbooks/cluster-installation.yml#L198C1-L220C54

[kmatasfp]

that seemed to have solved the issue, minor thing tough, if one disables systemd-resolved then symlink to resolv.conf needs to be deleted first before modifying it with ansible as described here https://gist.github.com/zoilomora/f7d264cefbb589f3f1b1fc2cea2c844c. Sadly do not know enough ansible to fix the above snippet.

[Tyler-Cash]

Which OS and version were you running specifically? I tried this on 22.04, I'm guessing you're on an older OS that may behave slightly differently

I've amended the tasks to include a cleanup of the file, before writing the nameserver to cover this case

[kmatasfp]

Kubernetes agent/node's run on Ubuntu 22.04.1, created from current cloud-init image from here https://cloud-images.ubuntu.com/jammy/current/

[onedr0p]

It appears future versions of Alpine Linux will hopefully fixed this once and for all.

https://www.openwall.com/lists/musl/2023/05/02/1

[onedr0p]

Flux updated to Alpine 3.18 in the latest RC, so hopefully no more DNS issues there at least!

[onedr0p]

On Debian 12 this is much easier to fix. According to the Debian docs on resolv.conf you can just do the following.

During the editor command your /etc/resolv.conf should look like this:

search .
nameserver <preferred-dns-server>