Full VM images have leftover resolv.conf, don't use systemd-resolved

[scottyeager]

Checking the Ubuntu full VM images (24.04 and 22.04) I notice that there's a resolv.conf written during image build:

https://github.com/threefoldtech/tf-images/blob/1a045e86c904b737cedf656cc8200fd76abec852/tfgrid3/ubuntu24.04/fullvm/ubuntu24-fullvm-flist-builder.sh#L30

This I would expect to be temporary, but in fact it is not cleaned up. So apps using resolv.conf inside deployed VMs will go directly to 1.1.1.1 and that server only.

On the other hand, systemd-resolved is already running when these images are deployed, and using it would be the normal thing to do in an Ubuntu machine. There are various advantages to using resolved and no obvious downside.

To change it, these lines would be added to the image build script, after all steps requiring DNS have been completed:

rm /etc/resolv.conf
ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

[PeterNashaat]

• Added the support of systemd-resolved here https://github.com/threefoldtech/tf-images/blob/ubuntu24/tfgrid3/ubuntu24.04/fullvm/ubuntu24-fullvm-flist-builder.sh
• Built flist https://hub.grid.tf/petep.3bot/ubuntu-24.04_fullvm.flist
• Testing
  • ping error for a few seconds after restarting the systemd-resolved service or in a new vm deployment

root@testresolve24:~# ping google.com
ping: google.com: Temporary failure in name resolution
root@testresolve24:~#
root@testresolve24:~# ping google.com
PING google.com (2a00:1450:400c:c09::71) 56 data bytes
64 bytes from wm-in-f113.1e100.net (2a00:1450:400c:c09::71): icmp_seq=1 ttl=60 time=18.2 ms
64 bytes from wm-in-f113.1e100.net (2a00:1450:400c:c09::71): icmp_seq=2 ttl=60 time=16.0 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 15.985/17.086/18.187/1.101 ms
root@testresolve24:~#

• Ping worked only after these logs in systemd-resolved service

Feb 16 1:17:28 testresolve24 systemd-resolved[12571]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 8.8.8.8.
Feb 16 1:17:42 testresolve24 systemd-resolved[12571]: Using degraded feature set TCP instead of UDP for DNS server 8.8.8.8.

• systemd-resolved may not have been fully initialized or properly connected to the upstream DNS.
• from the logs in service systemd-resolved it had trouble communicating with 8.8.8.8 using UDP + EDNS0 and switched to a degraded mode.
• Once systemd-resolved caches the response, new requests work fine.
• Restarted systemd-resolved again and tried to ping google.com and checked dig +edns=0 and dig +noedns both worked, while ping didn't.

root@testresolve24:~# ping google.com
ping: google.com: Temporary failure in name resolution
root@testresolve24:~#
root@testresolve24:~# dig +edns=0 @8.8.8.8 google.com

; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> +edns @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22529
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       64.233.166.100
google.com.             300     IN      A       64.233.166.101
google.com.             300     IN      A       64.233.166.102
google.com.             300     IN      A       64.233.166.139
google.com.             300     IN      A       64.233.166.138
google.com.             300     IN      A       64.233.166.113

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Sun Feb 16 10:41:58 UTC 2025
;; MSG SIZE  rcvd: 135

root@testresolve24:~# dig +noedns @8.8.8.8 google.com

; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> +noedns @8.8.8.8 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44624
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       64.233.166.101
google.com.             300     IN      A       64.233.166.139
google.com.             300     IN      A       64.233.166.138
google.com.             300     IN      A       64.233.166.113
google.com.             300     IN      A       64.233.166.100
google.com.             300     IN      A       64.233.166.102

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Sun Feb 16 10:42:46 UTC 2025
;; MSG SIZE  rcvd: 124

root@testresolve24:~#
root@testresolve24:~# resolvectl statistics
Transactions
                       Current Transactions:  0
                         Total Transactions: 16

Cache
                         Current Cache Size:  5
                                 Cache Hits:  6
                               Cache Misses: 20

Failure Transactions
                             Total Timeouts:  0
         Total Timeouts (Stale Data Served):  0
                    Total Failure Responses:  0
Total Failure Responses (Stale Data Served):  0

DNSSEC Verdicts
                                     Secure:  0
                                   Insecure:  0
                                      Bogus:  0
                              Indeterminate:  0
root@testresolve24:~#

[PeterNashaat]

• Fixed by adding

cat <<EOT > /etc/systemd/resolved.conf
[Resolve]
DNS=8.8.8.8 1.1.1.1 9.9.9.9
FallbackDNS=8.8.4.4 149.112.112.112 1.0.0.1
DNSSEC=no
Cache=yes

This ensures:
Stable and fast DNS resolution by specifying reliable, globally accessible DNS servers.
Prevention of degraded feature warnings from systemd-resolved.

These are public DNS resolvers :

• Google Public DNS: 8.8.8.8 (Primary), 8.8.4.4 (Secondary)
• Cloudflare DNS: 1.1.1.1, 1.0.0.1
• Quad9 DNS: 9.9.9.9, 149.112.112.112

By default, systemd-resolved tries to validate DNSSEC (Domain Name System Security Extensions).
However, this can cause:

• Delays if upstream DNS servers do not fully support it.
• Failures if DNSSEC validation is improperly configured.
• Extra overhead for a system where DNSSEC is unnecessary.

Since trusted resolvers (Google, Cloudflare, Quad9) already handle DNSSEC validation, disabling it (DNSSEC=no) avoids unnecessary processing.

How It Fixed the Issue

• Prevented "Using degraded feature set UDP instead of UDP+EDNS0" erorr logs.
• Ensured that systemd-resolved immediately uses the correct DNS servers.
• Eliminated temporary failures in name resolution when rebooting or restarting systemd-resolved service.

Now, DNS queries should always resolve immediately without degraded performance.

If future requirements reqiure DNSSEC validation, we can revisit enabling it while testing for any resolution delays. For now, this fix ensures optimal performance and reliability. 🚀

[PeterNashaat]

• Flist created https://hub.grid.tf/petep.3bot/ubuntu-24.04_fullvm.flist
• Tested and verified
• Will try to investigate more to enable dnssec and solve the issue at the same time

[scottyeager]

Hi @PeterNashaat,

I was not completely correct in my original description of the solution. That was based on applying this to already running VMs.

What might not have been clear is that I wasn't suggesting to remove the use of 1.1.1.1 entirely. That's useful for building the image. The switch to resolved should happen at the end of the build process and be used when the image boots as a VM later.

What I found in testing this approach with building an image is that I had to do it outside the change root. Here's the change that's needed:

# Clean up
rm ubuntu-noble/root/setup_inside_chroot.sh
rm -rf ubuntu-noble/dev/*
rm ubuntu-noble/etc/resolv.conf

# Create symlink to enable systemd-resolved
ln -s /run/systemd/resolve/stub-resolv.conf ubuntu-noble/etc/resolv.conf

With this change the image boots successfully and DNS is working inside the deployed VM.

[mik-tf]

I confirm this is still an ongoing issue. I had to run Scott's fix (first message of the issue) to make it work on full VM.