systemd-resolved returns synthesized DNS records for the local host

[tomhughes]

systemd version the issue has been seen with

256

Used distribution

Fedora 41

Linux kernel version used

6.11.5-300.fc41.x86_64

CPU architectures issue was seen on

x86_64

Component

systemd-resolved

Expected behaviour you didn't see

When resolving the local host systemd-resolved should return the address in the DNS and nothing else.

Unexpected behaviour you saw

When resolving the local host systemd-resolved returned DNS records and also every other IP address assigned to a local interface.

Steps to reproduce the problem

After upgrading to Fedora 41 (and hence systemd 256) this is what I see:

bericote [~] % resolvectl query bericote.compton.nu           
bericote.compton.nu: 172.16.15.2               -- link: br0
                     10.88.0.1                 -- link: podman0
                     2001:8b0:bd:1:1881:14ff:fe46:3cc7 -- link: br0
                     fd96:7c2e:b8d2:bf65::1    -- link: podman0
                     fe80::1881:14ff:fe46:3cc7%2 -- link: br0
                     fe80::bec7:46ff:fe9a:cf7c%4 -- link: wlp6s0
                     fe80::24b9:eff:fef3:5f08%5 -- link: podman0
                     fe80::8453:1eff:feb0:970f%6 -- link: veth0

-- Information acquired via protocol DNS in 516us.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
-- Data from: synthetic

Despite the claim that the data comes from DNS only 172.16.15.2 and 2001:8b0:bd:1:1881:14ff:fe46:3cc7 are actually present in the DNS.

It appears the default value of SYSTEMD_RESOLVED_SYNTHESIZE_HOSTNAME has changed from false to true in 6399be2 and explicitly setting it to false in systemd-resolved.service fixes this.

If you're wondering why this is a problem it's because my firefox happily picked one of the link local addresses to use and then apache 403ed because that address was not in it's ACL list.

Additional program output to the terminal or log subsystem illustrating the issue

No response

[pemensik]

I think the main problem is manual page for systemd-resovled nor resolved.conf does not contain any information, how to disable such synthesis. From manual page(s) it seems impossible. It is mentioned in ENVIRONMENT.md, but that is part of systemd package. Not where user would find it.

[tomhughes]

As far as I could see there's nothing in NEWS either.

[poettering]

This behaviour is clearly documented in the man page and the addresses returned are sorted by scope, i.e. link-local comes last.

Moroever the resolvectl output explicitly tells you that the data is synthetic.

The env var is documented, but we consider it a bit hackish and hence its in ENVIRONMENT.md and not on the main man page

I am not seeing anything actionable here?

[tomhughes]

I've just been doing a bit more digging and I think that part of the problem is that glibc is sometimes resorting the addresses. Here's a different machine with resolvectl:

gosford.compton.nu: 172.16.15.1                -- link: br0
                    172.16.15.1                -- link: wg0
                    217.169.17.27              -- link: ppp0
                    2001:8b0:bd:1:6cd2:7aff:fea1:9164 -- link: br0
                    2001:8b0:bd:1:21c:c0ff:fede:937a -- link: wg0
                    2001:8b0:bd:1:e2d5:5eff:fee7:6698 -- link: ppp0
                    fe80::e2d5:5eff:fee7:6698%2 -- link: enp1s0
                    fe80::20e:8eff:fe1f:4239%4 -- link: wlp5s0
                    fe80::807a:5eff:fe76:6e55%5 -- link: br0
                    fe80::edba:c5d6:4149:e12c%7 -- link: ppp0

-- Information acquired via protocol DNS in 6.2ms.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
-- Data from: synthetic

and then using getent ahosts to do a getaddrinfo query:

gosford [~] % getent ahosts gosford.compton.nu
fe80::e2d5:5eff:fee7:6698%2 STREAM gosford.compton.nu
fe80::e2d5:5eff:fee7:6698%2 DGRAM  
fe80::e2d5:5eff:fee7:6698%2 RAW    
fe80::20e:8eff:fe1f:4239%4 STREAM 
fe80::20e:8eff:fe1f:4239%4 DGRAM  
fe80::20e:8eff:fe1f:4239%4 RAW    
fe80::807a:5eff:fe76:6e55%5 STREAM 
fe80::807a:5eff:fe76:6e55%5 DGRAM  
fe80::807a:5eff:fe76:6e55%5 RAW    
2001:8b0:bd:1:6cd2:7aff:fea1:9164 STREAM 
2001:8b0:bd:1:6cd2:7aff:fea1:9164 DGRAM  
2001:8b0:bd:1:6cd2:7aff:fea1:9164 RAW    
2001:8b0:bd:1:21c:c0ff:fede:937a STREAM 
2001:8b0:bd:1:21c:c0ff:fede:937a DGRAM  
2001:8b0:bd:1:21c:c0ff:fede:937a RAW    
2001:8b0:bd:1:e2d5:5eff:fee7:6698 STREAM 
2001:8b0:bd:1:e2d5:5eff:fee7:6698 DGRAM  
2001:8b0:bd:1:e2d5:5eff:fee7:6698 RAW    
fe80::edba:c5d6:4149:e12c%7 STREAM 
fe80::edba:c5d6:4149:e12c%7 DGRAM  
fe80::edba:c5d6:4149:e12c%7 RAW    
172.16.15.1     STREAM 
172.16.15.1     DGRAM  
172.16.15.1     RAW    
172.16.15.1     STREAM 
172.16.15.1     DGRAM  
172.16.15.1     RAW    
217.169.17.27   STREAM 
217.169.17.27   DGRAM  
217.169.17.27   RAW    

which seems to have moved host of the link local addresses to the front of the list.

[pemensik]

The order of DNS records returned is not supposed to be fixed or mean anything specific. A good cache should rotate returned records for each query. Yes, man systemd-resolved(8) has SYNTHETIC RECORDS section, which describes this behaviour. It is not documented why these synthetized records are preferred over real responses from the network. What does it fix, when network has positive answers for such queries?

Link local ipv6 cannot be received via DNS with interface index included. AAAA record encodes just sin6_addr part of struct sockaddr_in6. It may confuse some programs.

I confirm my rawhide instance returns fe80::* first, followed by ipv4 address. When resolve and myhostname plugins are used, ipv6 link-local is listed first. With dns plugin ipv4 is listed first instead.

import socket
for i in socket.getaddrinfo(socket.gethostname(), 0): print(i)

Matches the order which getent ahosts $HOSTNAME returns with glibc-2.40.9000-7.fc42.x86_64. IPv4 private address is listed after link-local ipv6.

# rpm -q systemd-resolved
systemd-resolved-256.7-1.fc42.x86_64

# getent ahosts -s resolve $HOSTNAME
fe80::7c85:92ff:fe43:8871%2 STREAM rawhide
fe80::7c85:92ff:fe43:8871%2 DGRAM  
fe80::7c85:92ff:fe43:8871%2 RAW    
192.168.122.184 STREAM 
192.168.122.184 DGRAM  
192.168.122.184 RAW    

# getent ahosts -s resolve $HOSTNAME
fe80::7c85:92ff:fe43:8871%2 STREAM rawhide
fe80::7c85:92ff:fe43:8871%2 DGRAM  
fe80::7c85:92ff:fe43:8871%2 RAW    
192.168.122.184 STREAM 
192.168.122.184 DGRAM  
192.168.122.184 RAW    

# getent ahosts -s dns $HOSTNAME
192.168.122.184 STREAM rawhide
192.168.122.184 DGRAM  
192.168.122.184 RAW    
fe80::7c85:92ff:fe43:8871 STREAM 
fe80::7c85:92ff:fe43:8871 DGRAM  
fe80::7c85:92ff:fe43:8871 RAW    

# dig +short $HOSTNAME @192.168.122.1 
192.168.122.184

It seems the combination used with glibc does not work as advertised in the man page. Which does not explain why it does not just cache network response and generates own response without asking. Or why there this behaviour should not be configurable a normal way.

[pemensik]

Man page says:

The local, configured hostname is resolved to all locally configured IP addresses ordered by their scope, or — if none are configured — the IPv4 address 127.0.0.2 (which is on the local loopback interface) and the IPv6 address ::1 (which is the local host).

The above reports state that is not true. Still this is marked with not-a-bug tag.

The second part is why this is even done and why simple documented way to disable the behaviour is not in a man page. That part might be to discussion, whether or not this is desired feature, not a bug.

But the minimal part reporting wrong order could not be considered a feature. Lennart himself posted link-locals should be last, yet they are not. I have to say I don't understand.

[salahcoronya]

This also prevents from discovering their FQDN, which can cause problem. In my case, OpenLDAP / slapd stopped working because of this synthesis. OpenLDAP was configured to listen on all interface, but it stopped being able to start because it could no longer discover the canonical hostname and match it to olcServerID (used for mirror and multi-provider setups) anymore. You'll see this problem if the the machine hostname is bare (just "machine"), the client is listed in DNS (machine,example.com), there's a search domain for example.com, and you try to run "hostname --fqdn". With synthesis it just displays the hostname, without synthesis the full FQDN is returned as expected.

[gucci-on-fleek]

This also makes any queries about the system's hostname for non-A/AAAA rrtypes (MX, TXT, etc.) return an empty answer, even if the DNS contains records for those. Among other issues, this means that it's impossible to send email to the hostname of the system:

$ resolvectl --version
systemd 256 (256.9-2.fc41)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT +LIBARCHIVE

$ resolvectl status
Global
         Protocols: LLMNR=resolve -mDNS +DNSOverTLS DNSSEC=yes/supported
  resolv.conf mode: stub
Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001

Link 2 (ens3)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6
         Protocols: -DefaultRoute LLMNR=resolve -mDNS +DNSOverTLS DNSSEC=yes/supported

$ cat /etc/hostname
maxchernoff.ca

$ resolvectl query --type=mx maxchernoff.ca
maxchernoff.ca: resolve call failed: Name 'maxchernoff.ca' does not have any RR of the requested type

$ busybox nslookup -type=mx maxchernoff.ca
Server:		127.0.0.53
Address:	127.0.0.53:53

$ printf '[Service]\nEnvironment=SYSTEMD_RESOLVED_SYNTHESIZE_HOSTNAME=0\n' | sudo tee /etc/systemd/system/systemd-resolved.service.d/override.conf
[...]

$ sudo systemctl daemon-reload && sudo systemctl restart systemd-resolved.service

$ resolvectl query --type=mx maxchernoff.ca
maxchernoff.ca IN MX 10 aspmx1.migadu.com                   -- link: ens3
maxchernoff.ca IN MX 20 aspmx2.migadu.com                   -- link: ens3

-- Information acquired via protocol DNS in 22.8ms.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
-- Data from: network

$ busybox nslookup -type=mx maxchernoff.ca
Server:		127.0.0.53
Address:	127.0.0.53:53

Non-authoritative answer:
maxchernoff.ca	mail exchanger = 10 aspmx1.migadu.com
maxchernoff.ca	mail exchanger = 20 aspmx2.migadu.com

Even if this is working as documented, I'd argue that it's both unexpected and undesirable for MX and TXT queries about the local system to fail.

[pemensik]

This feature for example prevents committing on unconfigured machine with working reverse address. If git does not have user.email address set, it tries to get $USER@$FQDN as email address. For getting email address from relative hostname only it spawns $HOSTNAME query, attempting to use search domains to find working hostname full name. With myhostname and resolve nss plugins before dns it fails hard, preventing commit until user configures something.

When it is resolvable from network and not blocked by systemd, it will let you commit even without setting explicitly email. It needs just ability to resolve hostname -f, which systemd blocks in current configuration. For not good reason, I assume. When starting mutt, it does similar thing. I am not sure why even dnf list system does it too.

[pemensik]

It seems openldap libraries are source of unnecessary resolution of hostname. I have filled rhbz#2331728 for it.