systemd-resolved: failure to resolve duckduckgo.com when using 1.1.1.3

[living180]

systemd version the issue has been seen with

254.8

Used distribution

Gentoo stable

Linux kernel version used

6.6.13-gentoo-dist

CPU architectures issue was seen on

x86_64

Component

systemd-resolved

Expected behaviour you didn't see

resolvectl query duckduckgo.com successfully resolves to 40.114.177.246 (safe.duckduckgo.com)

Unexpected behaviour you saw

resolvectl query duckduckgo.com gives the error duckduckgo.com: resolve call failed: DNSSEC validation failed: no-signature

Steps to reproduce the problem

My system uses systemd-resolved as its DNS resolver. It points to an OpenWRT router running dnsmasq as its DNS server. dnsmasq on the router is then configured to use 1.1.1.3 (Cloudflare's "1.1.1.1 for Families") as its upstream DNS server. 1.1.1.3 returns a CNAME record for duckduckgo.com pointing to safe.duckduckgo.com. After upgrading to systemd 254.8, my system stopped being able to resolve duckduckgo.com. Using git bisect, I found that commit 0292727 (backported from 3b4cc14) was the first commit that exhibited the problem.

Manually running systemd-resolved produces the following in the output when running resolvectl query duckduckgo.com:

    DNSSEC validation failed for question duckduckgo.com IN A: no-signature
    DNSSEC validation failed for question duckduckgo.com IN AAAA: no-signature
    DNSSEC validation failed for question duckduckgo.com IN AAAA: no-signature
    DNSSEC validation failed for question duckduckgo.com IN A: no-signature

Also, despite 1.1.1.3 returning a similar CNAME record for www.duckduckgo.com, that name resolves correctly while duckduckgo.com does not.

Additional program output to the terminal or log subsystem illustrating the issue

% dig duckduckgo.com

; <<>> DiG 9.16.48 <<>> duckduckgo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9876
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;duckduckgo.com.			IN	A

;; ANSWER SECTION:
duckduckgo.com.		60	IN	CNAME	safe.duckduckgo.com.
safe.duckduckgo.com.	21	IN	A	40.114.177.246

;; Query time: 33 msec
;; SERVER: 10.1.1.1#53(10.1.1.1)
;; WHEN: Sun Feb 25 16:09:14 +03 2024
;; MSG SIZE  rcvd: 78

% resolvectl query duckduckgo.com 
duckduckgo.com: resolve call failed: DNSSEC validation failed: no-signature

% dig www.duckduckgo.com

; <<>> DiG 9.16.48 <<>> www.duckduckgo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15868
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.duckduckgo.com.		IN	A

;; ANSWER SECTION:
www.duckduckgo.com.	60	IN	CNAME	safe.duckduckgo.com.
safe.duckduckgo.com.	276	IN	A	40.114.177.246

;; Query time: 66 msec
;; SERVER: 10.1.1.1#53(10.1.1.1)
;; WHEN: Sun Feb 25 16:10:14 +03 2024
;; MSG SIZE  rcvd: 82

% resolvectl query www.duckduckgo.com
www.duckduckgo.com: 40.114.177.246             -- link: enp3s0
                    (safe.duckduckgo.com)

-- Information acquired via protocol DNS in 605.3ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

[living180]

I should add that I also tested the latest commits in the main branch and the 254-stable branch and they also exhibited the same behavior.

[living180]

Additionally, changing DNSSEC in resolved.conf from the default of allow-downgrade to no allows duckduckgo.com to resolve successfully.

[rpigott]

Eh, CNAME at the zone apex? Isn't that illegal? Seems like a Cloudflare bug.

[rouven0]

Just porting my thoughs from #31699 into here. When we encounter an apex-wide CNAME while fetching the DNSKEY or DS record, we should discard them as invalid and treat the domain as a domain that has DNSSEC disabled instead of trying to verify the nonexistent signatures.

[rpigott]

I think you may have misunderstood what is happening here. When we are validating, every name must either be signed with a full chain of trust, or have a valid insecure delegation somewhere in its ancestry.

CNAME at the zone apex is neither, as it isn't even a well-formed zone. Even non-validating recursive resolvers would be justified in rejecting such answers. However when we are not validating we aren't likely to notice this kind of error, and will return whatever the upstream recursive resolver is able to give us. dnssec=allow-downgrade is supposed give up validating if the upstream resolver doesn't appear capable of handling dnssec data, but it will still reject invalid signatures and misconfigured domains like these. In both cases this is expected behavior.

You should report these errors to the domain operator, they are unmistakably incorrect. This won't be fixed in sd-resolved in the same way that sd-resolved won't hallucinate the correct answer for you were the domain operator to accidentally configure the wrong address.

[rpigott]

Hm, so it seems like cloudflare is bending the rules here to make validation possible:

$ dig @1.1.1.3 +do +noall +ans +noauth duckduckgo.com SOA duckduckgo.com A    
duckduckgo.com.		14033	IN	SOA	dns1.p05.nsone.net. hostmaster.nsone.net. 1655203370 7200 7200 1209600 14400
duckduckgo.com.		60	IN	CNAME	safe.duckduckgo.com.
safe.duckduckgo.com.	100	IN	A	52.250.41.2
$ delv @1.1.1.3 duckduckgo.com A 
; unsigned answer
duckduckgo.com.		60	IN	CNAME	safe.duckduckgo.com.
safe.duckduckgo.com.	296	IN	A	52.250.41.2

There are worse hacks in sd-resolved in the name of compatibility I guess, so maybe we could support this idk.