DNS server marked as DNSSEC-incompatible after query for domain in NTA

[toreanderson]

systemd version the issue has been seen with

systemd-239-6.git9f3aed1.fc29.x86_64

Used distribution

Fedora 29

Expected behaviour you didn't see

That DNSSEC-related failures happening for FQDNS in the DNSSEC NTA doesn't cause systemd-resolved to conclude that the server has no support for DNSSEC.

Unexpected behaviour you saw

When looking up a FQDN with a domain component listed in the DNSSEC NTA, systemd-resolved might in some cases determine that the DNS server does not support DNSSEC. When configured with DNSSEC=yes, that means that DNS stops working completely, essentially.

Steps to reproduce the problem

1. Connect to a network using an OpenWrt 18.06.1 router (or probably any other router using dnsmasq with an internal .lan zone, which is found in system-resolved's default NTA). I assume that the router has the host name gw, so that it responds for the FQDN gw.lan.

2. Observe how a lookup for an FQDN/RR that the OpenWrt/dnsmasq has an answer for gets treated just fine:

$ resolvectl reset-server-features; resolvectl flush-caches; systemd-resolve gw.lan -t A
gw.lan IN A 192.168.1.1

-- Information acquired via protocol DNS in 4.5ms.
-- Data is authenticated: no

3. Observe how a lookup for an FQDN/RR that the OpenWrt/dnsmasq do not have an answer for causes a DNSSEC-related failure:

$ resolvectl reset-server-features; resolvectl flush-caches; systemd-resolve gw.lan -t AAAA
gw.lan: resolve call failed: DNSSEC validation failed: incompatible-server

As mentioned earlier, from this point on systemd-resolved consideres the server DNSSEC-incompatible for all queries, not just for the *.lan ones.

The same thing happens for A queries for non-existent hostnames (e.g., systemd-resolve nonexistent.lan -t A), or for other non-existent types for hostnames that do exist (e.g., systemd-resolve gw.lan -t TXT).

I'm attaching two outputs from tshark that shows the traffic between systemd-resolved and the upstream OpenWrt/dnsmasq server that happens as a result of the two commands above:

• The -t A query that doesn't cause any subsequent issues
• The -t AAAA query that causes DNSSEC to be disabled

I'd like to point out two differences between them:

• The former response has the authoritative flag set, the latter does not.
• The former response has the EDNS0 DO bit set, the latter does not.

This inconsistency might be considered a bug in dnsmasq, so I would understand it if this issue gets closed as «someone else's problem», assuming it is the reason why systemd-resolved behaves the way it does.

On the other hand, dnsmasq is an immensely popular DNS proxy/cache embedded in many home gateway products. Even if I was able to get this fixed in dnsmasq upstream, it will take a very long time before the current behaviour is no longer being encountered in the wild. Therefore, I'm thinking that this is something systemd-resolved could (or maybe even should, taking the robustness principle into account) potentially handle better to the benefit of its users.

[ShapeShifter499]

@toreanderson did you report this to folks working on dnsmasq? I think I have this same issue.

systemd-resolved[125641]: Server 192.168.1.1 does not support DNSSEC, downgrading to non-DNSSEC mode.

[root@kumo ~]# systemctl --version
systemd 243 (243.51-1-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

[toreanderson]

@ShapeShifter499 No - as I do not quite understand what dnsmasq has to with this issue.

Uh, scratch that. Thought I was responding to another issue. In any case, no, I have not.

[poettering]

Is this still relevant? If so @toreanderson can you provide debug log output when this happens on a current systemd-resolved version. I.e. run:

systemctl edit systemd-resolved

This opens an editor, then add there:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

Then save/exit, and issue:

systemctl restart systemd-resolved

This enables debug logging in resolved, then retrigger the issue, and link the relevant log output here when this issue happened.

[toreanderson]

@poettering I tried to reproduce the issue now, but was not able to.

I do not know if this is because of changes in systemd (which has been upgraded to systemd-246.6-3.fc33.x86_64 since submitting this issue) and/or due to changes in OpenWrt (which has been upgraded to version 18.06.2).

I am therefore closing this issue for now. @ShapeShifter499, if you are still seeing it, let me know and I can re-open it.