resolved: `gpg --send-keys` fails with stub-resolv.conf #23737

Tachi107 · 2022-06-14T09:59:41Z

systemd version the issue has been seen with

systemd 251 (251.2-5)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

Used distribution

Debian Testing (12)

Linux kernel version used (uname -srvmo)

Linux 5.17.0-1-amd64 #1 SMP PREEMPT Debian 5.17.3-1 (2022-04-18) x86_64 GNU/Linux

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see

gpg --send-keys should be able to send keys when using systemd-resolved and /etc/resolv.conf points to the resolved stub

Unexpected behaviour you saw

$ ls -lh /etc/resolv.conf
lrwxrwxrwx 1 root root 37 14 giu 11.54 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
$ gpg --keyserver pgp.surf.nl --send-key 66DEF15282990C2199EFA801A8A128A8AB1CEE49
gpg: sending key A8A128A8AB1CEE49 to hkp://pgp.surf.nl
gpg: keyserver send failed: Server indicated a failure
gpg: keyserver send failed: Server indicated a failure

$ ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
$ gpg --verbose --verbose --keyserver pgp.surf.nl --send-key 66DEF15282990C2199EFA801A8A128A8AB1CEE49
gpg: sending key A8A128A8AB1CEE49 to hkp://pgp.surf.nl
$ echo $?
0

Steps to reproduce the problem

Fully enable systemd-resolved, also with /etc/resolv.conf pointing to /run/systemd/resolve/stub-resolv.conf

Run gpg --keyserver "$keyserver" --send-key "$my_key"

Additional program output to the terminal or log subsystem illustrating the issue

$ resolvectl status 
Global
         Protocols: +LLMNR +mDNS +DNSOverTLS DNSSEC=yes/supported
  resolv.conf mode: stub
Current DNS Server: 9.9.9.9#dns.quad9.net
       DNS Servers: 9.9.9.9#dns.quad9.net

Link 2 (enp7s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS +DNSOverTLS DNSSEC=yes/supported
Current DNS Server: 192.168.178.1
       DNS Servers: 192.168.178.1

$ journalctl --no-hostname --boot --follow
giu 14 11:32:51 dirmngr[3727]: command 'KS_PUT' failed: Server indicated a failure <Unspecified source>

$ gpg --verbose --verbose --debug-all --keyserver pgp.surf.nl --send-key 66DEF15282990C2199EFA801A8A128A8AB1CEE49
gpg: reading options from '/home/tachi/.gnupg/gpg.conf'
gpg: reading options from '[cmdline]'
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing ipc clock lookup extprog
gpg: DBG: [not enabled in the source] start
gpg: DBG: chan_3 <- # Home: /home/tachi/.gnupg
gpg: DBG: chan_3 <- # Config: /home/tachi/.gnupg/dirmngr.conf
gpg: DBG: chan_3 <- OK Dirmngr 2.2.35 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.2.35
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkp://pgp.surf.nl
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER
gpg: DBG: chan_3 <- S KEYSERVER hkp://pgp.surf.nl
gpg: DBG: chan_3 <- OK
gpg: DBG: [not enabled in the source] keydb_new
gpg: DBG: [not enabled in the source] keydb_search enter
gpg: DBG: keydb_search: 1 search descriptions:
gpg: DBG: keydb_search   0: FPR20: '66DE F152 8299 0C21 99EF  A801 A8A1 28A8 AB1C EE49'
gpg: DBG: keydb_search: searching keybox (resource 0 of 1)
gpg: DBG: keydb_search: searched keybox (resource 0 of 1) => Success
gpg: DBG: [not enabled in the source] keydb_search leave (found)
gpg: DBG: [not enabled in the source] keydb_get_keybock enter
gpg: DBG: parse_packet(iob=2): type=6 length=51 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=12 length=12 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=13 length=57 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=12 length=12 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=2 length=144 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=12 length=6 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=2 length=563 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=12 length=6 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=14 length=56 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=2 length=120 (parse.../../g10/keydb.c.1257)
gpg: DBG: parse_packet(iob=2): type=12 length=6 (parse.../../g10/keydb.c.1257)
gpg: DBG: iobuf-2.0: underflow: buffer size: 1056; still buffered: 0 => space for 1056 bytes
gpg: DBG: iobuf-2.0: close '?'
gpg: DBG: [not enabled in the source] keydb_get_keyblock leave
gpg: DBG: build_packet() type=6
gpg: DBG: iobuf-3.0: close '?'
gpg: DBG: build_packet() type=13
gpg: DBG: build_packet() type=2
gpg: DBG: iobuf-4.0: close '?'
gpg: DBG: build_packet() type=2
gpg: DBG: iobuf-5.0: close '?'
gpg: DBG: build_packet() type=14
gpg: DBG: iobuf-6.0: close '?'
gpg: DBG: build_packet() type=2
gpg: DBG: iobuf-7.0: close '?'
gpg: DBG: iobuf-1.0: close '?'
gpg: sending key A8A128A8AB1CEE49 to hkp://pgp.surf.nl
gpg: DBG: chan_3 -> KS_PUT
gpg: DBG: chan_3 <- INQUIRE KEYBLOCK
gpg: DBG: chan_3 -> [ 44 20 98 33 04 5f ef 42 de 16 09 2b 06 01 04 01 ...(982 byte(s) skipped) ]
gpg: DBG: chan_3 -> [ 44 20 c8 0f 7a e9 7c ac 90 88 cc f5 6a 30 75 5d ...(16 byte(s) skipped) ]
gpg: DBG: chan_3 -> END
gpg: DBG: chan_3 <- INQUIRE KEYBLOCK_INFO
gpg: DBG: chan_3 -> D pub::255:22:A8A128A8AB1CEE49:1609515742::::::::::::::%0Afpr:::::::::66DEF15282990C2199EFA801A8A128A8AB1CEE49:<snip>
gpg: DBG: chan_3 -> END
gpg: DBG: chan_3 <- ERR 219 Server indicated a failure <Unspecified source>
gpg: DBG: free_packet() type=6
gpg: DBG: free_packet() type=13
gpg: DBG: free_packet() type=2
gpg: DBG: free_packet() type=2
gpg: DBG: free_packet() type=14
gpg: DBG: free_packet() type=2
gpg: keyserver send failed: Server indicated a failure
gpg: keyserver send failed: Server indicated a failure
gpg: DBG: chan_3 -> BYE
gpg: DBG: [not enabled in the source] stop
gpg: keydb: handles=1 locks=0 parse=1 get=1
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=1 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=2 cached=2 good=2 bad=0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks

The text was updated successfully, but these errors were encountered:

yuwata · 2022-06-27T16:05:38Z

resolvectl query or dig works for the host?? Could you show the output of these commands?

Tachi107 · 2022-06-27T20:23:25Z

resolvectl query or dig works for the host?? Could you show the output of these commands?

Yes, they work. delv doesn't, but that's probably related to #23289 (comment)

$ resolvectl query pgp.surf.nl
pgp.surf.nl: 2001:610:188:452:145:100:176:15   -- link: enp7s0
             145.100.176.15                    -- link: enp7s0

-- Information acquired via protocol DNS in 185.0ms.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
-- Data from: network

$ dig pgp.surf.nl

; <<>> DiG 9.18.1-1-Debian <<>> pgp.surf.nl
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60413
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;pgp.surf.nl.			IN	A

;; ANSWER SECTION:
pgp.surf.nl.		79	IN	A	145.100.176.15

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Jun 27 22:21:32 CEST 2022
;; MSG SIZE  rcvd: 56

$ delv pgp.surf.nl
;; broken trust chain resolving 'surf.nl/DS/IN': 127.0.0.53#53
;; broken trust chain resolving 'surf.nl/DNSKEY/IN': 127.0.0.53#53
;; broken trust chain resolving 'pgp.surf.nl/A/IN': 127.0.0.53#53
;; resolution failed: broken trust chain

yuwata · 2022-07-03T23:08:58Z

Do you have any trust anchors file in {/etc,/usr/lib,/run}/dnssec-trust-anchors.d ??
I cannot reproduce the issue...

Tachi107 · 2022-07-04T10:28:17Z

Yes, I don't have those dirs at all.

I've been also able to reproduce this on systemd 247.3 (Debian stable).

Edit: how did you try to reproduce this? Note that I have both DoT and DNSSEC enabled. I did not try changing DNS server though (only used quad9)

mrc0mmand · 2022-07-04T11:08:51Z

I just tried it with the latest main and can't reproduce it either:

# resolvectl
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=allow-downgrade/supported
    resolv.conf mode: stub
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com
                      2001:4860:4860::8888#dns.google 2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google

Link 2 (eth0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported
Current DNS Server: 9.9.9.9
       DNS Servers: 9.9.9.9

# resolvectl --cache=no query pgp.surf.nl
pgp.surf.nl: 2001:610:188:452:145:100:176:15   -- link: eth0
             145.100.176.15                    -- link: eth0

-- Information acquired via protocol DNS in 256.3ms.
-- Data is authenticated: yes; Data was acquired via local or encrypted transport: yes
-- Data from: network

# gpg -vvv --debug all --keyserver pgp.surf.nl --send-key 965903AAE5C10D90CF2D0BBFE9083485235552BB
gpg: reading options from '[cmdline]'
gpg: using character set 'utf-8'
gpg: enabled debug flags: packet mpi crypto filter iobuf memory cache memstat trust hashing ipc clock lookup extprog
gpg: DBG: [no clock] start
gpg: DBG: chan_3 <- # Home: /root/.gnupg
gpg: DBG: chan_3 <- # Config: /root/.gnupg/dirmngr.conf
gpg: DBG: chan_3 <- OK Dirmngr 2.3.6 at your service
gpg: DBG: connection to the dirmngr established
gpg: DBG: chan_3 -> GETINFO version
gpg: DBG: chan_3 <- D 2.3.6
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER --clear hkp://pgp.surf.nl
gpg: DBG: chan_3 <- OK
gpg: DBG: chan_3 -> KEYSERVER
gpg: DBG: chan_3 <- S KEYSERVER hkp://pgp.surf.nl
gpg: DBG: chan_3 <- OK
gpg: DBG: [no clock] keydb_new
gpg: DBG: [no clock] keydb_search enter
gpg: DBG: keydb_search: 1 search descriptions:
gpg: DBG: keydb_search   0: FPR20: '9659 03AA E5C1 0D90 CF2D  0BBF E908 3485 2355 52BB'
gpg: DBG: internal_keydb_search: searching keybox (resource 0 of 1)
gpg: DBG: internal_keydb_search: searched keybox (resource 0 of 1) => Success
gpg: DBG: [no clock] keydb_search leave (found)
gpg: DBG: [no clock] keydb_get_keyblock enter
gpg: DBG: parse_packet(iob=2): type=6 length=51 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=12 length=12 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=13 length=63 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=12 length=12 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=2 length=153 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=12 length=6 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=14 length=56 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=2 length=126 (parse.keydb.c.1161)
gpg: DBG: parse_packet(iob=2): type=12 length=6 (parse.keydb.c.1161)
gpg: DBG: iobuf-2.0: underflow: buffer size: 503; still buffered: 0 => space for 503 bytes
gpg: DBG: iobuf-2.0: close '?'
gpg: DBG: [no clock] keydb_get_keyblock leave
gpg: DBG: build_packet() type=6
gpg: DBG: iobuf-3.0: close '?'
gpg: DBG: build_packet() type=13
gpg: DBG: build_packet() type=2
gpg: DBG: iobuf-4.0: close '?'
gpg: DBG: build_packet() type=14
gpg: DBG: iobuf-5.0: close '?'
gpg: DBG: build_packet() type=2
gpg: DBG: iobuf-6.0: close '?'
gpg: DBG: [no clock] keydb_release
gpg: DBG: iobuf-1.0: close '?'
gpg: sending key E9083485235552BB to hkp://pgp.surf.nl
gpg: DBG: ecc_verify info: Edwards/Ed25519+EdDSA
gpg: DBG: ecc_verify name: Ed25519
gpg: DBG: ecc_verify    p:+7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffed
gpg: DBG: ecc_verify    a:+7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffec
gpg: DBG: ecc_verify    b:+52036cee2b6ffe738cc740797779e89800700a4d4141d8ab75eb4dca135978a3
gpg: DBG: ecc_verify  g.X:+216936d3cd6e53fec0a4e231fdd6dc5c692cc7609525a7b2c9562d608f25d51a
gpg: DBG: ecc_verify  g.Y:+6666666666666666666666666666666666666666666666666666666666666658
gpg: DBG: ecc_verify  g.Z:+01
gpg: DBG: ecc_verify    n:+1000000000000000000000000000000014def9dea2f79cd65812631a5cf5d3ed
gpg: DBG: ecc_verify    h:+08
gpg: DBG: ecc_verify    q: [264 bit]
gpg: DBG:                  405af4a4f6b4cfd8964f121bfd0c22502544d175698f42a78a01726dbb1d89c3 \
gpg: DBG:                  c4
gpg: DBG: ecc_verify data: [512 bit]
gpg: DBG:                  51c7a140c819bb30f8364b54bf10348b4e0b58aa2b7ef8bed769a17990b70ee4 \
gpg: DBG:                  3dc9451942ce836f131971c8f36f79aefc10289e3a746c94926a1305458c381a
gpg: DBG: ecc_verify  s_r: [256 bit]
gpg: DBG:                  779fe72e43d4f78669735ac14b49920621813e27aaa075964cad196c990b1214
gpg: DBG: ecc_verify  s_s: [256 bit]
gpg: DBG:                  563f5853bb059d14f819243bb728146dcbdf91e60c5e0866ec3df46f9b122d0e
gpg: DBG:   e_pk: 5af4a4f6b4cfd8964f121bfd0c22502544d175698f42a78a01726dbb1d89c3c4
gpg: DBG:      m: 51c7a140c819bb30f8364b54bf10348b4e0b58aa2b7ef8bed769a17990b70ee4 \
gpg: DBG:         3dc9451942ce836f131971c8f36f79aefc10289e3a746c94926a1305458c381a
gpg: DBG:      r: 779fe72e43d4f78669735ac14b49920621813e27aaa075964cad196c990b1214
gpg: DBG:  H(R+): 4659fb38be77c3516bcbca8682591792348deb86ad16ae09fc61c502935e903a \
gpg: DBG:         35a2c34620431e194b316b9379bb70681f8d127b4e35234f7e9c067d26706b4f
gpg: DBG:      s: 0e2d129b6ff43dec66085e0ce691dfcb6d1428b73b2419f8149d05bb53583f56
gpg: DBG: ecc_verify    => Good
gpg: DBG: ecc_verify info: Edwards/Ed25519+EdDSA
gpg: DBG: ecc_verify name: Ed25519
gpg: DBG: ecc_verify    p:+7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffed
gpg: DBG: ecc_verify    a:+7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffec
gpg: DBG: ecc_verify    b:+52036cee2b6ffe738cc740797779e89800700a4d4141d8ab75eb4dca135978a3
gpg: DBG: ecc_verify  g.X:+216936d3cd6e53fec0a4e231fdd6dc5c692cc7609525a7b2c9562d608f25d51a
gpg: DBG: ecc_verify  g.Y:+6666666666666666666666666666666666666666666666666666666666666658
gpg: DBG: ecc_verify  g.Z:+01
gpg: DBG: ecc_verify    n:+1000000000000000000000000000000014def9dea2f79cd65812631a5cf5d3ed
gpg: DBG: ecc_verify    h:+08
gpg: DBG: ecc_verify    q: [264 bit]
gpg: DBG:                  405af4a4f6b4cfd8964f121bfd0c22502544d175698f42a78a01726dbb1d89c3 \
gpg: DBG:                  c4
gpg: DBG: ecc_verify data: [512 bit]
gpg: DBG:                  5d0d4a5ce94904066012e53c5ba2f2bb4fdb5d80be52fb5e34c11d932dda7fc7 \
gpg: DBG:                  fd13b65cc653dd578d156e22b5532813f0dff3235bf53a32b7576648fbfdce43
gpg: DBG: ecc_verify  s_r: [256 bit]
gpg: DBG:                  9568700cd6f57efaf7be0ac4e9e4755e462f3f3a96c9a32b7dadbcf570781633
gpg: DBG: ecc_verify  s_s: [256 bit]
gpg: DBG:                  5cd16ab5b79ee7dfd0105cfdbb0daf1f605e8aba16550e125e55a24b2dd5a40c
gpg: DBG:   e_pk: 5af4a4f6b4cfd8964f121bfd0c22502544d175698f42a78a01726dbb1d89c3c4
gpg: DBG:      m: 5d0d4a5ce94904066012e53c5ba2f2bb4fdb5d80be52fb5e34c11d932dda7fc7 \
gpg: DBG:         fd13b65cc653dd578d156e22b5532813f0dff3235bf53a32b7576648fbfdce43
gpg: DBG:      r: 9568700cd6f57efaf7be0ac4e9e4755e462f3f3a96c9a32b7dadbcf570781633
gpg: DBG:  H(R+): 54a96bf0fe76e8ac761678e80473deb4ac0c39fd540abd33f2b0c7b98774e5aa \
gpg: DBG:         ec234495d30245076cc0494676f2487f641d4da0ef14c4fc3618d1e3935e9fc7
gpg: DBG:      s: 0ca4d52d4ba2555e120e5516ba8a5e601faf0dbbfd5c10d0dfe79eb7b56ad15c
gpg: DBG: ecc_verify    => Good
gpg: DBG: chan_3 -> KS_PUT
gpg: DBG: chan_3 <- INQUIRE KEYBLOCK
gpg: DBG: chan_3 -> [ 44 20 98 33 04 62 c2 c9 45 16 09 2b 06 01 04 01 ...(471 byte(s) skipped) ]
gpg: DBG: chan_3 -> END
gpg: DBG: chan_3 <- INQUIRE KEYBLOCK_INFO
gpg: DBG: chan_3 -> D pub::255:22:E9083485235552BB:1656932677:1659524677:::::::::::::%0Afpr:::::::::965903AAE5C10D90CF2D0BBFE9083485235552BB:%0Auid:::::1656932677::::systemd-resolved-test (Test key) <systemd-resolved@just.a.test>:::::::::%0Asub::255:18:1D0E4A07E3C7E3F5:1656932677:1659524677:::::::::::::%0Afpr:::::::::035BF81DA48496D9FCE93C811D0E4A07E3C7E3F5:%0A
gpg: DBG: chan_3 -> END
gpg: DBG: chan_3 <- S PROGRESS tick ? 0 0
gpg: DBG: chan_3 <- OK
gpg: DBG: free_packet() type=6
gpg: DBG: free_packet() type=13
gpg: DBG: free_packet() type=2
gpg: DBG: free_packet() type=14
gpg: DBG: free_packet() type=2
gpg: DBG: chan_3 -> BYE
gpg: DBG: [no clock] stop
gpg: keydb: handles=1 locks=0 parse=1 get=1
gpg:        build=0 update=0 insert=0 delete=0
gpg:        reset=0 found=1 not=0 cache=0 not=0
gpg: kid_not_found_cache: count=0 peak=0 flushes=0
gpg: sig_cache: total=2 cached=0 good=0 bad=0
gpg: objcache: keys=0/0/0 chains=0,0..0 buckets=0/0 attic=0
gpg: objcache: uids=0/0/0 chains=0,0..0 buckets=0/0
gpg: random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
              outmix=0 getlvl1=0/0 getlvl2=0/0
gpg: rndjent stat: collector=0x0000000000000000 calls=0 bytes=0
gpg: secmem usage: 0/65536 bytes in 0 blocks

yuwata · 2022-07-04T13:41:15Z

@Tachi107 I tried with 9.9.9.9 and both DNSOverTLS and DNSSEC are enabled.
Also, I cannot reproduce the issue reported at #23289 (comment).

Tachi107 · 2022-07-04T13:45:28Z

Uhm, strange... I'll try to reproduce it on non-Debian systems.

Tachi107 · 2022-07-13T19:58:11Z

I've been able to reproduce this on Fedora 36 Live (bare metal, not VM). I've encountered all the issues I described above. The commands were run on a clean Live image, and the only things I changed from the default system config were some resolved settings to enable DNSSEC and DoT.

$ cat /etc/systemd/resolved.conf.d/dns_over_tls.conf
[Resolve]
DNS=9.9.9.9#dns.quad9.net
DNSOverTLS=yes

$ cat /etc/systemd/resolved.conf.d/dnssec.conf
[Resolve]
DNSSEC=yes

$ resolvectl --version
systemd 250 (v250.3-8.fc36)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

$ resolvectl flush-caches

$ gpg --keyserver pgp.surf.nl --send-keys 4DFB675E669CBB97C7B56AC72EB75202983682D7
gpg: sending key 2EB75202983682D7 to hkp://pgp.surf.nl
gpg: keyserver send failed: Server indicated a failure
gpg: keyserver send failed: Server indicated a failure

$ delv +mtrace +vtrace apps.fedoraproject.org
;; fetch: apps.fedoraproject.org/A
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  17794
;; flags: qr tc rd ra ad; QUESTION: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;apps.fedoraproject.org.		IN	A

;; ANSWER SECTION:
;apps.fedoraproject.org.	269	IN	CNAME	wildcard.fedoraproject.org.
;apps.fedoraproject.org.	269	IN	RRSIG	CNAME 14 3 300 (
;						20220730141214 20220630141214 60624 fedoraproject.org.
;						r/zBHuC4AMeNCuxGb9QErCMbFtIi
;						DneL+E0hhUAB3moTGjljqZm/lhj4
;						pxI8Nv0OOIChsq9vD3ne+c44vHeO
;						NPGnNjE3pCSgJhe9lDYdO+ldWNkd
;						Jh4uUdq5nx73haFT )
;wildcard.fedoraproject.org. 58	IN	A	152.19.134.198
;wildcard.fedoraproject.org. 58	IN	A	85.236.55.6
;wildcard.fedoraproject.org. 58	IN	A	18.159.254.57
;wildcard.fedoraproject.org. 58	IN	A	18.133.140.134
;wildcard.fedoraproject.org. 58	IN	A	152.19.134.142
;wildcard.fedoraproject.org. 58	IN	A	185.141.165.254
;wildcard.fedoraproject.org. 58	IN	A	38.145.60.21
;wildcard.fedoraproject.org. 58	IN	A	18.192.40.85
;wildcard.fedoraproject.org. 58	IN	A	209.132.190.2
;wildcard.fedoraproject.org. 58	IN	RRSIG	A 14 3 60 (
;						20220730141214 20220630141214 60624 fedoraproject.org.
;						qm92mLmhexhwRgLziQPVljw19V3t
;						el6nQ9xrlZ1SdnNtqsftco3GTypH
;						0msXRdnsFxkZB/S5LsD48xQb7qta
;						xvh4GFd5mYMRroIH40A+k47OJ6+i
;						afcjCsAqGG25TmdB )


;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  47087
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;apps.fedoraproject.org.		IN	A

;; ANSWER SECTION:
;apps.fedoraproject.org.	57	IN	CNAME	wildcard.fedoraproject.org.
;apps.fedoraproject.org.	57	IN	RRSIG	CNAME 14 3 300 (
;						20220730141214 20220630141214 60624 fedoraproject.org.
;						r/zBHuC4AMeNCuxGb9QErCMbFtIi
;						DneL+E0hhUAB3moTGjljqZm/lhj4
;						pxI8Nv0OOIChsq9vD3ne+c44vHeO
;						NPGnNjE3pCSgJhe9lDYdO+ldWNkd
;						Jh4uUdq5nx73haFT )
;wildcard.fedoraproject.org. 57	IN	A	38.145.60.21
;wildcard.fedoraproject.org. 57	IN	A	85.236.55.6
;wildcard.fedoraproject.org. 57	IN	A	152.19.134.198
;wildcard.fedoraproject.org. 57	IN	A	185.141.165.254
;wildcard.fedoraproject.org. 57	IN	A	18.159.254.57
;wildcard.fedoraproject.org. 57	IN	A	152.19.134.142
;wildcard.fedoraproject.org. 57	IN	A	18.192.40.85
;wildcard.fedoraproject.org. 57	IN	A	38.145.60.20
;wildcard.fedoraproject.org. 57	IN	A	18.133.140.134
;wildcard.fedoraproject.org. 57	IN	A	209.132.190.2
;wildcard.fedoraproject.org. 57	IN	RRSIG	A 14 3 60 (
;						20220730141214 20220630141214 60624 fedoraproject.org.
;						qm92mLmhexhwRgLziQPVljw19V3t
;						el6nQ9xrlZ1SdnNtqsftco3GTypH
;						0msXRdnsFxkZB/S5LsD48xQb7qta
;						xvh4GFd5mYMRroIH40A+k47OJ6+i
;						afcjCsAqGG25TmdB )


;; validating apps.fedoraproject.org/CNAME: starting
;; validating apps.fedoraproject.org/CNAME: attempting positive response validation
;; fetch: fedoraproject.org/DNSKEY
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  58670
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;fedoraproject.org.		IN	DNSKEY

;; ANSWER SECTION:
;fedoraproject.org.	222	IN	DNSKEY	257 3 14 (
;						7ttmhus8JD56ybsvMVZVsXa3U2R+
;						2+WmOPIP7BU6t2LicosMZ2Ju3pfv
;						ijsa5LvBvVCB4xVtLSqEdLSvW4vJ
;						PLSAB2uyJwHPJMezh0SzGmVCImLU
;						6qDxsxjHqtZ76/Sf
;						) ; KSK; alg = ECDSAP384SHA384 ; key id = 58125
;fedoraproject.org.	222	IN	DNSKEY	256 3 14 (
;						04ZsDOgyzs3kJsJ4jEY3MYufkCOW
;						m1OI8N4M+dlBOBmweln0TSaKfafH
;						zNCkaPiVG4bdgdnrzwxmjpK5GQgs
;						iB47np+I8850Ea3EJG5ORDl3f//l
;						rr92HiYh5DxCNhkG
;						) ; ZSK; alg = ECDSAP384SHA384 ; key id = 60624
;fedoraproject.org.	222	IN	RRSIG	DNSKEY 14 2 300 (
;						20220730141214 20220630141214 58125 fedoraproject.org.
;						kh7KSKFrOxnB/po/koxjc40SAyMR
;						muwMXYjw6bwopP79lbNVKECz2JXs
;						0/OzJ5K5JMVI+AzqaatM5V8ZXTg5
;						+sZ0tEEBOxWp5W6L5ih/+GCp9Qmw
;						3VfSOuhV3dI9+j0V )


;; validating fedoraproject.org/DNSKEY: starting
;; validating fedoraproject.org/DNSKEY: attempting positive response validation
;; fetch: fedoraproject.org/DS
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:   2113
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;fedoraproject.org.		IN	DS

;; ANSWER SECTION:
;fedoraproject.org.	2190	IN	DS	58125 14 2 (
;						FCC70DB7608C9837F060D6D92DF9
;						E53A22D1F830752B9E7038FC48EA
;						411DFF46 )
;fedoraproject.org.	2190	IN	RRSIG	DS 8 2 3600 (
;						20220730152415 20220709142415 52626 org.
;						aYd/MkjeQqJk3BT/VM/Rcqz1NPB9
;						hqCgb35fSpBhgw06NYuHbIXQ6dXI
;						xJ3eF4xXOt3pW5TrryfEaJzKpXWS
;						fnUlSxMjepzMw9D17U+unySTuy2Z
;						WuOgVM/PbUGPY+oULlt81SP4JKFA
;						/W7g7qY7xdBRizX9JUpv5YzbzExz
;						64g= )


;; validating fedoraproject.org/DS: starting
;; validating fedoraproject.org/DS: attempting positive response validation
;; fetch: org/DNSKEY
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id:  45331
;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;org.				IN	DNSKEY


;; validating fedoraproject.org/DS: in fetch_callback_dnskey
;; validating fedoraproject.org/DS: fetch_callback_dnskey: got SERVFAIL
;; broken trust chain resolving 'fedoraproject.org/DS/IN': 127.0.0.53#53
;; validating fedoraproject.org/DNSKEY: in fetch_callback_ds
;; validating fedoraproject.org/DNSKEY: fetch_callback_ds: got broken trust chain
;; broken trust chain resolving 'fedoraproject.org/DNSKEY/IN': 127.0.0.53#53
;; validating apps.fedoraproject.org/CNAME: in fetch_callback_dnskey
;; validating apps.fedoraproject.org/CNAME: fetch_callback_dnskey: got broken trust chain
;; broken trust chain resolving 'apps.fedoraproject.org/A/IN': 127.0.0.53#53
;; resolution failed: broken trust chain

pemensik · 2022-11-04T11:08:22Z

I think this is again related to breakages caused by LLMNR single label queries not forwarded to correct destination. I am testing systemd-resolved-252-587.fc38.x86_64 in a rawhide container.

container configuration

# resolvectl
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=yes/supported
  resolv.conf mode: foreign
Current DNS Server: 127.0.0.1
       DNS Servers: 127.0.0.1

Link 2 (enp0s31f6)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=yes/supported

Link 3 (wlp4s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=yes/supported

Link 7 (tun0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=yes/supported

verification upstream server correctly provides everything required

# delv +mtrace +vtrace @127.0.0.1 nic.cz
;; fetch: nic.cz/A
;; received packet from 127.0.0.1#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  29106
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;nic.cz.				IN	A

;; ANSWER SECTION:
;nic.cz.			1356	IN	A	217.31.205.50
;nic.cz.			1356	IN	RRSIG	A 13 2 1800 (
;						20221114042553 20221031025553 52236 nic.cz.
;						dZ39MPPqGAea5P3aJXMpI/dWZFhv
;						jkJv4pODg6O31zZnI15/n9QxDEDb
;						U6FEV/eNGFzWuxgRuhDjytSHalpA
;						Ow== )


;; validating nic.cz/A: starting
;; validating nic.cz/A: attempting positive response validation
;; fetch: nic.cz/DNSKEY
;; received packet from 127.0.0.1#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  56084
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;nic.cz.				IN	DNSKEY

;; ANSWER SECTION:
;nic.cz.			984	IN	DNSKEY	257 3 13 (
;						+980vJo8ZrP/VmzJJlzv7UJuIdGS
;						BVoIncX/4SipSDMN6zCuZNtGOnbh
;						ZvQJgA01rJCpiHnTPXqJHgV/HdH5
;						ZA==
;						) ; KSK; alg = ECDSAP256SHA256 ; key id = 43082
;nic.cz.			984	IN	DNSKEY	256 3 13 (
;						3J6qqPDzjLTf1UmOeg16byXH9CJ5
;						ETo8N+49Bs1l5DN3H0VJ8aVdczJn
;						qXXvolA3Nci5S947k9+zo+wRjLz8
;						jQ==
;						) ; ZSK; alg = ECDSAP256SHA256 ; key id = 52236
;nic.cz.			984	IN	RRSIG	DNSKEY 13 2 1800 (
;						20221114055553 20221031042553 43082 nic.cz.
;						d273CcRpXWhK45WLbJjcgVT1hC/A
;						a47t6YmSsDmjIyADnGZcvMP8PF7z
;						EVxA57U2GeD5tIN5wu4gPpH5GQtO
;						Ew== )


;; validating nic.cz/DNSKEY: starting
;; validating nic.cz/DNSKEY: attempting positive response validation
;; fetch: nic.cz/DS
;; received packet from 127.0.0.1#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  61652
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;nic.cz.				IN	DS

;; ANSWER SECTION:
;nic.cz.			3457	IN	DS	43082 13 2 (
;						8A6B4C612016AFBDB7883ED45CB0
;						28A71CE1825D7910A321CFC03DAB
;						99C2D87C )
;nic.cz.			3457	IN	RRSIG	DS 13 2 3600 (
;						20221115102035 20221101085035 44490 cz.
;						ORo6lh8MEtqKq2Qcs3JSSe08OXXQ
;						lVQt7w3OQzgfp7PCTT523lhKWl32
;						MiwZENne0LtzE6IgK4ACPnYimPHG
;						bg== )


;; validating nic.cz/DS: starting
;; validating nic.cz/DS: attempting positive response validation
;; fetch: cz/DNSKEY
;; received packet from 127.0.0.1#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  30928
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;cz.				IN	DNSKEY

;; ANSWER SECTION:
;cz.			776	IN	DNSKEY	256 3 13 (
;						DQWs8lNomUuEL8imLBUxwWg/lplX
;						O0vjEPfN9i9ss8tial0oZDrOaiuN
;						ecxWhQqOjCX/6mVaxQX3ZOfHeYjs
;						fA==
;						) ; ZSK; alg = ECDSAP256SHA256 ; key id = 44490
;cz.			776	IN	DNSKEY	257 3 13 (
;						nqzH7xP1QU5UOVy/VvxFSlrB/XgX
;						9JDJzj51PzIj35TXjZTyalTlAT/f
;						7PAfaSD5mEG1N8Vk9NmI2nxgQqhz
;						DQ==
;						) ; KSK; alg = ECDSAP256SHA256 ; key id = 20237
;cz.			776	IN	RRSIG	DNSKEY 13 1 3600 (
;						20221115142134 20221101125134 20237 cz.
;						8fX2XbidbVDvKtS1T69a3TBibBix
;						yv/kuAcICJ7365VzXLcYNE08GsWB
;						jD38GVb84v2lCrKLMQkC+Wy9RqGO
;						TA== )


;; validating cz/DNSKEY: starting
;; validating cz/DNSKEY: attempting positive response validation
;; fetch: cz/DS
;; received packet from 127.0.0.1#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  10888
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;cz.				IN	DS

;; ANSWER SECTION:
;cz.			80876	IN	DS	20237 13 2 (
;						CFF0F3ECDBC529C1F0031BA1840B
;						FB835853B9209ED1E508FFF48451
;						D7B778E2 )
;cz.			80876	IN	RRSIG	DS 8 1 86400 (
;						20221116050000 20221103040000 18733 .
;						yfAZhz4zrxYmPZE08mhc8rA3SOxo
;						BrWKjqwMfS7zATpmu4fYQX+cmTAQ
;						qiu+KgfPdatuBMNDUGYq7uKC3SW0
;						Nbjohpr0AZqSo5i2YLGltwF9Zp9G
;						nUXBsgIRTchOdQvg5D2kQL6mTOhb
;						9FUQMOMZmGgo/QwERMHs/xe3K7tO
;						1pk/kGx187s5lqm68tcjuqwGE+OH
;						k2UJZuZbt2ioPhtFNDF2owUIwRu/
;						MHjZCpXh+t3WaNb7Vc1TOZPo0lc8
;						Px2zIx7Xudrt/hXAegHvi/G59dpL
;						OWSvbrhHkGz0l8RzwK20PNnbWfyh
;						89QqD+PQrWu2jOKlYBK6cV5T9AX4
;						6gwYAA== )


;; validating cz/DS: starting
;; validating cz/DS: attempting positive response validation
;; fetch: ./DNSKEY
;; received packet from 127.0.0.1#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  19809
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;.				IN	DNSKEY

;; ANSWER SECTION:
;.			167243	IN	DNSKEY	257 3 8 (
;						AwEAAaz/tAm8yTn4Mfeh5eyI96WS
;						VexTBAvkMgJzkKTOiW1vkIbzxeF3
;						+/4RgWOq7HrxRixHlFlExOLAJr5e
;						mLvN7SWXgnLh4+B5xQlNVz8Og8kv
;						ArMtNROxVQuCaSnIDdD5LKyWbRd2
;						n9WGe2R8PzgCmr3EgVLrjyBxWezF
;						0jLHwVN8efS3rCj/EWgvIWgb9tar
;						pVUDK/b58Da+sqqls3eNbuv7pr+e
;						oZG+SrDK6nWeL3c6H5Apxz7LjVc1
;						uTIdsIXxuOLYA4/ilBmSVIzuDWfd
;						RUfhHdY6+cn8HFRm+2hM8AnXGXws
;						9555KrUB5qihylGa8subX2Nn6UwN
;						R1AkUTV74bU=
;						) ; KSK; alg = RSASHA256 ; key id = 20326
;.			167243	IN	DNSKEY	256 3 8 (
;						AwEAAeB54o2xvW6vY4qQZ0krDsEZ
;						Ce6MsRWCqsXd4+cNJZMePnlV/xwD
;						rIbbeH1SJzv742rOHzgAKM1/3SQH
;						HSkoEIPx8XQdHAZBxfhaXl3e8c5W
;						rE3aGXS5AeTWAkt85ccqWgKyitxj
;						FmJEOol0BqS2xueltaDwgWcC10nP
;						UY+y5l/kTOYyptYQS4gg1uJNXIob
;						/R1XIEJ10ZCurkYqZxgqyHc7tZv0
;						9N23o9rnGdjnYiArH7FjlXD8Rvjd
;						e8YWkmfdbCEWnchrnxDK8KV2/ZvB
;						pG/WYnRKXYPUceGCw59OJdJ5M7ut
;						km547RB3eEd8CVVhbXopZlsKq3GC
;						rBwaIVe9ci0=
;						) ; ZSK; alg = RSASHA256 ; key id = 18733
;.			167243	IN	RRSIG	DNSKEY 8 0 172800 (
;						20221121000000 20221031000000 20326 .
;						HbS5SOekfhv6fxvAdQE9IyoqoY/w
;						CbexFlIsL5bOLfzR1p3OqSu7hwvO
;						fhrmsp68yTQqjPj9FHBki/ZxB3xM
;						dl/+tFSEfY58EDBHsIOEG3eh36If
;						0199l4pzugoiPHGwbBtUw2vyx0hS
;						GVta/QE2cqg+wLHIx/DuD4AcgWBh
;						K3c5Fwhu4jxPy4/w+LE5+ojmPlah
;						zgk3WWAmhfSjEaVvJsA1pR/0tA8/
;						w/+O7f2gP2sJd9dGIpbgL1lwXxte
;						GcTfmnlNdTE8m2d75RLyYcD7IohE
;						GYyxABJYWoSVITa03E3TLl1zF6Er
;						ZMJWv6BMQW31hVEJ5DgBd9+4/hUv
;						nqPpbw== )


;; validating ./DNSKEY: starting
;; validating ./DNSKEY: attempting positive response validation
;; validating ./DNSKEY: verify rdataset (keyid=20326): success
;; validating ./DNSKEY: marking as secure (DS)
;; validating cz/DS: in fetch_callback_dnskey
;; validating cz/DS: keyset with trust secure
;; validating cz/DS: resuming validate
;; validating cz/DS: verify rdataset (keyid=18733): success
;; validating cz/DS: marking as secure, noqname proof not needed
;; validating cz/DNSKEY: in fetch_callback_ds
;; validating cz/DNSKEY: dsset with trust secure
;; validating cz/DNSKEY: verify rdataset (keyid=20237): success
;; validating cz/DNSKEY: marking as secure (DS)
;; validating nic.cz/DS: in fetch_callback_dnskey
;; validating nic.cz/DS: keyset with trust secure
;; validating nic.cz/DS: resuming validate
;; validating nic.cz/DS: verify rdataset (keyid=44490): success
;; validating nic.cz/DS: marking as secure, noqname proof not needed
;; validating nic.cz/DNSKEY: in fetch_callback_ds
;; validating nic.cz/DNSKEY: dsset with trust secure
;; validating nic.cz/DNSKEY: verify rdataset (keyid=43082): success
;; validating nic.cz/DNSKEY: marking as secure (DS)
;; validating nic.cz/A: in fetch_callback_dnskey
;; validating nic.cz/A: keyset with trust secure
;; validating nic.cz/A: resuming validate
;; validating nic.cz/A: verify rdataset (keyid=52236): success
;; validating nic.cz/A: marking as secure, noqname proof not needed
; fully validated
nic.cz.			1356	IN	A	217.31.205.50
nic.cz.			1356	IN	RRSIG	A 13 2 1800 20221114042553 20221031025553 52236 nic.cz. dZ39MPPqGAea5P3aJXMpI/dWZFhvjkJv4pODg6O31zZnI15/n9QxDEDb U6FEV/eNGFzWuxgRuhDjytSHalpAOw==

systemd-resolved not handling all upstream queries the same way, failing the whole chain.

# delv +mtrace +vtrace @127.0.0.53 nic.cz
;; fetch: nic.cz/A
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  22215
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;nic.cz.				IN	A

;; ANSWER SECTION:
;nic.cz.			1320	IN	A	217.31.205.50
;nic.cz.			1320	IN	RRSIG	A 13 2 1800 (
;						20221114042553 20221031025553 52236 nic.cz.
;						dZ39MPPqGAea5P3aJXMpI/dWZFhv
;						jkJv4pODg6O31zZnI15/n9QxDEDb
;						U6FEV/eNGFzWuxgRuhDjytSHalpA
;						Ow== )


;; validating nic.cz/A: starting
;; validating nic.cz/A: attempting positive response validation
;; fetch: nic.cz/DNSKEY
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  20958
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;nic.cz.				IN	DNSKEY

;; ANSWER SECTION:
;nic.cz.			948	IN	DNSKEY	257 3 13 (
;						+980vJo8ZrP/VmzJJlzv7UJuIdGS
;						BVoIncX/4SipSDMN6zCuZNtGOnbh
;						ZvQJgA01rJCpiHnTPXqJHgV/HdH5
;						ZA==
;						) ; KSK; alg = ECDSAP256SHA256 ; key id = 43082
;nic.cz.			948	IN	DNSKEY	256 3 13 (
;						3J6qqPDzjLTf1UmOeg16byXH9CJ5
;						ETo8N+49Bs1l5DN3H0VJ8aVdczJn
;						qXXvolA3Nci5S947k9+zo+wRjLz8
;						jQ==
;						) ; ZSK; alg = ECDSAP256SHA256 ; key id = 52236
;nic.cz.			948	IN	RRSIG	DNSKEY 13 2 1800 (
;						20221114055553 20221031042553 43082 nic.cz.
;						d273CcRpXWhK45WLbJjcgVT1hC/A
;						a47t6YmSsDmjIyADnGZcvMP8PF7z
;						EVxA57U2GeD5tIN5wu4gPpH5GQtO
;						Ew== )


;; validating nic.cz/DNSKEY: starting
;; validating nic.cz/DNSKEY: attempting positive response validation
;; fetch: nic.cz/DS
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:    981
;; flags: qr rd ra ad; QUESTION: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
;; QUESTION SECTION:
;nic.cz.				IN	DS

;; ANSWER SECTION:
;nic.cz.			3421	IN	DS	43082 13 2 (
;						8A6B4C612016AFBDB7883ED45CB0
;						28A71CE1825D7910A321CFC03DAB
;						99C2D87C )
;nic.cz.			3421	IN	RRSIG	DS 13 2 3600 (
;						20221115102035 20221101085035 44490 cz.
;						ORo6lh8MEtqKq2Qcs3JSSe08OXXQ
;						lVQt7w3OQzgfp7PCTT523lhKWl32
;						MiwZENne0LtzE6IgK4ACPnYimPHG
;						bg== )


;; validating nic.cz/DS: starting
;; validating nic.cz/DS: attempting positive response validation
;; fetch: cz/DNSKEY
;; received packet from 127.0.0.53#53
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id:  26634
;; flags: qr aa rd ra; QUESTION: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;cz.				IN	DNSKEY


;; validating nic.cz/DS: in fetch_callback_dnskey
;; validating nic.cz/DS: fetch_callback_dnskey: got SERVFAIL
;; broken trust chain resolving 'nic.cz/DS/IN': 127.0.0.53#53
;; validating nic.cz/DNSKEY: in fetch_callback_ds
;; validating nic.cz/DNSKEY: fetch_callback_ds: got broken trust chain
;; broken trust chain resolving 'nic.cz/DNSKEY/IN': 127.0.0.53#53
;; validating nic.cz/A: in fetch_callback_dnskey
;; validating nic.cz/A: fetch_callback_dnskey: got broken trust chain
;; broken trust chain resolving 'nic.cz/A/IN': 127.0.0.53#53
;; resolution failed: broken trust chain

It just returns SERVFAIL on single label names, thus breaking the ability to validate whole chain.

pemensik · 2022-11-04T11:14:09Z

When I set also LLMNR=no, it starts to work again.

pemensik · 2022-11-04T11:28:15Z

Better demonstrated with drill from ldns-utils package.

LLMNR=no

# drill -S @127.0.0.53 apps.fedoraproject.org
;; Number of trusted keys: 1
;; Chasing: apps.fedoraproject.org. A


DNSSEC Trust tree:
apps.fedoraproject.org. (CNAME)
|---fedoraproject.org. (DNSKEY keytag: 60624 alg: 14 flags: 256)
    |---fedoraproject.org. (DNSKEY keytag: 58125 alg: 14 flags: 257)
    |---fedoraproject.org. (DS keytag: 58125 digest type: 2)
        |---org. (DNSKEY keytag: 41406 alg: 8 flags: 256)
            |---org. (DNSKEY keytag: 26974 alg: 8 flags: 257)
            |---org. (DS keytag: 26974 digest type: 2)
                |---. (DNSKEY keytag: 18733 alg: 8 flags: 256)
                    |---. (DNSKEY keytag: 20326 alg: 8 flags: 257)
;; Chase successful

LLMNR=resolve

# drill -S @127.0.0.53 apps.fedoraproject.org
;; Number of trusted keys: 1
;; Chasing: apps.fedoraproject.org. A


DNSSEC Trust tree:
apps.fedoraproject.org. (CNAME)
|---fedoraproject.org. (DNSKEY keytag: 60624 alg: 14 flags: 256)
    |---fedoraproject.org. (DNSKEY keytag: 58125 alg: 14 flags: 257)
    |---fedoraproject.org. (DS keytag: 58125 digest type: 2)
No trusted keys found in tree: first error was: No DNSSEC public key(s)
;; Chase failed.

mrc0mmand added the resolve label Jun 14, 2022

yuwata added the needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer label Jun 27, 2022

yuwata added bug 🐛 Programming errors, that need preferential fixing and removed needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer labels Jun 27, 2022

yuwata added this to the v252 milestone Jun 27, 2022

yuwata added the cant-reproduce label Jul 4, 2022

poettering added the dnssec label Sep 5, 2022

poettering modified the milestones: v252, v253 Sep 5, 2022

Tachi107 mentioned this issue Nov 4, 2022

systemd-resolved breaks delv #25105

Open

pemensik mentioned this issue Nov 4, 2022

Handle LLMNR only by nss-resolve #23622

Open

pemensik mentioned this issue Dec 10, 2022

Local DNSSEC validation settings should not override query request #19227

Open

yuwata modified the milestones: v253, v254 Jan 17, 2023

resolved: `gpg --send-keys` fails with stub-resolv.conf #23737

resolved: `gpg --send-keys` fails with stub-resolv.conf #23737

Tachi107 commented Jun 14, 2022

yuwata commented Jun 27, 2022

Tachi107 commented Jun 27, 2022

yuwata commented Jul 3, 2022

Tachi107 commented Jul 4, 2022 •

edited

mrc0mmand commented Jul 4, 2022

yuwata commented Jul 4, 2022

Tachi107 commented Jul 4, 2022

Tachi107 commented Jul 13, 2022

pemensik commented Nov 4, 2022 •

edited

pemensik commented Nov 4, 2022

pemensik commented Nov 4, 2022

resolved: gpg --send-keys fails with stub-resolv.conf #23737

resolved: gpg --send-keys fails with stub-resolv.conf #23737

Comments

Tachi107 commented Jun 14, 2022

yuwata commented Jun 27, 2022

Tachi107 commented Jun 27, 2022

yuwata commented Jul 3, 2022

Tachi107 commented Jul 4, 2022 • edited

mrc0mmand commented Jul 4, 2022

yuwata commented Jul 4, 2022

Tachi107 commented Jul 4, 2022

Tachi107 commented Jul 13, 2022

pemensik commented Nov 4, 2022 • edited

pemensik commented Nov 4, 2022

pemensik commented Nov 4, 2022

resolved: `gpg --send-keys` fails with stub-resolv.conf #23737

resolved: `gpg --send-keys` fails with stub-resolv.conf #23737

Tachi107 commented Jul 4, 2022 •

edited

pemensik commented Nov 4, 2022 •

edited