stub-resolv.conf doesn't seem to work with ipv6

[ShapeShifter499]

Running Arch Linux here. I'm finding it difficult to figure this out. I have both systemd-networkd and systemd-resolved enabled on my system and a single config for a Ethernet connection. My system is headless and I'm connected over LAN via SSH.

It's mentioned that it's recommended to use /run/systemd/resolve/stub-resolv.conf with a symlink. But it doesn't seem to work with IPv6. things like "dig -6 google.com" fail.

Am I doing something wrong, how do I set this up with both IPv4 and IPv6? I have a IPv4 and IPv6 IP from my ISP and IPv6 connections work on a different system on the same LAN with GUI and NetworkManager

0 lrwxrwxrwx 1 root root 37 Mar 31 07:27 /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0

[root@kumo systemd]# resolvectl 
Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: yes
    DNSSEC supported: yes
  Current DNS Server: 192.168.1.1
         DNS Servers: 192.168.1.1
Fallback DNS Servers: fd51:3eef:8010::1
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (enp1s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: yes
    DNSSEC supported: yes
  Current DNS Server: 192.168.1.1
         DNS Servers: 192.168.1.1
                      fd51:3eef:8010::1
[root@kumo systemd]# 

/etc/systemd/resolved.conf.d/override.conf

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=192.168.1.1
FallbackDNS=fd51:3eef:8010::1
LLMNR=yes
MulticastDNS=yes
DNSSEC=yes

[root@kumo systemd]# dig -6 google.com 

; <<>> DiG 9.13.7 <<>> -6 google.com
;; global options: +cmd
;; connection timed out; no servers could be reached
[root@kumo systemd]# 

[poettering]

iirc "dig -6" uses IPv6 as transport, but the resolved stub doesn't listen on ipv6. That's because on ipv4 we have a whole subnet for the local host (127.0.0.0/8) while on IPv6 a single address only. On IPv4 we can thus pick an ip address from that range (127.0.0.53) to listen on, leaving the usual 127.0.0.1 everybody uses unaffected, and allowing something else to listen on that. On IPv6 all that is not possible, since there's only one address, and hence we opted not to listen on it at all given that it shouldn't matter what local address to listen on. After all the stub traffic never leaves the machine, hence why does it matter if it listens on ipv4 or ipv6?

So, what are you trying to do? And why does it matter if ipv4 or ipv6 is used to contact the local dns stub resolver?

[ShapeShifter499]

@poettering I was following what was recommended in the man file. I'm not actually 100% sure what "resolved stub" does This file is something that appears to be new since I don't ever remember reading about the file before.

Specifically this bit in man systemd-resolved

Note that /run/systemd/resolve/stub-resolv.conf should not be used directly by applications, but only
 through a symlink from /etc/resolv.conf. This file may be symlinked from /etc/resolv.conf in order to
 connect all local clients that bypass local DNS APIs to systemd-resolved with correct search domains
 settings. This mode of operation is recommended.

I had a Nextcloud setup working just fine but the issue popped up when I tried to install ClamAV and it gave me Invalid DNS reply. Falling back to HTTP mode. Following the FAQ from https://www.clamav.net/documents/troubleshooting-faq had me trying to mess around with /etc/resolv.conf. Since my server gets everything from my router via DHCP I'd typically setup a symlink ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf but I did double check the man page to be sure.

I'm probably misunderstanding the man page. Maybe the man page should be updated to be more clear about what "/run/systemd/resolve/stub-resolv.conf" is for and that IPv6 DNS is not supported.

Can someone please explain what "stub-resolv.conf" is for? Why was it added?

[ShapeShifter499]

I have switched the symlink back to ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf everything appears to work now. Still wondering about the above questions though.

[ShapeShifter499]

I spoke too soon, I'm now seeing a lot of these kind of messages.

Apr 01 05:17:12 kumo systemd-resolved[23971]: DNSSEC validation failed for question in-addr.arpa IN DS: incompatible-server

[poettering]

You apparently turned on DNSSEC validation but your local DNS server doesn#t actually support that. Turn it off.

Again, why did you use "dig -6" instead of regular "dig"?

[ShapeShifter499]

@poettering ok so DNSSEC validation is a OpenWRT issue. As it should just be forwarding DNS requests, I'll look into that.

I used 'dig' and 'dig -6' because I wanted to see both IPv4 and IPv6 connections working. I'm supposed to have both IPs from my ISP.

[ShapeShifter499]

@poettering so what does the local "DNS stub resolver" do exactly then? And why is it "recommended" in the man page? I thought unless you wanted some special DNS enteries for yourself that there wasn't a DNS server of any kind running under Linux by default. Everything went to the LAN router or a external DNS server like what Google hosts.

[ShapeShifter499]

@poettering any answers to my last post?

[carloabelli]

@ShapeShifter499

so what does the local "DNS stub resolver" do exactly then?

Using /run/systemd/resolve/resolv.conf will allow applications to use external DNS servers directly, bypassing systemd-resolved. Using /run/systemd/resolve/stub-resolv.conf will direct application to use systemd-resolved (listening on 127.0.0.53 locally) as an intermediary.

And why is it "recommended" in the man page?

The benefits to using systemd-resolved as an intermediary are numerous. You encountered one of them: if enabled, systemd-resolved will enforce DNSSEC on every outbound DNS request, otherwise, whether DNSSEC is enforced is application specific.

I thought unless you wanted some special DNS enteries for yourself that there wasn't a DNS server of any kind running under Linux by default. Everything went to the LAN router or a external DNS server like what Google hosts.

In both of these situations almost all of the requests are still being sent to an external DNS server.

If you want to test IPv6 connectivity, use something like ping -6 instead. Also there is no need for DNS to go over IPv6 to get AAAA records. Something like dig ipv6.google.com AAAA will do.

[ShapeShifter499]

So it turns out part of the issue was my system date was way out of sync. That can be blamed on having that system off for some time and "systemd-timesyncd" not wanting to update a system time that is weeks out of sync. I had to manually update the time with 'timedatectl' before "systemd-timesyncd" took over. I can thank folks at the #ArchLinux IRC channel over at Freenode for helping me finally find my glaring error.

Other issue with OpenWrt was solved by updating to the latest stable firmware, 18.06.2 at the time. Then removing 'dnsmaq' and 'odhcpd-ipv6only' packages installed by default and installing the 'dnsmasq-full' package. Then making sure the DNSSEC options were switched on, which at the time is below.

Adding the following two lines to /etc/config/dhcp

    option dnssec '1'
    option dnsseccheckunsigned '1'

@carloabelli Before I go and close this as solved and just 'user confusion', I want to make one thing clear. Making use of "/run/systemd/resolve/stub-resolv.conf" will still allow my system to resolve IPv6 enabled domain name addresses?

[carloabelli]

Yes, resolving IPv6 enabled domains (querying AAAA records) is entirely independent of the IP version used to access the DNS server. A client with only IPv4 connectivity can still retrieve AAAA records (but they are going to be pretty useless).

Note that if you only have IPv6 connectivity, just make sure systemd-resolved has IPv6 DNS servers listed under resolvectl. The local query will still be IPv4, but the external query will be done with IPv6.

[ShapeShifter499]

@carloabelli Thank you for helping me with the confusion I had. It didn't help I had a misconfigured system in the process.

Anyways I'm now going to close this issue ticket. Issue was user confusion and user error. Solved because a few people on IRC pointed out issues with how I had my system set up and @carloabelli finally helped me understand what "stub-resolv" is for. Things appear to be working correctly from my viewpoint.

[vovata]

Yes, resolving IPv6 enabled domains (querying AAAA records) is entirely independent of the IP version used to access the DNS server. A client with only IPv4 connectivity can still retrieve AAAA records (but they are going to be pretty useless).

Note that if you only have IPv6 connectivity, just make sure systemd-resolved has IPv6 DNS servers listed under resolvectl. The local query will still be IPv4, but the external query will be done with IPv6.

@carloabelli I have an interesting case. The system has only IPv6 connectivity. The systemd-resolved is configured to use stub-resolv.conf file. And the DNS server is coming from DHCP.
resolvectl dns returns correct dns server for the network interface.
nslookup returns sorrect IPv6 address when it is executed with the same dns server address.
But nslookup doesn't work without explicit dns server argument.
What could be the problem with the local dns service on 127.0.0.53?

[WanWizard]

It isn't specific to dig, all tools do this.

I currently can't connect to a remote server with "ssh -6" because that doesn't resolve (same reason as dig), and I need to force IPv6 for this connection since that server has an IPv4 config issue at the moment, and can't be reached via IPv4.

[saivert]

No this is only an issue for dig since in this case -6means use IPv6 transport (query DNS servers over IPv6) and doesn't mean resolve only AAAA record as for other tools like ping and ssh. Also nothing prevents systemd-resolved from adding a ULA on the loopback interface for the 127.0.0.53 equivalent. In fact one can do this by configuring the DNSStubListenerExtra= option in resolved.conf. What annoys me more is that link-local DNS servers (e.g fe80::1%interfacename) is apparently poorly supported by most DNS resolvers.