After connected to VPN, systemd-resolve still use ISP's DNS server( which was polluted because of regulation )

[joshuafc]

Submission type

• Request for enhancement (RFE)

systemd version the issue has been seen with

systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

Used distribution

Ubuntu 17.04 Linux version 4.10.0-21-generic (buildd@lgw01-12) (gcc version 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2) ) #23-Ubuntu SMP Fri Apr 28 16:14:22 UTC 2017

Because of the Internet regulation, ISP's DNS server can not resolve some domian correctly( for example www.google.com www.youtube.com ). Under normal conditions, we use vpn and set the vpn as default internet gateway when needed to access above website.
When I start to use Ubuntu 17.04, It is found that when connected to vpn, all internet package transfer through vpn except domain name resolve. Infact, VPN server had tell correct DNS server IP address through DHCP. But systemd-resolve still use ISP's polluted resolve.
At /etc/resolv.conf, I found that DNS resolve had be taken over by systemd-resolve. There can't find any easy way to tell systemd-resolve to use a specified DNS server, or disable any DNS server according " systemd-resolve --help ".
Willing to accept guidance to correct my use of systemd-resolve,
If you can, Please add some necessary feature to systemd-resolve to solve such problem,

In case of bug report: Expected behaviour you didn't see

…

In case of bug report: Unexpected behaviour you saw

…

In case of bug report: Steps to reproduce the problem

…

[poettering]

Hmm, not sure I follow. How exactly did you configure your DNS? What does "systemd-resolve --status" report about the DNS servers? Is /etc/resolv.conf a file or a symlink?

[joshuafc]

@poettering thanks for your reply.

here is the information

zhoub@ubuntu:\~$ ll /etc/resolv.conf 
rwxrwxrwx 1 root root 29 Jun  6 09:35 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
zhoub@ubuntu:~$ cat /etc/resolv.conf 
\# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
\#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
\# 127.0.0.53 is the systemd-resolved stub resolver.
\# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
zhoub@ubuntu:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 6 (ppp0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 2 (ens33)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no
         DNS Servers: 192.168.83.2
          DNS Domain: localdomain
zhoub@ubuntu:~$ ping www.google.com
PING www.google.com (93.46.8.89) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
106 packets transmitted, 0 received, 100% packet loss, time 107507ms

ping failed, 93.46.8.89 is incorrect , ISP's DNS server resolve it to 93.46.8.89 because of Internet regulation. VPN's DNS Server resolve it to 172.217.26.36, which can ping successfully.

[aigarius]

Can confirm the same effect with a OpenVPN connection.

Link 8 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.20.0.50
                      172.20.0.14

Link 3 (wlan0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.192.1
          DNS Domain: net

Running "systemd-resolve internal.name.local" fails to find it (it only tries the 192.* DNS server), running "systemd-resolve -i tun0 internal.name.local" resolves the name correctly (via 172.* DNS server), but no applications actually supply that interface, naturally.

Systemd version 232
Ubuntu 17.04

Currently I have to workaround this by overwriting the /etc/resolv.conf file each time I connect to the VPN, but even that now stops working as some apps use the systemd-resolv directly if it is available.

Alternative solution that worked for me is using a patched version of NetworkManager from https://bugs.launchpad.net/network-manager/+bug/1624317 that forces using the VPN DNS server by adding a "DNS Domain: ~." declaration to the tun0 section of systemd-resolve

[mikken]

You (joshuafc and aigarius) need to do 2 things:

1. Make sure that DNS servers obtained from OpenVPN are used by systemd-resolved. You can do it with this little script: https://github.com/jonathanio/update-systemd-resolved
2. While OpenVPN is connected, block (reject with iptables or nftables) all requests that systemd-resolved makes to local DNS server

[mikken]

aigarius, for you the domain .local should be configured as "routing" domain on this link (tun0).

[PureRumble]

@mikken, that script you're telling us to use is 400 lines long. What does it do? Any bugs? Is it safe or does it expose our systems to vulnerabilities? Will it break something else? Why do we need a 400 lines long script (and then to do even more stuff) to achieve something as essential as getting DNS lookups to go through the VPN connection?

Nearly every time someone connects to a VPN service the user's expected system behaviour is that the VPN service's DNS servers should be used and all traffic should go through the VPN connection; this is especially true for novice users. This should therefore be the default behaviour.

But not only is this not the default behavior, there also isn't some easy way to make systemd-resolved to work as desired (e.g. you have to use 400 lines long scripts, block packets to your ISP's DNS server at the IP-level, etc).

systemd-resolved is a "system service that provides network name resolution". It is beyond essential that a system that "provides network name resolution" should be able to with little effort to be made to route DNS requests through a VPN connection. Also, it's default behavior out-of-the-box should comply with the most common use cases (e.g. user connects to a VPN service ==> all data goes through VPN connection).

[flo-wer]

See #7182

[snabb]

My current workaround is to do the following tricks every time I start my workplace VPN:

1. Find out the link number (a.k.a. interface index) of the ISP interface: systemd-resolve --status or ip l.
2. Connect the VPN.
3. Remove DNS settings from the ISP interface (using the link number from step 1) by sending a D-Bus command to systemd-resolved. This is an example using link number 2: sudo busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDNS 'ia(iay)' 2 0
4. Inspect with systemd-resolve --status to ensure that only the correct DNS servers are there.

Apparently there isn't any simple cli tool for managing systemd-resolved's settings. That busctl command isn't very user-friendly. :)

[poettering]

So, resolved is actually doing the right thing here, given the configuration it was given. @joshuafc, resolved only knows the DNS server 192.168.83.2 in your setup, but you want it to know 208.67.222.222 and 8.8.8.8, right? If so, this information needs to be made available to resolved, and the ppp scripts (given that you are using ppp/pptp) need to push it into resolved, which i think ubuntu updated the "resolvconf" package to do. But that's really outside of the scope of upstream resolved. @nox, can you help? is there some ppp/pptp hookup missing for resolved in ubuntu?

[poettering]

@snabb #7576 adds the requested "simple cli" functionality to systemd-resolve.

[jetsaredim]

@poettering I'm confused by your most recent comment. If you review the contents of the most recent comment by @aigarius it shows that resolved does in fact have the DNS server addresses. He's just using google.com as an example to show the geo-located difference between his native connection and the vpn connection.

I came to this bug because I have a similar situation and basically the exact same symptoms as described in @aigarius' most recent comment with an openconnect vpn configuration. It's clear from the output that systemd-resolved has the DNS server info for the DNS servers inside the VPN, but it's not updating the DNS cache to bring in the new DNS servers and thus having trouble resolving things inside the private VPN network that should resolve easily. And in fact do resolve easily if you point relevant commands at the correct interface/DNS server.

A simple restart of systemd-resolved upon vpn connect/disconnect seems to fix the problem, but alas it really shouldn't be needed. Systemd-resolved should just be flushing the cache when a new DNS server is added.

[arno01]

A simple restart of systemd-resolve upon vpn connect/disconnect does not really help.

1. connect to any VPN server and visit the https://www.dnsleaktest.com or https://ipleak.net
2. see your ISP DNS are being leaked. Not good.

The only reasonable workaround so far is the one provided by @snabb #6076 (comment) Edit: and the DOMAIN-ROUTE . along with the update-systemd-resolved script so it would instruct systemd-resolve to use the domain name routing (see below for more on this concept).

We need some way to make sure that the DNS servers provided by the VPN server are the only ones that should be used as long as the VPN connection stays active.

Upd/Edited:
While the script https://github.com/jonathanio/update-systemd-resolved can be updated used to ensure that the only VPN provided DNS servers are used by leveraging DNS Domain: ~. (see below), I am still thinking that systemd-resolve should handle the DNS priority management, specifically for the VPN DNS case as it really should have the VPN DNS set as a default, otherwise people are likely to face serious consequences (in some extremely regulated countries) without the knowledge about the DNS leaks and with a great trust to a VPN... As far as I understand, the DNS Domain: ~. is the right way of managing the DNS priority with the systemd-resolve, except that it is never set in the openVPN configuration profiles, nor the update-systemd-resolved script is in place by default. Which is still causing a big frustration to the users.
And yet, besides the original openvpn client, there is network-manager-openvpn-gnome which is not using the update-systemd-resolved script. The systemd-resolve could probably help in this kind of situation, perhaps, as long as if it was instructed to use the VPN's DNS only or would rather be just switching the DNS to the one which was activated the last, since the former would imply teaching gnome's network-manager (or any other kind of openVPN client) how to talk to systemd-resolve via the D-Bus. Edit: hum, turns out this is something that is already WIP (see Upd4 below).

Upd2:
It appears DOMAIN-ROUTE . does the trick when the script is used.

jonathanio/update-systemd-resolved@b417bac

script-security 2
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre

# prevent DNS leakage
dhcp-option DOMAIN-ROUTE .

Upd3:
To those who are interested, the ad hoc fix would be the following (pretty much same thing that the script does):

Do the following while connected to a VPN server.

$ systemd-resolve --status | grep -A8 tun0 
Link 24 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4

$ sudo busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDomains 'ia(sb)' 24 1 . true

$ systemd-resolve --status | grep -A8 tun0 
Link 24 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4
          DNS Domain: ~.

According to https://www.freedesktop.org/software/systemd/man/systemd.network.html

Domain name routing has no equivalent in the traditional glibc API,
which has no concept of domain name servers limited to a specific link.

Upd4:
I found people are working from both directions, some are patching the systemd-resolved to support systemd-resolved when NetworkManager's dns-priority is used (see man nm-settings):

https://bugzilla.gnome.org/show_bug.cgi?id=783569#c26
https://bugs.launchpad.net/network-manager/+bug/1624317/comments/81

And some are working on the NetworkManager to support the routing-only domains concept which was introduced by systemd-resolved - https://bugzilla.gnome.org/show_bug.cgi?id=746422

[jahnf]

I had the same problem with DNS Leaks.
System: Ubuntu 18.04 with systemd-resolve and imported ovpn settings from my VPN Provider into NetworkManager.

I created a script that I wanted to share which solved my problems, in the background via Network Manager - no additional manual calls needed anymore.
I put it together with the information in this thread and other forums. It has to go into /etc/NetworkManager/dispatcher.d/ (only executable and writeable by root).

This is executed by Network Manager every time there is a connection change - In my case I check if the vpn connection id contains a certain substring, if yes I set the DNS Domain for the new VPN connection - and no DNS leaks any more.

#!/bin/bash

IF=$1
STATUS=$2

# contains(string, substring)
#
# Returns 0 if the specified string contains the specified substring,
# otherwise returns 1.
contains() {
    string="$1"
    substring="$2"
    if test "${string#*$substring}" != "$string"
    then
        return 0    # $substring is in $string
    else
        return 1    # $substring is not in $string
    fi
}

function notify-send() {
    #Detect the name of the display in use
    local display=":$(ls /tmp/.X11-unix/* | sed 's#/tmp/.X11-unix/X##' | head -n 1)"

    #Detect the user using such display
    local user=$(who | grep '('$display')' | awk '{print $1}')

    #Detect the id of the user
    local uid=$(id -u $user)

    sudo -u $user DISPLAY=$display DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$uid/bus notify-send "$@"
}

case "$STATUS" in
    vpn-up)
      # If the VPN connection name does not contain 'ev_' return from the script
      contains "${CONNECTION_ID}" "ev_" || exit 0
      # Get Link number from systemd-resolve for the new interface
      LINKNUM=`systemd-resolve --status | grep $VPN_IP_IFACE | sed 's/.*Link \([0-9]\+\).*/\1/'`
      case $LINKNUM in
      ''|*[!0-9]*)
        # Bad number, notify the user... 
        notify-send "VPN DNS Warning" "Could not find Link number systemd-resolve for ${VPN_IP_IFACE}"
      ;; 
      *)
        # We have a link number, lets set DNS Domain: ~.   
        busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDomains 'ia(sb)' $LINKNUM 1 . true
        # Check if setting was successfully set
        LOBJECT=`busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager GetLink 'i' $LINKNUM | sed 's/o.\+"\(.*\)"/\1/'`
        LDOMAINS=`busctl get-property org.freedesktop.resolve1 $LOBJECT org.freedesktop.resolve1.Link Domains`
        if (echo $LDOMAINS | grep "\".\" true"); then
          notify-send "VPN DNS set successfully" "All DNS requests are routed through ${VPN_IP_IFACE}"
        else
          notify-send -i dialog-error "VPN DNS Error" "DNS Domains . could not be set for ${VPN_IP_IFACE}" 
        fi
      ;;
      esac
    ;;
    *)
    ;;
esac

[godfuture]

First of all, like many other power users I ask myself why systemd-resolve is such a deaf piece of software that I need scripts to force my DNS server.

Before systemd I used nslookup which named the server that resolved the host. Now I have to activate debugging to see the dns servers used. This is all way to complicated...I need to be able to easily verify if my VPN is leaking dns requests.

Last , but not least, the network-manager integration seems not ready, because the scripts needed to update systemd-resolve cannot be executed. I guess the network-manager does not behave systemd-resolve friendly and therefore options are ignored.

Nevertheless, I came close to what I expected using the steps @arno01 mentioned. The script combined with the dns domain in openvpn config forces my dns requests to pass my VPN. I checked it with (pretending systemd-resolve debug is turned on):
journalctl -u systemd-resolved | grep "Using DNS server"

Till then I start my VPN through a shortcut...asking for admin rights.

[Forage]

With some drawbacks in more complicated configurations, I can confirm that the script of @jahnf works if you just stick to a single VPN.

I tried making adaptions to deal with multiple VPN connections by letting it kick in on vpn-down as well, in addition to a second contains string check. While this works when bringing multiple connections up, as soon as you disable one of the two connections again you will loose the DNS of the remaining connection.

Anyway, better then nothing. Many thanks.

[arno01]

@Forage I was able to setup two OpenVPN connections without exposing the ISP's DNS server.

To avoid ISP's DNS server exposure with systemd-resolve, make sure you are using DOMAIN-ROUTE ., see Upd2 in my comment above #6076 (comment)

When running two or more OpenVPN's, you need to ensure you are not using redirect-gateway def1 in more than one configuration, otherwise you will get a route add ERROR: Linux route add command failed: external program exited with error status: 2 and then closing that connection will silently remove the default gateway of the other OpenVPN connection (you will notice that when you stop it, it will give a route delete error ERROR: Linux route delete command failed: external program exited with error status: 2)

But then, having redirect-gateway def1 in the only one OpenVPN connection will make the whole thing prone to a failure, i.e. in case when the connection gets lost, all your traffic will go through your ISP again, except the routes (if were) set for the other OpenVPN connection.
There might be more options for the gateway, maybe some more dynamic setting? I haven't been checking that further.

Of course, if you either need to access only internal VPN resources or pass certain IP ranges through the VPN, you can use route in their configuration, e.g.:

# icanhazip.com
route 69.162.69.0 255.255.255.0

# dnsleaktest.com
route 23.239.16.0 255.255.255.0

[Queuecumber]

I mean I can do all this but from a user perspective, in my VPN settings in network manager, under the IPv4 section, I explicitly set the DNS server to use my VPNs dns server, and yet systemd-resolved still is refusing to do it. That sounds exactly like a systemd-resolved bug to me.

[arcana261]

Dear people!

I have exactly the same problem here..

Let's be honest guys! DNS servers are NOT EQUAL. There is a difference between THEORY and REALITY.

I live in an unfortunate part of the world where DNS requests are hijacked.. So DNS probe from my VPN connection is different than that of my ISP provided DNS server. Not even this.. since ISP's in my country hijack DNS requests, they almost NEVER respect TTL's. Leading to unexpected results which can last up to days until they are fixed.

Here is output of systemd-resolve --status:

arcana@arcana-ubuntu:~$ systemd-resolve --status
Global
         DNS Servers: 4.2.2.4
                      8.8.8.8
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 26 (vpn0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
          DNS Domain: example.com

Link 25 (veth73cfd00)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no

As you can see, my VPN connection has assigned 8.8.8.8 as DNS server, also my global configuration states that as 4.2.2.4 and 8.8.8.8. Which I have set them specifically in order to get around DNS hijacking while having my VPN connection activated.

I have done this in three steps... First.. my openconnect server has DNS configuration as follows located in ocserv.conf:

dns = 8.8.8.8

Second I have modified configuration for Network Manager... Designating DNS servers manually:

[connection]
id=ArcanumNet
uuid=0aa04538-b8e2-4b3c-9991-fef368b34d50
type=wifi
permissions=
timestamp=1536188751

[wifi]
mac-address=54:27:1E:0B:6C:BD
mac-address-blacklist=
mode=infrastructure
seen-bssids=6C:FD:B9:80:15:65;
ssid=ArcanumNet

[wifi-security]
key-mgmt=wpa-psk
psk=UthinK"YoucAN`1PASS

[ipv4]
dns=4.2.2.4;8.8.8.8;
dns-search=
method=auto

And third step, I have instructed resolved to use some designated global name servers:

[Resolve]
DNS=4.2.2.4 8.8.8.8
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

But even still, resolved would pick my local router's DNS server simply because it responds a little faster! And I have no option at all to disable this.. No option to exclude it.. Here is journal log when I try to resolve pornhub.com. In this log, you can see that 192.168.2.1 is picked, which is hijacked by ISP! And I am doomed. Leaving me crying tears of blood every time I edit resolv.conf manually by hand.

I just am a fan of systemd... But this optimization behavior of resolved comes with no configuration at all.. it comes with no way to exclude a DNS server, increase timeouts, disable this behavior or whatever. And the more I think about it.. The more I come to realize that I should either replace and disable resolved OR to nuke my ubuntu system, find some lonely linux distro which does not use systemd and just stay there...

OR maybe we can come to some terms about how we can configure this, I would gladly submit a PR in time.

Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN AAAA
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 47668 for <pornhub.com IN AAAA> scope dns on */*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 47668.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 4.2.2.4 for transaction 47668.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 47668.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN A
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 23735 for <pornhub.com IN A> scope dns on */*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 23735.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 4.2.2.4 for transaction 23735.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 23735.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN A
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 30675 for <pornhub.com IN A> scope dns on vpn0/*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 30675.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 8.8.8.8 for transaction 30675.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 30675.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN A
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 53622 for <pornhub.com IN A> scope dns on wlp3s0/*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 53622.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 192.168.2.1 for transaction 53622.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 53622.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.2139 path=n/a interface=n/a mem
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Processing incoming packet on transaction 53622. (rcode=SUCCESS)
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Verified we get a response at feature level UDP+EDNS0 from DNS server 192.168.2.1.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Added positive unauthenticated cache entry for pornhub.com IN A 25s on */INET/192.168.2.1
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 53622 for <pornhub.com IN A> on scope dns on wlp3s0/* now complete with <success> from networ

[godfuture]

Hey guys, is it possible to use systemd, but not resolved? I mean, could I go back using nslookup and other simple tools even systemd does all its other jobs?

[mikken]

godfuture, of course. Don't run systemd-resolved service and edit nsswitch.conf to use resolv.conf instead.

[mdunc]

Just lost a whole morning of work because of systemd-resolved. Most DNS lookups were working fine over my VPNs DNS server (openconnect VPN), but there was one address which systemd-resolved was very aggressive about looking up with my ISPs DNS server (fresh up-to-date Ubuntu 18.04 install). Things worked fine for the first day or so, but now this one address just simply won't resolve correctly. When I see here that people are running huge scripts to fix something like this, my first thought was "Do I even really need to run systemd-resolved if it's going to be this difficult to do the simple job of using my VPNs DNS server reliably? What do I gain here that I didn't have before with just NetworkManager?"

@godfuture, I followed the accepted solution here - https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu. DNS now behaves much more sanely and reliably and nslookup has become useful again too.

[zerkms]

to all available servers

It is configured with only those 2 available servers. Only those 2 servers must be available to it. I explicitly specified just 2 servers, it must respect the configuration. If for some reason it also wants to use other servers - it at least must show them in the systemd-resolve --status, otherwise it makes no sense really.

On global you can't set a routing domain

I'm not sure what that exactly means, all my DNS traffic must flow to those 2 configured servers only.

I can suggest creating some dummy interface tun0, attach your DNS servers to it and set routing domain

I don't need a routing domain, those are global DNS servers because I want all DNS traffic to flow through it exclusively.

[mikken]

I see, check again that resolution you see with tcpdump is done by systemd-resolved, not system resolver that takes /etc/resolv.conf as its configuration.
If yes, check systemd-resolved debug logs that it contacts those servers. If so, file another bug maybe.

[zerkms]

The very reason I'm here is because long ago I have experienced exactly the same symptoms like other people in this post.

Given this issue was closed and people deny there is a bug - I don't see how creating another one would change anything 🤷 .

I see, check again that resolution you see with tcpdump is done by systemd-resolved, not system resolver that takes /etc/resolv.conf as its configuration.

# cat /etc/resolv.conf 
nameserver 127.0.0.53
options edns0

If yes, check systemd-resolved debug logs that it contacts those servers

It contacts 192.168.1.111 which is assigned by DHCP, but it's not present anywhere in systemd-resolved output.

[mcatanzaro]

I insist that piling onto this issue report with additional questions or support requests isn't going to be useful.

So, no, I don't see how it's not a systemd-resolve bug.

systemd-resolved sends your DNS exactly where it is told to do so. It's not systemd-resolved's fault that you (or other software on your computer) have told it to do something that's not what you want. In your case, my guess -- without seeing the output of resolvectl, we can only guess -- is that DHCP has supplied 192.168.1.111 as a DNS server for one of your network interfaces, which will almost always happen. If you really don't have any global routing domains set, then systemd-resolved will send requests to all configured DNS servers, so it's going to be used. If you think you can avoid using the DHCP-provided server simply by changing your global DNS servers, you are mistaken: that's just not how systemd-resolved works. One way to get the behavior you want would be to set your VPN's DNS servers on the appropriate network interface, and also set a global routing domain for that same interface. If you used NetworkManager to configure your VPN, it would take care of all this for you.

It contacts 192.168.1.111 which is assigned by DHCP, but it's not present anywhere in systemd-resolved output.

We need to see the output of the resolvectl command to know that. My guess is that it will be very prominently listed in the output. If it's not configured, and systemd-resolved is using it, that would be an amazingly strange bug, so strange that I highly doubt that's happening.

So I see zero evidence of a bug -- it's much more likely just a misconfiguration -- although for us to be sure, you would need to post the output of resolvectl so that we can see your actual configuration. (I see you didn't take the time to install that when mikken asked for it.) But please don't post that here, because this is the wrong place. There are probably 20 different people with who knows how many different problems posting in this thread, and that is totally unmanageable. Even in the unlikely-but-possible event that you did manage to find a systemd-resolved bug, it's not going to be fixed by posting in a closed issue that has absolutely nothing to do with your problem. The original problem here is likely a three-year-old NetworkManager bug -- long since fixed -- and if not, it's something particular to the original reporter's configuration. I again assure everyone reading this that there is almost no chance that you are experiencing the same issue. Having a similar symptom doesn't mean it's the same issue. I'm surprised that I need to explain this more than once. Instead, create a support request somewhere (e.g. the systemd mailing list), or create a completely new issue report if you're confident that you have really found a problem.

[mcatanzaro]

If for some reason it also wants to use other servers - it at least must show them in the systemd-resolve --status, otherwise it makes no sense really.

OK, one correction: I didn't know about systemd-resolve --status, but I see it is equivalent to resolvectl. So yeah, if systemd-resolved is using your DHCP-provided DNS server when it's not present in the output of systemd-resolve --status, that is indeed a (very strange) bug. But it's not this bug. Please report a new bug for this.

[poettering]

systemd-resolve is the old obsolete name of resolvectl. Ignore its existance, unless you run really old distros.

[poettering]

Please do not report any bugs here against distros that do not provide resolvectl, but do provide systemd-resolve, as that means your distro is really old, and we only track issues related to current upstream systemd versions here, see submission form.

[eugeneseppel]

Lennart, not everyone uses the latest distros, most "ordinary" end-users and, of course, enterprise users prefer stable LTS distros. We use computers for work and entertainment, not for hacking.
In Ubuntu 18.04.5 LTS (installed by our IT department) there is no resolvectl.

$ systemd-resolve --status
shows something like this:
Global
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 142 (tun0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 192.168.160.1
DNS Domain: amazonaws.com
~.
Link 2 (eth0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 192.168.1.1
DNS Domain: ~.

so "~." exists for both eth0 and tun0, but only enterprise 192.168.160.1 DNS can resolve the internal resources.
If I do
ping wiki.hq.<companyname>.com
or try to open it in the web browser it is not resolved. So as a workaround I run the script that every 120 seconds does
host wiki.hq.<companyname>.com
it helps ping/firefox/ssh to work correctly
So "host" uses DNS from VPN, while ping/firefox -- use ISP's DNS if the item is not cached. That's why I'm thinking it is most probably a systemd-resolve issue.

[poettering]

Please understand that this is a bug tracker for upstream issues. For older systemd versions please contact your downstream distro. It's pointless to post issues here about really old versions here. It frustrates both you and me, and wastes both our time. Thank you for understanding.

[mcatanzaro]

So "host" uses DNS from VPN, while ping/firefox -- use ISP's DNS if the item is not cached. That's why I'm thinking it is most probably a systemd-resolve issue.

Well you've definitely messed up your systemd-resolved configuration such that internal names may be resolved by either DNS server: you probably want tun0's DNS to resolve requests for wiki.hq.<companyname>.com, but there is no DNS domain set for it, and you have a ~. on eth0, allowing those names to be resolved by either server.

It might also indicate a messed up your /etc/resolv.conf such that systemd-resolved is not the only present server. I am just guessing there, but that might explain the difference in behavior between ping (which probably uses nsswitch) and host (which looks at /etc/resolv.conf directly).

Notice that your problem is totally unrelated to the last user who commented here, and also unrelated to the original bug report. There are simply too many inventive ways to misconfigure your DNS for us to handle every possible misconfiguration in this one issue report. I'm going to suggest it's probably time to lock this issue.

[sliddjur]

@zerkms I suppose you use split tunnel VPN. I get two DNS servers from our company in a split-tunnel DNS setup aswell. This is what I did.

Does that help you?
nmcli connection modify <vpn-connection-name> ipv4.dns-search '~.'

When I am connected to our company VPN I am now resolving everything through VPN. I have doublechecked with Wireshark to see if I am leaking any DNS, but I couldnt find anything yet.

➜  ~ resolvectl status tun0
Link 41 (tun0)
      Current Scopes: DNS         
DefaultRoute setting: yes         
       LLMNR setting: yes         
MulticastDNS setting: no          
  DNSOverTLS setting: no          
      DNSSEC setting: no          
    DNSSEC supported: no          
  Current DNS Server: 172.18.66.13
         DNS Servers: 172.18.66.13
                      172.18.66.22
          DNS Domain: ~.          

➜  ~ resolvectl status wlan0
Link 3 (wlan0)
      Current Scopes: none
DefaultRoute setting: no  
       LLMNR setting: yes 
MulticastDNS setting: no  
  DNSOverTLS setting: no  
      DNSSEC setting: no  
    DNSSEC supported: no  

Disconnect from VPN, and the default resolver is now on wlan0

➜  ~ nmcli connection down <vpn-connection-name>
Connection '<vpn-connection-name>' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/19)

➜  ~ resolvectl status wlan0                 
Link 3 (wlan0)
      Current Scopes: DNS          
DefaultRoute setting: yes          
       LLMNR setting: yes          
MulticastDNS setting: no           
  DNSOverTLS setting: no           
      DNSSEC setting: no           
    DNSSEC supported: no           
  Current DNS Server: 8.8.8.8
         DNS Servers: 8.8.8.8
          DNS Domain: ~.           
                      local.lan    
➜  ~ resolvectl status tun0                  
Failed to resolve interface "tun0", ignoring: No such device

[zerkms]

@sliddjur I use ipsec, there is no dedicated interface in my implementation: all traffic flows through one and the only physical interface.

[yuqian90]

This issue is closed, but when I tried to use OpenVPN 2.4.7 on Ubuntu 20.04, this issue still causes DNS leaks. After some troubleshooting, I realized that I had to do all of the following:

1. Install openvpn-systemd-resolved

sudo apt-get install openvpn-systemd-resolved

2. Add these lines to the .ovpn config. This sets the "DNS Domain" of the VPN interface tun0 to ~.

script-security 2
up /etc/openvpn/update-systemd-resolved
up-restart
down /etc/openvpn/update-systemd-resolved
down-pre
dhcp-option DOMAIN-ROUTE .

3. After starting openvpn, I still had to do one more step, which is to remove the existing ~. from my default network interface (eno1 in my case). Otherwise DNS requests go to both tun0 and eno1 and still causes DNS leaks. This is the command:

sudo systemd-resolve -i eno1 --set-domain lan

Only after all three steps, openvpn finally stops leaking DNS. In systemd-resolve --status, this is what I now see. Without Step 3 above, the DNS Domain of eno1 is set to ~. and lan and thus still leaks DNS.

$ systemd-resolve --status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNS Domain: lan
...
Link 8 (tun0)
      Current Scopes: DNS
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 1.1.1.1
         DNS Servers: 1.1.1.1
                      8.8.8.8
          DNS Domain: ~.

...

Link 2 (eno1)
      Current Scopes: DNS
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 192.168.X.X
         DNS Servers: 192.168.X.X
          DNS Domain: lan

This is such a pain to do something simple. I want to point out that none of these were needed before systemd-resolved was introduced. Is there any plans to fix this?

[mcatanzaro]

This is such a pain to do something simple. I want to point out that none of these were needed before systemd-resolved was introduced. Is there any plans to fix this?

No, it would need to be fixed by the openvpn developers, not by systemd-resolved. systemd-resolved only sends your DNS exactly where you tell it to, so there is nothing to fix. If you don't tell it what you want, it cannot do what you want.

Your post is good documentation of how to manually tell systemd-resolved where your DNS is intended to go if you want to manually configure openvpn. Alternatively, if you are a desktop user, you could use NetworkManager-openvpn, which would take care of all of this for you automatically.

[yuqian90]

This is such a pain to do something simple. I want to point out that none of these were needed before systemd-resolved was introduced. Is there any plans to fix this?

No, it would need to be fixed by the openvpn developers, not by systemd-resolved. systemd-resolved only sends your DNS exactly where you tell it to, so there is nothing to fix. If you don't tell it what you want, it cannot do what you want.

Your post is good documentation of how to manually tell systemd-resolved where your DNS is intended to go if you want to manually configure openvpn. Alternatively, if you are a desktop user, you could use NetworkManager-openvpn, which would take care of all of this for you automatically.

Thanks. Let me try to find out if OpenVPN is fixing this.

[yuqian90]

This is such a pain to do something simple. I want to point out that none of these were needed before systemd-resolved was introduced. Is there any plans to fix this?

No, it would need to be fixed by the openvpn developers, not by systemd-resolved. systemd-resolved only sends your DNS exactly where you tell it to, so there is nothing to fix. If you don't tell it what you want, it cannot do what you want.

Your post is good documentation of how to manually tell systemd-resolved where your DNS is intended to go if you want to manually configure openvpn. Alternatively, if you are a desktop user, you could use NetworkManager-openvpn, which would take care of all of this for you automatically.

Hi @mcatanzaro one thing I want to point out is that, OpenVPN is not the only VPN client experiencing this issue. Others such as FortiClient have the same problem. Is it right to say that multiple VPN clients are broken and the issue is caused by those VPN clients themselves and not by systemd-resolved?

[poettering]

I put together some docs in #17678 that explain in detail what VPN should do to get the right config into systemd-resolved. PTAL.

[mcatanzaro]

Hi @mcatanzaro one thing I want to point out is that, OpenVPN is not the only VPN client experiencing this issue. Others such as FortiClient have the same problem. Is it right to say that multiple VPN clients are broken and the issue is caused by those VPN clients themselves and not by systemd-resolved?

I'm surprised this is still a problem in 2020, since systemd-resolved has been default in Ubuntu for four years now, but yes, every VPN client that does not know how to configure systemd-resolved is going to be broken. Thanks for documenting how to do this, Lennart!

[AndrewVos]

Wow, so I've just found this and have been wondering why I've had so many problems with ivpn. Time to go back to nordvpn.

[lestofante]

I put together some docs in #17678 that explain in detail what VPN should do to get the right config into systemd-resolved. PTAL.

Something is not working properly AFAIK,
I follow the guide and issued the privacy suggested command, correct DNS ip is used but on the interface "end0" instead of the "tun0"; this happen only if the DNS server has the same ip as the VPN server, otherwise it seems to work correctly

resolvectl domain tun0 '~.'
resolvectl default-route tun0 true
resolvectl dns tun0 192.216.219.147

systemd-resolve --status

Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: uplink
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google

Link 2 (end0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 127.0.0.1
       DNS Servers: 127.0.0.1

Link 19 (tun0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 195.216.219.147
       DNS Servers: 195.216.219.147
        DNS Domain: ~.

but making a dns request (in this case a simple ping www.google.com) causes:

[nftables][end0] DNS outbound Denied: IN= OUT=end0 SRC=192.168.0.3 DST=195.216.219.147 LEN=65 TOS=0x00 PREC=0x00 TTL=64 ID=50728 DF PROTO=UDP SPT=40224 DPT=53 LEN=45

• the request is on end0, while it should be on tun0.
• ping hangs as there is no answer from the DNS, so i dont think any parallel request is sent to tun0
• setting the dns server to 8.8.8.8 or anything that is not the VPN's ip, seems to fix the issue

As we talk about privacy setup, and as DNS request are not encrypted, letting them go to the VPN server without passing trough the tun0 is pretty much as bad as requesting to the ISP, as they are easy to intercept from the ISP.
Also may be problematic in buisness setup, where the dns server may be listening only to request from the LAN rather than be accessible from the internet

also i find very strange this different behavior based on the destination ip, am i triggering some edge case?

[tobiasbrunner]

this happen only if the DNS server has the same ip as the VPN server

To avoid routing the VPN traffic itself via TUN device, some route-based VPNs install a bypass route via physical interface for the VPN server's IP address. You might want to assign a second (maybe private) IP address to the VPN server that you use as DNS server address.

[lestofante]

To avoid routing the VPN traffic itself via TUN device, some route-based VPNs install a bypass route via physical interface for the VPN server's IP address. You might want to assign a second (maybe private) IP address to the VPN server that you use as DNS server address.

ok, this make a lot of sense and i guess a simple routing table cannot handle my case as it does not know the protocol, or at least the destination port.
As i dont have access to server configuration, i guess my only solution is to make sure to use a different DNS server or do some iptables magic