New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After connected to VPN, systemd-resolve still use ISP's DNS server( which was polluted because of regulation ) #6076

Open
joshuafc opened this Issue Jun 2, 2017 · 38 comments

Comments

@joshuafc

joshuafc commented Jun 2, 2017

Submission type

  • Request for enhancement (RFE)

systemd version the issue has been seen with

systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

Used distribution

Ubuntu 17.04 Linux version 4.10.0-21-generic (buildd@lgw01-12) (gcc version 6.3.0 20170406 (Ubuntu 6.3.0-12ubuntu2) ) #23-Ubuntu SMP Fri Apr 28 16:14:22 UTC 2017

Because of the Internet regulation, ISP's DNS server can not resolve some domian correctly( for example www.google.com www.youtube.com ). Under normal conditions, we use vpn and set the vpn as default internet gateway when needed to access above website.
When I start to use Ubuntu 17.04, It is found that when connected to vpn, all internet package transfer through vpn except domain name resolve. Infact, VPN server had tell correct DNS server IP address through DHCP. But systemd-resolve still use ISP's polluted resolve.
At /etc/resolv.conf, I found that DNS resolve had be taken over by systemd-resolve. There can't find any easy way to tell systemd-resolve to use a specified DNS server, or disable any DNS server according " systemd-resolve --help ".
Willing to accept guidance to correct my use of systemd-resolve,
If you can, Please add some necessary feature to systemd-resolve to solve such problem,

In case of bug report: Expected behaviour you didn't see

In case of bug report: Unexpected behaviour you saw

In case of bug report: Steps to reproduce the problem

@poettering

This comment has been minimized.

Member

poettering commented Jun 13, 2017

Hmm, not sure I follow. How exactly did you configure your DNS? What does "systemd-resolve --status" report about the DNS servers? Is /etc/resolv.conf a file or a symlink?

@joshuafc

This comment has been minimized.

joshuafc commented Jun 20, 2017

@poettering thanks for your reply.

here is the information

zhoub@ubuntu:\~$ ll /etc/resolv.conf 
rwxrwxrwx 1 root root 29 Jun  6 09:35 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
zhoub@ubuntu:~$ cat /etc/resolv.conf 
\# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
\#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
\# 127.0.0.53 is the systemd-resolved stub resolver.
\# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
zhoub@ubuntu:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 6 (ppp0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 2 (ens33)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no
         DNS Servers: 192.168.83.2
          DNS Domain: localdomain
zhoub@ubuntu:~$ ping www.google.com
PING www.google.com (93.46.8.89) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
106 packets transmitted, 0 received, 100% packet loss, time 107507ms

ping failed, 93.46.8.89 is incorrect , ISP's DNS server resolve it to 93.46.8.89 because of Internet regulation. VPN's DNS Server resolve it to 172.217.26.36, which can ping successfully.

screenshot2
screenshot
screenshot1

@aigarius

This comment has been minimized.

aigarius commented Sep 21, 2017

Can confirm the same effect with a OpenVPN connection.

Link 8 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 172.20.0.50
                      172.20.0.14

Link 3 (wlan0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.192.1
          DNS Domain: net

Running "systemd-resolve internal.name.local" fails to find it (it only tries the 192.* DNS server), running "systemd-resolve -i tun0 internal.name.local" resolves the name correctly (via 172.* DNS server), but no applications actually supply that interface, naturally.

Systemd version 232
Ubuntu 17.04

Currently I have to workaround this by overwriting the /etc/resolv.conf file each time I connect to the VPN, but even that now stops working as some apps use the systemd-resolv directly if it is available.

Alternative solution that worked for me is using a patched version of NetworkManager from https://bugs.launchpad.net/network-manager/+bug/1624317 that forces using the VPN DNS server by adding a "DNS Domain: ~." declaration to the tun0 section of systemd-resolve

@mikken

This comment has been minimized.

mikken commented Oct 9, 2017

You (joshuafc and aigarius) need to do 2 things:

  1. Make sure that DNS servers obtained from OpenVPN are used by systemd-resolved. You can do it with this little script: https://github.com/jonathanio/update-systemd-resolved
  2. While OpenVPN is connected, block (reject with iptables or nftables) all requests that systemd-resolved makes to local DNS server
@mikken

This comment has been minimized.

mikken commented Oct 9, 2017

aigarius, for you the domain .local should be configured as "routing" domain on this link (tun0).

@PureRumble

This comment has been minimized.

PureRumble commented Oct 29, 2017

@mikken, that script you're telling us to use is 400 lines long. What does it do? Any bugs? Is it safe or does it expose our systems to vulnerabilities? Will it break something else? Why do we need a 400 lines long script (and then to do even more stuff) to achieve something as essential as getting DNS lookups to go through the VPN connection?

Nearly every time someone connects to a VPN service the user's expected system behaviour is that the VPN service's DNS servers should be used and all traffic should go through the VPN connection; this is especially true for novice users. This should therefore be the default behaviour.

But not only is this not the default behavior, there also isn't some easy way to make systemd-resolved to work as desired (e.g. you have to use 400 lines long scripts, block packets to your ISP's DNS server at the IP-level, etc).

systemd-resolved is a "system service that provides network name resolution". It is beyond essential that a system that "provides network name resolution" should be able to with little effort to be made to route DNS requests through a VPN connection. Also, it's default behavior out-of-the-box should comply with the most common use cases (e.g. user connects to a VPN service ==> all data goes through VPN connection).

@flo-wer

This comment has been minimized.

flo-wer commented Oct 29, 2017

See #7182

@snabb

This comment has been minimized.

snabb commented Oct 29, 2017

My current workaround is to do the following tricks every time I start my workplace VPN:

  1. Find out the link number (a.k.a. interface index) of the ISP interface: systemd-resolve --status or ip l.
  2. Connect the VPN.
  3. Remove DNS settings from the ISP interface (using the link number from step 1) by sending a D-Bus command to systemd-resolved. This is an example using link number 2: sudo busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDNS 'ia(iay)' 2 0
  4. Inspect with systemd-resolve --status to ensure that only the correct DNS servers are there.

Apparently there isn't any simple cli tool for managing systemd-resolved's settings. That busctl command isn't very user-friendly. :)

@poettering

This comment has been minimized.

Member

poettering commented Dec 8, 2017

So, resolved is actually doing the right thing here, given the configuration it was given. @joshuafc, resolved only knows the DNS server 192.168.83.2 in your setup, but you want it to know 208.67.222.222 and 8.8.8.8, right? If so, this information needs to be made available to resolved, and the ppp scripts (given that you are using ppp/pptp) need to push it into resolved, which i think ubuntu updated the "resolvconf" package to do. But that's really outside of the scope of upstream resolved. @nox, can you help? is there some ppp/pptp hookup missing for resolved in ubuntu?

@poettering

This comment has been minimized.

Member

poettering commented Dec 8, 2017

@snabb #7576 adds the requested "simple cli" functionality to systemd-resolve.

@jetsaredim

This comment has been minimized.

jetsaredim commented Jan 5, 2018

@poettering I'm confused by your most recent comment. If you review the contents of the most recent comment by @aigarius it shows that resolved does in fact have the DNS server addresses. He's just using google.com as an example to show the geo-located difference between his native connection and the vpn connection.

I came to this bug because I have a similar situation and basically the exact same symptoms as described in @aigarius' most recent comment with an openconnect vpn configuration. It's clear from the output that systemd-resolved has the DNS server info for the DNS servers inside the VPN, but it's not updating the DNS cache to bring in the new DNS servers and thus having trouble resolving things inside the private VPN network that should resolve easily. And in fact do resolve easily if you point relevant commands at the correct interface/DNS server.

A simple restart of systemd-resolved upon vpn connect/disconnect seems to fix the problem, but alas it really shouldn't be needed. Systemd-resolved should just be flushing the cache when a new DNS server is added.

@arno01

This comment has been minimized.

arno01 commented May 8, 2018

A simple restart of systemd-resolve upon vpn connect/disconnect does not really help.

  1. connect to any VPN server and visit the https://www.dnsleaktest.com or https://ipleak.net
  2. see your ISP DNS are being leaked. Not good.

The only reasonable workaround so far is the one provided by @snabb #6076 (comment) Edit: and the DOMAIN-ROUTE . along with the update-systemd-resolved script so it would instruct systemd-resolve to use the domain name routing (see below for more on this concept).

We need some way to make sure that the DNS servers provided by the VPN server are the only ones that should be used as long as the VPN connection stays active.

Upd/Edited:
While the script https://github.com/jonathanio/update-systemd-resolved can be updated used to ensure that the only VPN provided DNS servers are used by leveraging DNS Domain: ~. (see below), I am still thinking that systemd-resolve should handle the DNS priority management, specifically for the VPN DNS case as it really should have the VPN DNS set as a default, otherwise people are likely to face serious consequences (in some extremely regulated countries) without the knowledge about the DNS leaks and with a great trust to a VPN... As far as I understand, the DNS Domain: ~. is the right way of managing the DNS priority with the systemd-resolve, except that it is never set in the openVPN configuration profiles, nor the update-systemd-resolved script is in place by default. Which is still causing a big frustration to the users.
And yet, besides the original openvpn client, there is network-manager-openvpn-gnome which is not using the update-systemd-resolved script. The systemd-resolve could probably help in this kind of situation, perhaps, as long as if it was instructed to use the VPN's DNS only or would rather be just switching the DNS to the one which was activated the last, since the former would imply teaching gnome's network-manager (or any other kind of openVPN client) how to talk to systemd-resolve via the D-Bus. Edit: hum, turns out this is something that is already WIP (see Upd4 below).

Upd2:
It appears DOMAIN-ROUTE . does the trick when the script is used.

jonathanio/update-systemd-resolved@b417bac

script-security 2
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre

# prevent DNS leakage
dhcp-option DOMAIN-ROUTE .

Upd3:
To those who are interested, the ad hoc fix would be the following (pretty much same thing that the script does):

Do the following while connected to a VPN server.

$ systemd-resolve --status | grep -A8 tun0 
Link 24 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4

$ sudo busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDomains 'ia(sb)' 24 1 . true

$ systemd-resolve --status | grep -A8 tun0 
Link 24 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4
          DNS Domain: ~.

According to https://www.freedesktop.org/software/systemd/man/systemd.network.html

Domain name routing has no equivalent in the traditional glibc API,
which has no concept of domain name servers limited to a specific link.

Upd4:
I found people are working from both directions, some are patching the systemd-resolved to support systemd-resolved when NetworkManager's dns-priority is used (see man nm-settings):

https://bugzilla.gnome.org/show_bug.cgi?id=783569#c26
https://bugs.launchpad.net/network-manager/+bug/1624317/comments/81

And some are working on the NetworkManager to support the routing-only domains concept which was introduced by systemd-resolved - https://bugzilla.gnome.org/show_bug.cgi?id=746422

@jahnf

This comment has been minimized.

jahnf commented May 14, 2018

I had the same problem with DNS Leaks.
System: Ubuntu 18.04 with systemd-resolve and imported ovpn settings from my VPN Provider into NetworkManager.

I created a script that I wanted to share which solved my problems, in the background via Network Manager - no additional manual calls needed anymore.
I put it together with the information in this thread and other forums. It has to go into /etc/NetworkManager/dispatcher.d/ (only executable and writeable by root).

This is executed by Network Manager every time there is a connection change - In my case I check if the vpn connection id contains a certain substring, if yes I set the DNS Domain for the new VPN connection - and no DNS leaks any more.

#!/bin/bash

IF=$1
STATUS=$2

# contains(string, substring)
#
# Returns 0 if the specified string contains the specified substring,
# otherwise returns 1.
contains() {
    string="$1"
    substring="$2"
    if test "${string#*$substring}" != "$string"
    then
        return 0    # $substring is in $string
    else
        return 1    # $substring is not in $string
    fi
}

function notify-send() {
    #Detect the name of the display in use
    local display=":$(ls /tmp/.X11-unix/* | sed 's#/tmp/.X11-unix/X##' | head -n 1)"

    #Detect the user using such display
    local user=$(who | grep '('$display')' | awk '{print $1}')

    #Detect the id of the user
    local uid=$(id -u $user)

    sudo -u $user DISPLAY=$display DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$uid/bus notify-send "$@"
}

case "$STATUS" in
    vpn-up)
      # If the VPN connection name does not contain 'ev_' return from the script
      contains "${CONNECTION_ID}" "ev_" || exit 0
      # Get Link number from systemd-resolve for the new interface
      LINKNUM=`systemd-resolve --status | grep $VPN_IP_IFACE | sed 's/.*Link \([0-9]\+\).*/\1/'`
      case $LINKNUM in
      ''|*[!0-9]*)
        # Bad number, notify the user... 
        notify-send "VPN DNS Warning" "Could not find Link number systemd-resolve for ${VPN_IP_IFACE}"
      ;; 
      *)
        # We have a link number, lets set DNS Domain: ~.   
        busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDomains 'ia(sb)' $LINKNUM 1 . true
        # Check if setting was successfully set
        LOBJECT=`busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager GetLink 'i' $LINKNUM | sed 's/o.\+"\(.*\)"/\1/'`
        LDOMAINS=`busctl get-property org.freedesktop.resolve1 $LOBJECT org.freedesktop.resolve1.Link Domains`
        if (echo $LDOMAINS | grep "\".\" true"); then
          notify-send "VPN DNS set successfully" "All DNS requests are routed through ${VPN_IP_IFACE}"
        else
          notify-send -i dialog-error "VPN DNS Error" "DNS Domains . could not be set for ${VPN_IP_IFACE}" 
        fi
      ;;
      esac
    ;;
    *)
    ;;
esac
@godfuture

This comment has been minimized.

godfuture commented May 16, 2018

First of all, like many other power users I ask myself why systemd-resolve is such a deaf piece of software that I need scripts to force my DNS server.

Before systemd I used nslookup which named the server that resolved the host. Now I have to activate debugging to see the dns servers used. This is all way to complicated...I need to be able to easily verify if my VPN is leaking dns requests.

Last , but not least, the network-manager integration seems not ready, because the scripts needed to update systemd-resolve cannot be executed. I guess the network-manager does not behave systemd-resolve friendly and therefore options are ignored.

Nevertheless, I came close to what I expected using the steps @arno01 mentioned. The script combined with the dns domain in openvpn config forces my dns requests to pass my VPN. I checked it with (pretending systemd-resolve debug is turned on):
journalctl -u systemd-resolved | grep "Using DNS server"

Till then I start my VPN through a shortcut...asking for admin rights.

@Forage

This comment has been minimized.

Forage commented Jun 20, 2018

With some drawbacks in more complicated configurations, I can confirm that the script of @jahnf works if you just stick to a single VPN.

I tried making adaptions to deal with multiple VPN connections by letting it kick in on vpn-down as well, in addition to a second contains string check. While this works when bringing multiple connections up, as soon as you disable one of the two connections again you will loose the DNS of the remaining connection.

Anyway, better then nothing. Many thanks.

@arno01

This comment has been minimized.

arno01 commented Jun 20, 2018

@Forage I was able to setup two OpenVPN connections without exposing the ISP's DNS server.

To avoid ISP's DNS server exposure with systemd-resolve, make sure you are using DOMAIN-ROUTE ., see Upd2 in my comment above #6076 (comment)

When running two or more OpenVPN's, you need to ensure you are not using redirect-gateway def1 in more than one configuration, otherwise you will get a route add ERROR: Linux route add command failed: external program exited with error status: 2 and then closing that connection will silently remove the default gateway of the other OpenVPN connection (you will notice that when you stop it, it will give a route delete error ERROR: Linux route delete command failed: external program exited with error status: 2)

But then, having redirect-gateway def1 in the only one OpenVPN connection will make the whole thing prone to a failure, i.e. in case when the connection gets lost, all your traffic will go through your ISP again, except the routes (if were) set for the other OpenVPN connection.
There might be more options for the gateway, maybe some more dynamic setting? I haven't been checking that further.

Of course, if you either need to access only internal VPN resources or pass certain IP ranges through the VPN, you can use route in their configuration, e.g.:

# icanhazip.com
route 69.162.69.0 255.255.255.0

# dnsleaktest.com
route 23.239.16.0 255.255.255.0
@Queuecumber

This comment has been minimized.

Queuecumber commented Jul 31, 2018

I mean I can do all this but from a user perspective, in my VPN settings in network manager, under the IPv4 section, I explicitly set the DNS server to use my VPNs dns server, and yet systemd-resolved still is refusing to do it. That sounds exactly like a systemd-resolved bug to me.

@arcana261

This comment has been minimized.

arcana261 commented Sep 5, 2018

Dear people!

I have exactly the same problem here..

Let's be honest guys! DNS servers are NOT EQUAL. There is a difference between THEORY and REALITY.

I live in an unfortunate part of the world where DNS requests are hijacked.. So DNS probe from my VPN connection is different than that of my ISP provided DNS server. Not even this.. since ISP's in my country hijack DNS requests, they almost NEVER respect TTL's. Leading to unexpected results which can last up to days until they are fixed.

Here is output of systemd-resolve --status:

arcana@arcana-ubuntu:~$ systemd-resolve --status
Global
         DNS Servers: 4.2.2.4
                      8.8.8.8
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 26 (vpn0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
          DNS Domain: example.com

Link 25 (veth73cfd00)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no

As you can see, my VPN connection has assigned 8.8.8.8 as DNS server, also my global configuration states that as 4.2.2.4 and 8.8.8.8. Which I have set them specifically in order to get around DNS hijacking while having my VPN connection activated.

I have done this in three steps... First.. my openconnect server has DNS configuration as follows located in ocserv.conf:

dns = 8.8.8.8

Second I have modified configuration for Network Manager... Designating DNS servers manually:

[connection]
id=ArcanumNet
uuid=0aa04538-b8e2-4b3c-9991-fef368b34d50
type=wifi
permissions=
timestamp=1536188751

[wifi]
mac-address=54:27:1E:0B:6C:BD
mac-address-blacklist=
mode=infrastructure
seen-bssids=6C:FD:B9:80:15:65;
ssid=ArcanumNet

[wifi-security]
key-mgmt=wpa-psk
psk=UthinK"YoucAN`1PASS

[ipv4]
dns=4.2.2.4;8.8.8.8;
dns-search=
method=auto

And third step, I have instructed resolved to use some designated global name servers:

[Resolve]
DNS=4.2.2.4 8.8.8.8
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

But even still, resolved would pick my local router's DNS server simply because it responds a little faster! And I have no option at all to disable this.. No option to exclude it.. Here is journal log when I try to resolve pornhub.com. In this log, you can see that 192.168.2.1 is picked, which is hijacked by ISP! And I am doomed. Leaving me crying tears of blood every time I edit resolv.conf manually by hand.

I just am a fan of systemd... But this optimization behavior of resolved comes with no configuration at all.. it comes with no way to exclude a DNS server, increase timeouts, disable this behavior or whatever. And the more I think about it.. The more I come to realize that I should either replace and disable resolved OR to nuke my ubuntu system, find some lonely linux distro which does not use systemd and just stay there...

OR maybe we can come to some terms about how we can configure this, I would gladly submit a PR in time.

Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN AAAA
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 47668 for <pornhub.com IN AAAA> scope dns on */*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 47668.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 4.2.2.4 for transaction 47668.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 47668.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN A
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 23735 for <pornhub.com IN A> scope dns on */*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 23735.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 4.2.2.4 for transaction 23735.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 23735.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN A
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 30675 for <pornhub.com IN A> scope dns on vpn0/*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 30675.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 8.8.8.8 for transaction 30675.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 30675.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Cache miss for pornhub.com IN A
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 53622 for <pornhub.com IN A> scope dns on wlp3s0/*.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using feature level UDP+EDNS0 for transaction 53622.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Using DNS server 192.168.2.1 for transaction 53622.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Sending query packet with id 53622.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.2139 path=n/a interface=n/a mem
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Processing incoming packet on transaction 53622. (rcode=SUCCESS)
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Verified we get a response at feature level UDP+EDNS0 from DNS server 192.168.2.1.
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Added positive unauthenticated cache entry for pornhub.com IN A 25s on */INET/192.168.2.1
Sep 06 03:24:47 arcana-ubuntu systemd-resolved[3873]: Transaction 53622 for <pornhub.com IN A> on scope dns on wlp3s0/* now complete with <success> from networ
@godfuture

This comment has been minimized.

godfuture commented Sep 6, 2018

Hey guys, is it possible to use systemd, but not resolved? I mean, could I go back using nslookup and other simple tools even systemd does all its other jobs?

@mikken

This comment has been minimized.

mikken commented Sep 6, 2018

godfuture, of course. Don't run systemd-resolved service and edit nsswitch.conf to use resolv.conf instead.

@mdunc

This comment has been minimized.

mdunc commented Oct 3, 2018

Just lost a whole morning of work because of systemd-resolved. Most DNS lookups were working fine over my VPNs DNS server (openconnect VPN), but there was one address which systemd-resolved was very aggressive about looking up with my ISPs DNS server (fresh up-to-date Ubuntu 18.04 install). Things worked fine for the first day or so, but now this one address just simply won't resolve correctly. When I see here that people are running huge scripts to fix something like this, my first thought was "Do I even really need to run systemd-resolved if it's going to be this difficult to do the simple job of using my VPNs DNS server reliably? What do I gain here that I didn't have before with just NetworkManager?"

@godfuture, I followed the accepted solution here - https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu. DNS now behaves much more sanely and reliably and nslookup has become useful again too.

@lucidyan

This comment has been minimized.

lucidyan commented Oct 12, 2018

@mdunc you don't need to disable systemd for correct VPN work
Try this, it helped for me:
https://askubuntu.com/questions/1032476/ubuntu-18-04-no-dns-resolution-when-connected-to-openvpn

@mdunc

This comment has been minimized.

mdunc commented Oct 12, 2018

@lucidyan, thanks for the link, but that looks specific to openvpn. I have to use openconnect since it's a Cisco AnyConnect VPN (split-tunnel so only corporate traffic goes over the VPN, but I should still be doing DNS lookups through the VPN). I'll set up another Ubuntu 18.04 machine this weekend and do some more testing, but on my main machine, disabling systemd-resolved did the trick.

Just to be clear though, systemd-resolved was seeing the DNS server for my VPN and I could access some of our internal services, but others would still resolve as if it was using my ISP's DNS server. For example, confluence.mycompany.com resolved correctly, but git.mycompany.com did not. And since systemd-resolved neuters the usefulness of nslookup a bit by masking what server is actually being used for the lookup with its own IP (this decision to turn DNS resolution in to a black box of sorts baffles me...I'm not aware of any other OS that does this sort of thing. I admit I'm new to systemd-resolved, but this just feels so incredibly over-engineered), I had difficulty troubleshooting this and ended up ditching systemd-resolved altogether. I do want to dig in to this further though and figure this out. It looks like this is going to be how DNS resolution is going to work now so probably worth the time to properly troubleshoot this.

@mdunc

This comment has been minimized.

mdunc commented Oct 14, 2018

Installed Ubuntu 18.10 beta and brought it up to date. Installed openconnect and openconnect-network-manager-gnome. When connecting to the VPN, the DNS server does get added.

image

Here are the IPv4 settings for the connection
image

DNS resolution continues to go over the ISP DNS though. Only queries to the "vb.loc" domain go over the VPN. The expected behavior is that all DNS queries go over the VPN and this is indeed what happens when not using systemd-resolved and in other operating systems.

This could just be an issue with Ubuntu 18.10 though. This behavior is different than what I had with 18.04 where some names would resolve as expected, but others would continue to resolve through my ISPs DNS server. Now that I think about it more, it's like a domain name could be cached when resolved against the ISP's DNS server, then that cache is returned after connecting to the VPN. Is the DNS cache not cleared when connecting to a VPN?

@mdunc

This comment has been minimized.

mdunc commented Oct 14, 2018

Alright, I've been able to reproduce the problem in Ubuntu 18.10 after manually adding mycompany.com to the search domain list in the VPN connection with nm-connection-editor.

Here is a good DNS lookup happening over the VPN:

$ nslookup confluence.mycompany.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	confluence.mycompany.com
Address: 10.200.xxx.xxx

Here is another DNS lookup made immediately after which returns an IP as if it were looked up on the ISP's DNS server:

$ nslookup jira.mycompany.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	jira.mycompany.com
Address: 23.29.xxx.xxx

Here is the log from systemd-resolved:

Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Got DNS stub UDP query packet for id 7408
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Looking up RR for jira.mycompany.com IN A.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for jira.mycompany.com IN A
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 15460 for <jira.mycompany.com IN A> scope dns on vpn0/*.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP+EDNS0 for transaction 15460.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 10.10.7.10 for transaction 15460.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 15460.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for jira.mycompany.com IN A
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 43142 for <jira.mycompany.com IN A> scope dns on enp0s31f6/*.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP+EDNS0 for transaction 43142.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 192.168.1.1 for transaction 43142.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 43142.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Processing query...
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Processing incoming packet on transaction 43142 (rcode=SUCCESS).
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Added positive unauthenticated cache entry for jira.mycompany.com IN A 6537s on */INET/192.168.1.1
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 43142 for <jira.mycompany.com IN A> on scope dns on enp0s31f6/* now complete with <success> from network (unsigned).
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 15460.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Sending response packet with id 7408 on interface 1/AF_INET.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 43142.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Got DNS stub UDP query packet for id 2394
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Looking up RR for jira.mycompany.com IN AAAA.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for jira.mycompany.com IN AAAA
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 16591 for <jira.mycompany.com IN AAAA> scope dns on vpn0/*.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP+EDNS0 for transaction 16591.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 10.10.7.10 for transaction 16591.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 16591.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for jira.mycompany.com IN AAAA
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 36903 for <jira.mycompany.com IN AAAA> scope dns on enp0s31f6/*.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP+EDNS0 for transaction 36903.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 192.168.1.1 for transaction 36903.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 36903.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Processing query...
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Processing incoming packet on transaction 36903 (rcode=SUCCESS).
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Not caching negative entry without a SOA record: jira.mycompany.com IN AAAA
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 36903 for <jira.mycompany.com IN AAAA> on scope dns on enp0s31f6/* now complete with <success> from network (unsigned).
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 16591.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Sending response packet with id 2394 on interface 1/AF_INET.
Oct 13 20:52:23 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 36903.

So it's actually doing the lookup twice for some reason. Once from the VPN and again from the ISP. This lookup should never be going over the ISP's DNS server when I'm connected to the VPN. It then ignores the result from the VPN and only returns the result from the ISP.

@mdunc

This comment has been minimized.

mdunc commented Oct 14, 2018

In the time I took to write that up, the good DNS lookup I showed above now no longer works and behaves just like the other. I've made no changes. I had to lookup a new domain name to get another good lookup. Here is the log for a good lookup:

Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Got DNS stub UDP query packet for id 22510
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Looking up RR for git.mycompany.com IN A.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Removing cache entry for user-att-76-218-232-0.a1089.d.akamai.net IN A (expired 3s ago)
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for git.mycompany.com IN A
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 60727 for <git.mycompany.com IN A> scope dns on enp0s31f6/*.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP for transaction 60727.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 192.168.1.1 for transaction 60727.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 60727.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for git.mycompany.com IN A
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 48028 for <git.mycompany.com IN A> scope dns on vpn0/*.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP+EDNS0 for transaction 48028.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 10.10.7.10 for transaction 48028.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 48028.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Processing query...
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Processing incoming packet on transaction 48028 (rcode=SUCCESS).
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Verified we get a response at feature level UDP+EDNS0 from DNS server 10.10.7.10.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Added positive unauthenticated cache entry for git.mycompany.com IN A 3600s on */INET/10.10.7.10
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 48028 for <git.mycompany.com IN A> on scope dns on vpn0/* now complete with <success> from network (unsigned).
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 60727.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Sending response packet with id 22510 on interface 1/AF_INET.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 48028.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Got DNS stub UDP query packet for id 49316
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Looking up RR for git.mycompany.com IN AAAA.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for git.mycompany.com IN AAAA
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 30986 for <git.mycompany.com IN AAAA> scope dns on enp0s31f6/*.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP for transaction 30986.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 192.168.1.1 for transaction 30986.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 30986.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Cache miss for git.mycompany.com IN AAAA
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 63320 for <git.mycompany.com IN AAAA> scope dns on vpn0/*.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using feature level UDP+EDNS0 for transaction 63320.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Using DNS server 10.10.7.10 for transaction 63320.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Sending query packet with id 63320.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Processing query...
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Processing incoming packet on transaction 63320 (rcode=SUCCESS).
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Added NODATA cache entry for git.mycompany.com IN AAAA 3600s
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Transaction 63320 for <git.mycompany.com IN AAAA> on scope dns on vpn0/* now complete with <success> from network (unsigned).
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 30986.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Sending response packet with id 49316 on interface 1/AF_INET.
Oct 13 20:59:40 mark-Latitude-E7470 systemd-resolved[7979]: Freeing transaction 63320.

Am I reading that correctly in that it is also using the ISP DNS server? While this isn't a big deal for me, that's pretty undesirable behavior. DNS lookups for a domain should not be happening over another connection when a connection is specifically configured for a that domain. This is leaking DNS requests all over the place.

It also seems like the good lookups only work briefly. After a minute or so, they only return the IP from the ISP DNS. Here is the log for the same "good" domain I queried when I started writing this comment which is now failing to resolve as expected:

Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Got DNS stub UDP query packet for id 40280
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Looking up RR for git.mycompany.com IN A.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Cache miss for git.mycompany.com IN A
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Transaction 64811 for <git.mycompany.com IN A> scope dns on vpn0/*.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Using feature level UDP+EDNS0 for transaction 64811.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Using DNS server 10.10.7.10 for transaction 64811.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Sending query packet with id 64811.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Positive cache hit for git.mycompany.com IN A
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Transaction 49034 for <git.mycompany.com IN A> on scope dns on enp0s31f6/* now complete with <success> from cache (unsigned).
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Freeing transaction 64811.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Freeing transaction 49034.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Sending response packet with id 40280 on interface 1/AF_INET.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Processing query...
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Got DNS stub UDP query packet for id 47542
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Looking up RR for git.mycompany.com IN AAAA.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Cache miss for git.mycompany.com IN AAAA
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Transaction 1813 for <git.mycompany.com IN AAAA> scope dns on vpn0/*.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Using feature level UDP+EDNS0 for transaction 1813.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Using DNS server 10.10.7.10 for transaction 1813.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Sending query packet with id 1813.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Cache miss for git.mycompany.com IN AAAA
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Transaction 15488 for <git.mycompany.com IN AAAA> scope dns on enp0s31f6/*.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Using feature level UDP for transaction 15488.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Using DNS server 192.168.1.1 for transaction 15488.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Sending query packet with id 15488.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Processing query...
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Processing incoming packet on transaction 15488 (rcode=SUCCESS).
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Not caching negative entry without a SOA record: git.mycompany.com IN AAAA
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Transaction 15488 for <git.mycompany.com IN AAAA> on scope dns on enp0s31f6/* now complete with <success> from network (unsigned).
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Freeing transaction 1813.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Sending response packet with id 47542 on interface 1/AF_INET.
Oct 13 21:07:21 mark-Latitude-E7470 systemd-resolved[8482]: Freeing transaction 15488.
@mdunc

This comment has been minimized.

mdunc commented Oct 21, 2018

@poettering can you take another look at this and the logs I pasted above? If this is expected default behavior, I'll be pretty surprised.

@Forage

This comment has been minimized.

Forage commented Oct 21, 2018

After the update from Ubuntu 18.04 to 18.10 I no longer have DNS leaks after connecting to a VPN with openvpn. Previously used work-arounds are no longer needed.

@mdunc

This comment has been minimized.

mdunc commented Oct 21, 2018

I just set up a fresh 18.10 VM, installed network-manager-openconnect-gnome, connected to my company's Cisco AnyConnect VPN with it (only requires setting configuring the gateway in the VPN connection; no certs or anything like that), and ran a DNS test.
image
That's all my ISP, not my company. Resolution of things like git.mycompany.com is not working.

Edit: Adding resolvectl status

mark@mark-VirtualBox:~$ resolvectl status
Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test
 
Link 3 (vpn0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 10.10.7.10
         DNS Servers: 10.10.7.10
                      10.100.0.8
          DNS Domain: mycompany.com
 
Link 2 (enp0s3)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 192.168.1.1
         DNS Servers: 192.168.1.1
          DNS Domain: ~.
                      homenet

Edit 2:
And adding the logs

Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Got DNS stub UDP query packet for id 4071
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Looking up RR for git.mycompany.com IN A.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Switching to DNS server 192.168.1.1 for interface enp0s3.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Switching to DNS server 10.10.7.10 for interface vpn0.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Cache miss for git.mycompany.com IN A
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Transaction 15819 for <git.mycompany.com IN A> scope dns on vpn0/*.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using feature level UDP+EDNS0 for transaction 15819.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using DNS server 10.10.7.10 for transaction 15819.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Sending query packet with id 15819.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Cache miss for git.mycompany.com IN A
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Transaction 43023 for <git.mycompany.com IN A> scope dns on enp0s3/*.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using feature level UDP+EDNS0 for transaction 43023.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using DNS server 192.168.1.1 for transaction 43023.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Sending query packet with id 43023.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Processing query...
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Processing incoming packet on transaction 43023 (rcode=SUCCESS).
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Verified we get a response at feature level UDP+EDNS0 from DNS server 192.168.1.1.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Added positive unauthenticated cache entry for git.mycompany.com IN A 103s on */INET/192.168.1.1
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Transaction 43023 for <git.mycompany.com IN A> on scope dns on enp0s3/* now complete with <success> from network (unsigned).
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Freeing transaction 15819.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Sending response packet with id 4071 on interface 1/AF_INET.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Freeing transaction 43023.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Got DNS stub UDP query packet for id 18085
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Looking up RR for git.mycompany.com IN AAAA.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Cache miss for git.mycompany.com IN AAAA
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Transaction 17301 for <git.mycompany.com IN AAAA> scope dns on vpn0/*.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using feature level UDP+EDNS0 for transaction 17301.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using DNS server 10.10.7.10 for transaction 17301.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Sending query packet with id 17301.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Cache miss for git.mycompany.com IN AAAA
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Transaction 59919 for <git.mycompany.com IN AAAA> scope dns on enp0s3/*.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using feature level UDP+EDNS0 for transaction 59919.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Using DNS server 192.168.1.1 for transaction 59919.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Sending query packet with id 59919.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Processing query...
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Processing incoming packet on transaction 59919 (rcode=SUCCESS).
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Not caching negative entry without a SOA record: git.mycompany.com IN AAAA
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Transaction 59919 for <git.mycompany.com IN AAAA> on scope dns on enp0s3/* now complete with <success> from network (unsigned).
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Freeing transaction 17301.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Sending response packet with id 18085 on interface 1/AF_INET.
Oct 21 12:18:48 mark-VirtualBox systemd-resolved[3103]: Freeing transaction 59919.

Just like in my earlier comments, it is looking up on both DNS servers (leaking!) and only returning the results from my ISP. I don't know why it is looking up AAAA records when I don't even have ipv6 configured either.

Additionally, I again had to manually configure the domain search zone to include "mycompany.com" for it to even try to look up over the VPN. The behavior in every other OS is that I connect to the VPN then every lookup goes over the VPN. The observed behavior here is that when connected to the VPN, only records in the domain search zone are looked up over the VPN as well as the ISP.
This should not be happening. I can understand an argument for having only records matching a domain search zone going over the VPN, but no way should they be silently going to the ISP as well.

@arno01

This comment has been minimized.

arno01 commented Oct 23, 2018

To anyone concerned about his DNS requests are either:

A. not going to the ISP DNS servers through the VPN;
B. not going through the DNS'es provided by the VPN server;

Cases A and B are different.
For the A. - it would be enough to add redirect-gateway def1 to one's openvpn configuration (equivalent of checking the checkbox default gateway in GUI). Except for when DNS server is sitting in the private network (RFC1918) or when one does not want use his ISP-provided DNS servers at any cost, even through the VPN tunnel.

For the B. - This is due to systemd-resolve's domain name routing and I think I have already explained it in my above post.

To see where the DNS requests would go, it should be enough just to run systemd-resolve --status command and see if the DNS Domain: equals to ~.. If not, then add dhcp-option DOMAIN-ROUTE . option to your openvpn profile, so it would prevent the DNS leakage for both case A. and B. Note though, that this would work when jonathanio/update-systemd-resolved@b417bac script is used along with the openvpn. Or that can be done manually by invoking busctl call. The whole issue is that the VPN software did not take into account systemd-resolve's domain name routing yet... this is why we have to use scripts like that.

@mdunc looking at your configuration, you can see that your link enp0s3 has the DNS Domain: ~.. This explains why you are having the DNS leakage.

@mdunc

This comment has been minimized.

mdunc commented Oct 23, 2018

@arno01 , thanks for the response. While I think this is terrible and insecure default behavior to have ~. in the search zone for the default connection and inconsistent with the behavior in every other OS, that does explain the leaking. What I do not understand though is that when I'm connected to my VPN with a mycompany.com search zone, why are the requests to mycompany.com not returned from VPN DNS server? They are being looked up as you can see in the logs, but only results from the ISP are returned. Does ~. take precedence over specific named search zones? It also does not explain my earlier comment where results would only be returned correctly for a minute or so before going to be returning results from the ISP.

@arno01

This comment has been minimized.

arno01 commented Oct 23, 2018

@mdunc I do not have ~. in the search zone for the default connection (though, before I had it there.. strange.. maybe it appears there as soon as I am connected to VPN? not sure.). Probably this is coming from your DHCP server or the way your network manager configures the connection. What have you got when running nmcli device show , nmcli connection show <name of your interface/wifi>, can also grep for "dns" there. And what is in your /etc/nsswitch.conf file for hosts:? Maybe you will get some clue from there?

About your earlier comment, I can't say much at the moment.. maybe @poettering has some ideas?
Otherwise, have you made sure to try querying the DNS servers after/before systemd-resolve --flush-caches ?

@mdunc

This comment has been minimized.

mdunc commented Oct 25, 2018

Output of nmcli connection show Wired\ Connection\ 1

connection.id:                          Wired connection 1
connection.uuid:                        546ae255-85ba-3c02-b7dc-c02f84654586
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              --
connection.autoconnect:                 yes
connection.autoconnect-priority:        -999
connection.autoconnect-retries:         -1 (default)
connection.auth-retries:                -1
connection.timestamp:                   1540440496
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             08:00:27:AE:B0:66
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       ""
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv6.method:                            ignore
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       ""
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       0 (disabled)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.dhcp-duid:                         --
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           Wired connection 1
GENERAL.UUID:                           546ae255-85ba-3c02-b7dc-c02f84654586
GENERAL.DEVICES:                        enp0s3
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/2
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         10.0.2.15/24
IP4.GATEWAY:                            10.0.2.2
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 10.0.2.2, mt = 100
IP4.ROUTE[2]:                           dst = 23.29.62.70/32, nh = 10.0.2.2, mt = 100
IP4.ROUTE[3]:                           dst = 10.0.2.2/32, nh = 0.0.0.0, mt = 100
IP4.ROUTE[4]:                           dst = 10.0.2.0/24, nh = 0.0.0.0, mt = 100
IP4.ROUTE[5]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
IP4.DNS[1]:                             192.168.1.1
IP4.DOMAIN[1]:                          homenet
DHCP4.OPTION[1]:                        broadcast_address = 10.0.2.255
DHCP4.OPTION[2]:                        dhcp_lease_time = 86400
DHCP4.OPTION[3]:                        dhcp_message_type = 5
DHCP4.OPTION[4]:                        dhcp_server_identifier = 10.0.2.2
DHCP4.OPTION[5]:                        domain_name = homenet
DHCP4.OPTION[6]:                        domain_name_servers = 192.168.1.1
DHCP4.OPTION[7]:                        expiry = 1540526598
DHCP4.OPTION[8]:                        filename = Ubuntu 18.10.pxe
DHCP4.OPTION[9]:                        ip_address = 10.0.2.15
DHCP4.OPTION[10]:                       network_number = 10.0.2.0
DHCP4.OPTION[11]:                       next_server = 10.0.2.4
DHCP4.OPTION[12]:                       requested_broadcast_address = 1
DHCP4.OPTION[13]:                       requested_domain_name = 1
DHCP4.OPTION[14]:                       requested_domain_name_servers = 1
DHCP4.OPTION[15]:                       requested_domain_search = 1
DHCP4.OPTION[16]:                       requested_host_name = 1
DHCP4.OPTION[17]:                       requested_interface_mtu = 1
DHCP4.OPTION[18]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[19]:                       requested_netbios_name_servers = 1
DHCP4.OPTION[20]:                       requested_netbios_scope = 1
DHCP4.OPTION[21]:                       requested_ntp_servers = 1
DHCP4.OPTION[22]:                       requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[23]:                       requested_routers = 1
DHCP4.OPTION[24]:                       requested_static_routes = 1
DHCP4.OPTION[25]:                       requested_subnet_mask = 1
DHCP4.OPTION[26]:                       requested_time_offset = 1
DHCP4.OPTION[27]:                       requested_wpad = 1
DHCP4.OPTION[28]:                       routers = 10.0.2.2
DHCP4.OPTION[29]:                       subnet_mask = 255.255.255.0
IP6.ADDRESS[1]:                         fe80::a00:27ff:feae:b066/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[2]:                           dst = fe80::/64, nh = ::, mt = 256

Output of nmcli connection show vpn0:

connection.id:                          vpn0
connection.uuid:                        aa03a0b7-534c-4ad7-b891-bc194919ad35
connection.stable-id:                   --
connection.type:                        tun
connection.interface-name:              vpn0
connection.autoconnect:                 no
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.auth-retries:                -1
connection.timestamp:                   1540440496
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
ipv4.method:                            manual
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       ""
ipv4.dns-priority:                      100
ipv4.addresses:                         10.120.0.71/24
ipv4.gateway:                           --
ipv4.routes:                            { ip = 54.152.126.226/32, mt = 50 }; { ip = 52.2.233.62/32, mt = 50 }; { ip = 13.54.56.44/32, mt = 5
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     --
ipv4.dhcp-fqdn:                         --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv6.method:                            link-local
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       ""
ipv6.dns-priority:                      100
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     stable-privacy
ipv6.dhcp-duid:                         --
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     --
ipv6.token:                             --
tun.mode:                               1 (tun)
tun.owner:                              124
tun.group:                              --
tun.pi:                                 no
tun.vnet-hdr:                           no
tun.multi-queue:                        no
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           vpn0
GENERAL.UUID:                           aa03a0b7-534c-4ad7-b891-bc194919ad35
GENERAL.DEVICES:                        vpn0
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        no
GENERAL.DEFAULT6:                       no
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/3
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/3
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.ADDRESS[1]:                         10.120.0.71/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 54.152.126.226/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[2]:                           dst = 52.2.233.62/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[3]:                           dst = 13.54.56.44/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[4]:                           dst = 52.3.68.170/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[5]:                           dst = 52.57.179.114/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[6]:                           dst = 52.58.208.13/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[7]:                           dst = 34.231.89.67/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[8]:                           dst = 10.99.1.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[9]:                           dst = 10.199.1.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[10]:                          dst = 10.20.7.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[11]:                          dst = 10.10.12.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[12]:                          dst = 10.200.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[13]:                          dst = 10.20.4.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[14]:                          dst = 10.150.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[15]:                          dst = 10.100.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[16]:                          dst = 192.168.25.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[17]:                          dst = 192.168.3.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[18]:                          dst = 172.31.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[19]:                          dst = 192.168.4.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[20]:                          dst = 192.168.100.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[21]:                          dst = 172.29.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[22]:                          dst = 172.28.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[23]:                          dst = 172.25.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[24]:                          dst = 172.22.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[25]:                          dst = 172.20.1.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[26]:                          dst = 172.17.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[27]:                          dst = 172.16.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[28]:                          dst = 162.217.193.0/25, nh = 0.0.0.0, mt = 50
IP4.ROUTE[29]:                          dst = 10.10.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[30]:                          dst = 10.10.4.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[31]:                          dst = 10.120.0.0/24, nh = 0.0.0.0, mt = 50
IP6.ADDRESS[1]:                         fe80::1415:4272:1c4f:15b1/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[2]:                           dst = fe80::/64, nh = ::, mt = 256

Output of nmcli device show:

GENERAL.DEVICE:                         enp0s3
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         08:00:27:AE:B0:66
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     Wired connection 1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/1
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         10.0.2.15/24
IP4.GATEWAY:                            10.0.2.2
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 10.0.2.2, mt = 100
IP4.ROUTE[2]:                           dst = 23.29.62.70/32, nh = 10.0.2.2, mt = 100
IP4.ROUTE[3]:                           dst = 10.0.2.2/32, nh = 0.0.0.0, mt = 100
IP4.ROUTE[4]:                           dst = 10.0.2.0/24, nh = 0.0.0.0, mt = 100
IP4.ROUTE[5]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000
IP4.DNS[1]:                             192.168.1.1
IP4.DOMAIN[1]:                          homenet
IP6.ADDRESS[1]:                         fe80::a00:27ff:feae:b066/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[2]:                           dst = fe80::/64, nh = ::, mt = 256

GENERAL.DEVICE:                         vpn0
GENERAL.TYPE:                           tun
GENERAL.HWADDR:                         (unknown)
GENERAL.MTU:                            1406
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     vpn0
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/3
IP4.ADDRESS[1]:                         10.120.0.71/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 54.152.126.226/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[2]:                           dst = 52.2.233.62/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[3]:                           dst = 13.54.56.44/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[4]:                           dst = 52.3.68.170/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[5]:                           dst = 52.57.179.114/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[6]:                           dst = 52.58.208.13/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[7]:                           dst = 34.231.89.67/32, nh = 0.0.0.0, mt = 50
IP4.ROUTE[8]:                           dst = 10.99.1.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[9]:                           dst = 10.199.1.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[10]:                          dst = 10.20.7.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[11]:                          dst = 10.10.12.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[12]:                          dst = 10.200.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[13]:                          dst = 10.20.4.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[14]:                          dst = 10.150.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[15]:                          dst = 10.100.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[16]:                          dst = 192.168.25.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[17]:                          dst = 192.168.3.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[18]:                          dst = 172.31.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[19]:                          dst = 192.168.4.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[20]:                          dst = 192.168.100.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[21]:                          dst = 172.29.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[22]:                          dst = 172.28.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[23]:                          dst = 172.25.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[24]:                          dst = 172.22.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[25]:                          dst = 172.20.1.0/24, nh = 0.0.0.0, mt = 50
IP4.ROUTE[26]:                          dst = 172.17.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[27]:                          dst = 172.16.0.0/16, nh = 0.0.0.0, mt = 50
IP4.ROUTE[28]:                          dst = 162.217.193.0/25, nh = 0.0.0.0, mt = 50
IP4.ROUTE[29]:                          dst = 10.10.0.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[30]:                          dst = 10.10.4.0/22, nh = 0.0.0.0, mt = 50
IP4.ROUTE[31]:                          dst = 10.120.0.0/24, nh = 0.0.0.0, mt = 50
IP6.ADDRESS[1]:                         fe80::1415:4272:1c4f:15b1/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[2]:                           dst = fe80::/64, nh = ::, mt = 256

GENERAL.DEVICE:                         lo
GENERAL.TYPE:                           loopback
GENERAL.HWADDR:                         00:00:00:00:00:00
GENERAL.MTU:                            65536
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         127.0.0.1/8
IP4.GATEWAY:                            --
IP6.ADDRESS[1]:                         ::1/128
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ::1/128, nh = ::, mt = 256

Contents of nsswitch.conf:

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files systemd
group:          files systemd
shadow:         files
gshadow:        files

hosts:          files mdns4_minimal [NOTFOUND=return] dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

Again, this is a default Ubuntu 18.10 install with nothing done to it other than adding an openconnect VPN.

@arno01

This comment has been minimized.

arno01 commented Oct 26, 2018

Hum, you are right @mdunc, the systemd-resolved goes through each DNS server it sees over each interface in the system. Confirmed by connecting to 2 VPN's, having this set:

  1. systemctl edit systemd-resolved.service
    [Service]
    Environment=SYSTEMD_LOG_LEVEL=debug
    
  2. systemctl restart systemd-resolved
  3. journalctl -u systemd-resolved.service -f

Then, looking at the code, I could see that it indeed tries each DNS (first alive one per interface) it finds over all network interfaces in the system:

https://github.com/systemd/systemd/blob/v237/src/resolve/resolved-dns-stub.c
https://github.com/systemd/systemd/blob/v237/src/resolve/resolved-dns-query.c#L779-L786
https://github.com/systemd/systemd/blob/v237/src/resolve/resolved-dns-query.h#L36-L47
https://github.com/systemd/systemd/blob/v237/src/resolve/resolved-dns-transaction.c#L525

int manager_dns_stub_start(Manager *m) {

TCP:

1. static int manager_dns_stub_tcp_fd(Manager *m) {
2.  r = sd_event_add_io(m->event, &m->dns_stub_tcp_event_source, fd, EPOLLIN, on_dns_stub_stream, m);
3. static int on_dns_stub_stream(sd_event_source *s, int fd, uint32_t revents, void *userdata) { => "Got DNS stub TCP query packet for id"
4. dns_stub_process_query(s->manager, s, s->read_packet);

UDP:

1. static int manager_dns_stub_udp_fd(Manager *m) {
2.  r = sd_event_add_io(m->event, &m->dns_stub_udp_event_source, fd, EPOLLIN, on_dns_stub_packet, m);
3. static int on_dns_stub_packet(sd_event_source *s, int fd, uint32_t revents, void *userdata) => "Got DNS stub UDP query packet for id"
4. dns_stub_process_query(m, NULL, p);

5. static void dns_stub_process_query(Manager *m, DnsStream *s, DnsPacket *p) {
6. r = dns_query_go(q);

7. int dns_query_go(DnsQuery *q) {
8. static int dns_query_candidate_go(DnsQueryCandidate *c) {
9. int dns_transaction_go(DnsTransaction *t) {

10 (tcp). static int dns_transaction_open_tcp(DnsTransaction *t) { 
10 (udp). static int dns_transaction_emit_udp(DnsTransaction *t) {
11. r = dns_transaction_pick_server(t);

12. static int dns_transaction_pick_server(DnsTransaction *t) {
13. log_debug("Using DNS server %s for transaction %u.", dns_server_string(t->server), t->id);

Reading @poettering 's comments from the other thread #5755 it makes this also clear.

It looks like systemd-resolved deserves fix ASAP. Otherwise it violates privacy expectation of many, making VPN connection useless in cases when one wants to get around the ISP provided DNSes.

@poettering any comments on this, please?


Meanwhile, the workaround would be the one provided by @snabb above, or as follows:

Say you are connected to your ISP over WiFi wlp1s0 and to two VPN links: tun0, tun1.

$ systemd-resolve --status | grep -A1 -Ew "Link|DNS"
Link 4 (tun1)
      Current Scopes: DNS
       LLMNR setting: yes
--
         DNS Servers: 22.x.x.x.x
                      22.x.x.x.x
          DNS Domain: companyC.com
--
Link 3 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
--
         DNS Servers: 10.x.x.x.x
                      10.x.x.x.x
          DNS Domain: companyB.com
--
Link 2 (wlp1s0)
      Current Scopes: DNS
       LLMNR setting: yes
--
         DNS Servers: 155.x.x.x
                      155.x.x.x
          DNS Domain: yourISP.com

Then use Link 2 (corresponding to your wifi connection) and set it to 0 (remove):

$ sudo busctl call org.freedesktop.resolve1 /org/freedesktop/resolve1 org.freedesktop.resolve1.Manager SetLinkDNS 'ia(iay)' 2 0

That will remove the ISP link from the DNS scope, so systemd-resolved won't use the ISP DNS servers.

$ systemd-resolve --status | grep -A1 -Ew "Link|DNS"
...
Link 2 (wlp1s0)
      Current Scopes: none
--
          DNS Domain: yourISP.com

Or you can just block the DNS servers provided by your ISP:

$ sudo nmcli d show wlp1s0 |grep DNS | awk '{print $NF}' | while read IP; do route add -host ${IP} reject; done

To unblock just change add to del.

@shdown

This comment has been minimized.

shdown commented Oct 29, 2018

Hello,

I would like to add that falling back to ISP’s DNS server once a custom one “stopped working” is bad and probably unwanted in most cases: systemd-resolved cannot discriminate between “this particular DNS server is down” and “there is (temporarily) no connection to the Internet at all”.

So sometimes my connection breaks and systemd-resolved switches to ISP’s DNS, which is very annoying. I’m going to replace it with a static /etc/resolv.conf with namespace 8.8.8.8 in it.

@sridharbasam

This comment has been minimized.

sridharbasam commented Oct 30, 2018

I worked around this issue by disabling systemd-resolved and switching to dnsmasq. dnsmasq-base is already distributed as part of Ubuntu 18.10.

  • edit NetworkManager.conf
    dns=dnsmasq rc-manager=resolvconf

  • sudo systemctl disable systemd-resolved

  • sudo systemctl stop systemd-resolved

  • sudo systemctl restart NetworkManager

@godfuture

This comment has been minimized.

godfuture commented Nov 13, 2018

In general, there is quite a user base that does not agree with systemd-resolve behavior (#5755). Including me. As Poettering does not give the impression that they will change it or allow customization, the only way to get around it, is to de-install systemd-resolve. A pitty...

@arno01

This comment has been minimized.

arno01 commented Dec 4, 2018

Looks like there is a fix waiting #11050

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment