Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-resolved does not keep the order of the DNS servers #5755

Open
diego-treitos opened this issue Apr 18, 2017 · 55 comments
Open

systemd-resolved does not keep the order of the DNS servers #5755

diego-treitos opened this issue Apr 18, 2017 · 55 comments
Labels

Comments

@diego-treitos
Copy link

@diego-treitos diego-treitos commented Apr 18, 2017

Submission type

  • Bug report
  • Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

Version 232

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Ubuntu 17.04

In case of bug report: Expected behaviour you didn't see

When having 2 nameservers like:

192.168.0.1
8.8.8.8

Defined in that order in /etc/resolv.conf I would expect to have the same behaviour than in resolv.conf: First use 192.168.0.1 and if for some reason it is not available, use 8.8.8.8.

I am seeing that systemd-resolved is switching nameservers randomly

Abr 18 16:40:01 boi systemd-resolved[1692]: Switching to DNS server 8.8.8.8 for interface eth0.
Abr 18 16:40:01 boi systemd-resolved[1692]: Switching to DNS server 192.168.0.1 for interface eth0.
Abr 18 16:40:06 boi systemd-resolved[1692]: Switching to DNS server 8.8.8.8 for interface eth0.
Abr 18 16:40:06 boi systemd-resolved[1692]: Switching to DNS server 192.168.0.1 for interface eth0.
Abr 18 16:40:11 boi systemd-resolved[1692]: Switching to DNS server 8.8.8.8 for interface eth0.
Abr 18 16:40:16 boi systemd-resolved[1692]: Switching to DNS server 192.168.0.1 for interface eth0.
Abr 18 16:40:16 boi systemd-resolved[1692]: Switching to DNS server 8.8.8.8 for interface eth0.
Abr 18 16:40:21 boi systemd-resolved[1692]: Switching to DNS server 192.168.0.1 for interface eth0.
Abr 18 19:16:09 boi systemd-resolved[1692]: Switching to DNS server 8.8.8.8 for interface eth0.

In case of bug report: Unexpected behaviour you saw

Random nameserver use

In case of bug report: Steps to reproduce the problem

just have 2 nameservers and use systemd-resolved service

@poettering
Copy link
Member

@poettering poettering commented Apr 24, 2017

resolved will always begin with the first configured DNS service, and switch to any other only after failures to contact it. If you turn on debug logging in resolved (by setting the SYSTEMD_LOG_LEVEL=debug env var for it), then you'll see the precise reason it switched over. Switching over can have many reasons: the IP route to the destination is missing, the server might simple not respond at all, or only with an error...

To turn on debug loggin, use "systemctl edit systemd-resolved", then write the two lines:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

and issue "systemctl restart systemd-resolved", then watch the output with "journalctl -u systemd-resolved -f", and look for the lines announcing the switch and the context before it.

I am pretty sure the output you'll see then will explain enough, hence I am closing this now. Feel free to reopen if it doesn't.

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented Apr 25, 2017

I cannot reopen the issue, I am afraid.

Regarding the issue, first of all, what I said is that I would expect systemd-resolved to behave just like the plain system resolv.conf file: first try one DNS and then the other (per request).
I am not seeing this in systemd-resolved it seems that when it switches (for whatever reason) it stays with that server and subsequent requests are checked against the primary DNS server.
In my case I have:

Primary DNS: 192.168.0.250
Secondary DNS: 8.8.8.8

The primary DNS works just fine. I never see it offline. Actually, when systemd-resolved switches to 8.8.8.8 I can just test the resolving like this:

$ dig +short router.lar

$ dig +short router.lar @192.168.0.250
192.168.0.1

So here we see that despite the primary DNS server being available, systemd-resolved is not using it.
This is happening in two different computers I have.

I've never had any problems with any of them until I upgraded them the new Ubuntu version that uses systemd-resolved. In one of them, I already disabled systemd-resolved and it works just fine (just like before using systemd-resolved). So clearly there is something wrong with the systemd-resolved behaviour.

Just in case I enabled the debug as you requested and this is what I see for the requests:

Abr 25 11:00:42 boi systemd-resolved[5221]: Got DNS stub UDP query packet for id 22949
Abr 25 11:00:42 boi systemd-resolved[5221]: Looking up RR for router.lar IN A.
Abr 25 11:00:42 boi systemd-resolved[5221]: NXDOMAIN cache hit for router.lar IN A
Abr 25 11:00:42 boi systemd-resolved[5221]: Transaction 63967 for <router.lar IN A> on scope dns on eth0/* now complete with <rcode-failure> from cache (unsigned).
Abr 25 11:00:42 boi systemd-resolved[5221]: Freeing transaction 63967.
Abr 25 11:00:42 boi systemd-resolved[5221]: Sending response packet with id 22949 on interface 1/AF_INET.

And this is what I see when it switches (Added >>> to easily see the switch):

>>> Apr 25 07:40:06 boi systemd-resolved[5221]: Switching to DNS server 8.8.8.8 for interface eth0.
Apr 25 07:40:06 boi systemd-resolved[5221]: Cache miss for go.trouter.io IN AAAA
Apr 25 07:40:06 boi systemd-resolved[5221]: Transaction 47232 for <go.trouter.io IN AAAA> scope dns on eth0/*.
Apr 25 07:40:06 boi systemd-resolved[5221]: Using feature level UDP+EDNS0+DO for transaction 47232.
Apr 25 07:40:06 boi systemd-resolved[5221]: Using DNS server 8.8.8.8 for transaction 47232.
Apr 25 07:40:06 boi systemd-resolved[5221]: Sending query packet with id 47232.
Apr 25 07:40:06 boi systemd-resolved[5221]: Timeout reached on transaction 29131.
Apr 25 07:40:06 boi systemd-resolved[5221]: Retrying transaction 29131.
>>> Apr 25 07:40:06 boi systemd-resolved[5221]: Switching to DNS server 192.168.0.250 for interface eth0.
Apr 25 07:40:06 boi systemd-resolved[5221]: Cache miss for go.trouter.io IN A
Apr 25 07:40:06 boi systemd-resolved[5221]: Transaction 29131 for <go.trouter.io IN A> scope dns on eth0/*.
Apr 25 07:40:06 boi systemd-resolved[5221]: Using feature level UDP for transaction 29131.
Apr 25 07:40:06 boi systemd-resolved[5221]: Sending query packet with id 29131.
Apr 25 07:40:06 boi systemd-resolved[5221]: Got DNS stub UDP query packet for id 350
Apr 25 07:40:06 boi systemd-resolved[5221]: Looking up RR for go.trouter.io IN A.
Apr 25 07:40:06 boi systemd-resolved[5221]: Processing query...
Apr 25 07:40:06 boi systemd-resolved[5221]: Got DNS stub UDP query packet for id 30693
Apr 25 07:40:06 boi systemd-resolved[5221]: Looking up RR for go.trouter.io IN AAAA.
Apr 25 07:40:06 boi systemd-resolved[5221]: Processing query...
Apr 25 07:40:11 boi systemd-resolved[5221]: Got DNS stub UDP query packet for id 63769
Apr 25 07:40:11 boi systemd-resolved[5221]: Looking up RR for browser.pipe.aria.microsoft.com IN A.
Apr 25 07:40:11 boi systemd-resolved[5221]: Processing query...
Apr 25 07:40:11 boi systemd-resolved[5221]: Timeout reached on transaction 47737.
Apr 25 07:40:11 boi systemd-resolved[5221]: Retrying transaction 47737.
>>> Apr 25 07:40:11 boi systemd-resolved[5221]: Switching to DNS server 8.8.8.8 for interface eth0.
Apr 25 07:40:11 boi systemd-resolved[5221]: Cache miss for browser.pipe.aria.microsoft.com IN A

Looks like it switched each time it has a problem resolving a record and then it keeps using that name server for next requests.

@poettering
Copy link
Member

@poettering poettering commented Apr 25, 2017

Regarding the issue, first of all, what I said is that I would expect systemd-resolved to behave just like the plain system resolv.conf file: first try one DNS and then the other (per request).

This is what happens. However, in contrast to classic nss-dns we have memory: when we noticed that a DNS server didn't respond or returned some failure, or for some other reason wasn't working for us, and we skip to the next, then we remember that and the next lookup is attempted with the new one. If that one fails too, then we'll skip to the next one and the next one and so on, until we reach the end of the list and start from the beginning of the list again.

This behaviour has the big advantage that we can build on what we learnt about a DNS server before, and don't waste the same timeout on a DNS server for each lookup should it not respond.

Or to say this differently: If you specify multiple DNS servers, then that's not a way to merge DNS zones or so. It's simply a way to define alternative servers should the first DNS server not work correctly.

If you want to route lookups in specific zones to specific DNS servers, then resolved doesn't really offer a nice way for that. A hack is to define multiple interfaces however, and configure different DNS servers and domains for them.

@poettering
Copy link
Member

@poettering poettering commented Apr 25, 2017

Apr 25 07:40:06 boi systemd-resolved[5221]: Timeout reached on transaction 29131.
Apr 25 07:40:06 boi systemd-resolved[5221]: Retrying transaction 29131.
Apr 25 07:40:06 boi systemd-resolved[5221]: Switching to DNS server 192.168.0.250 for interface eth0.

This is where the server switches, and the lines before tell you why: the DNS server didn't respond to our query with transaction ID 29131. Why it didn't respond isn't known: somehow no UDP response packet was received. This could be because the query or the response packet simply got dropped on the way, or because the server refused to reply... Either way, resolved will retry but use a different DNS server, in the hope that works better.

@poettering
Copy link
Member

@poettering poettering commented Apr 25, 2017

Apr 25 07:40:11 boi systemd-resolved[5221]: Timeout reached on transaction 47737.
Apr 25 07:40:11 boi systemd-resolved[5221]: Retrying transaction 47737.
Apr 25 07:40:11 boi systemd-resolved[5221]: Switching to DNS server 8.8.8.8 for interface eth0.

and here the same thing, when it swicthes back: the response for transaction 47747 wasn't received either, hence resolved tries the other server again, switching back.

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented Apr 25, 2017

This is where the server switches, and the lines before tell you why: the DNS server didn't respond to our query with transaction ID 29131. Why it didn't respond isn't known: somehow no UDP response packet was received. This could be because the query or the response packet simply got dropped on the way, or because the server refused to reply... Either way, resolved will retry but use a different DNS server, in the hope that works better.

Yes I see that. And precisely because it is using UDP it will be easier for some packages to get droped and that the DNS switches.
Surely you see the advantages of the configuration I have in place. In networks like small companies, you may send those nameservers via DHCP to all computers in your network so they have resolution for local and external domains. However, if for some reason the local DNS goes down, all your computers can still resolve internet domains. In other words, it is much easier that your local DNS fails than that the google DNS does, so it is like a strong failover.

With the current systemd implementation you lose that priority in resolving names as it works more like a round-robin, and I understand the advantages of that in many scenarios (quick DNS failover switch).

I think it would be great to have some configuration options on this like:

  • Choose between RR mode or Prioritized mode
  • Number of attempts before switching to next nameserver

Or even to periodically check for primary nameserver availability so you can go back to use it asap.

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented Apr 25, 2017

@poettering
Copy link
Member

@poettering poettering commented Apr 25, 2017

BTW, odd thing is that it looks easier to switch to the external nameserver, when this is never able to resolve the local domains.

Not sure I grok what you are trying to say? Note that if a DNS lookup results in a NODATA or NXDOMAIN reply, then that's considered final, and no other DNS server is tried. Again, defining multiple DNS servers is not a way to merge zones, it's a way to deal with unreliable servers, the assumption is always that all DNS servers configured provide the same dataset.

So I think I grok what you are trying to do, but quite frankly, I think that even without resolved involved, this scheme is not reliable, and basically just taking benefit from a specific implementation detail of nss-dns/glibc. You are merging two concepts in what you are trying to do: fallback due to unreliable servers, and "merging" of zones. And I think for the latter it would be better to do proper per-domain request routing, for which an RFE is file in #5573 for example

@thomasleplus
Copy link

@thomasleplus thomasleplus commented May 27, 2017

I have a similar situation than @diego-treitos. My company has a single internal DNS and so our DHCP server provides it as primary DNS, and OpenDNS as secondary. If any request to our DNS fails, systemd will switch to OpenDNS and I loose the ability to connect to internal servers. And since OpenDNS doesn't fail, I am never switching back to our DNS unless I disconnect and reconnect my network.

I agree that the proper solution would be having a reliable DNS server or, even better, two internal servers for redundancy. But while I try to convince our sysadmins of that, IMHO it would be nice to have an option.

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented May 29, 2017

I agree with that. I know that this may not be a direct problem with systemd, but this service is being used to replace a previous one, so I think it would be nice if it could work just like the service it is replacing.

@chrisisbd
Copy link

@chrisisbd chrisisbd commented Jun 7, 2017

Yes, I agree that this is a problem. I have just upgraded a system to ubuntu 17.04 and what used to work in 16.04 now no longer works. We need a way to say that the second DNS is only to be used if the first one fails, the first one should always be tried first.

@chrisisbd
Copy link

@chrisisbd chrisisbd commented Jun 7, 2017

Here's my output after adding the debug logging, it doesn't seem to make much sense:-

Jun 7 10:36:28 t470 systemd-resolved[2161]: Using system hostname 't470'.
Jun 7 10:36:28 t470 systemd-resolved[2161]: New scope on link *, protocol dns, family *
Jun 7 10:36:28 t470 systemd-resolved[2161]: Found new link 3/wlp4s0
Jun 7 10:36:28 t470 systemd-resolved[2161]: Found new link 2/enp0s31f6
Jun 7 10:36:28 t470 systemd-resolved[2161]: Found new link 1/lo
Jun 7 10:36:28 t470 systemd-resolved[2161]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.283 object=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.283 object=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.283 object=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Got message type=signal sender=org.freedesktop.DBus destination=:1.283 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 error=n/a
Jun 7 10:36:28 t470 systemd-resolved[2161]: Got message type=signal sender=org.freedesktop.DBus destination=:1.283 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 error=n/a
Jun 7 10:37:38 t470 systemd-resolved[2161]: Got DNS stub UDP query packet for id 1936
Jun 7 10:37:38 t470 systemd-resolved[2161]: Looking up RR for esprimo.zbmc.eu IN A.
Jun 7 10:37:38 t470 systemd-resolved[2161]: Switching to fallback DNS server 8.8.8.8.
Jun 7 10:37:38 t470 systemd-resolved[2161]: Cache miss for esprimo.zbmc.eu IN A
Jun 7 10:37:38 t470 systemd-resolved[2161]: Transaction 53812 for <esprimo.zbmc.eu IN A> scope dns on /.
Jun 7 10:37:38 t470 systemd-resolved[2161]: Using feature level UDP+EDNS0+DO+LARGE for transaction 53812.
Jun 7 10:37:38 t470 systemd-resolved[2161]: Using DNS server 8.8.8.8 for transaction 53812.
Jun 7 10:37:38 t470 systemd-resolved[2161]: Sending query packet with id 53812.
Jun 7 10:37:38 t470 systemd-resolved[2161]: Processing query...
Jun 7 10:37:39 t470 systemd-resolved[2161]: Processing incoming packet on transaction 53812.
Jun 7 10:37:39 t470 systemd-resolved[2161]: Verified we get a response at feature level UDP+EDNS0+DO from DNS server 8.8.8.8.
Jun 7 10:37:39 t470 systemd-resolved[2161]: Added NXDOMAIN cache entry for esprimo.zbmc.eu IN ANY 1799s
Jun 7 10:37:39 t470 systemd-resolved[2161]: Transaction 53812 for <esprimo.zbmc.eu IN A> on scope dns on / now complete with from network (unsigned).
Jun 7 10:37:39 t470 systemd-resolved[2161]: Sending response packet with id 1936 on interface 1/AF_INET.
Jun 7 10:37:39 t470 systemd-resolved[2161]: Freeing transaction 53812.
Jun 7 10:37:39 t470 systemd-resolved[2161]: Got DNS stub UDP query packet for id 1919
Jun 7 10:37:39 t470 systemd-resolved[2161]: Looking up RR for esprimo IN A.
Jun 7 10:37:39 t470 systemd-resolved[2161]: Sending response packet with id 1919 on interface 1/AF_INET.
Jun 7 10:37:39 t470 systemd-resolved[2161]: Processing query...

So why does it switch to using 8.8.8.8, it doesn't seem to have even tried 192.168.1.2.

@poettering
Copy link
Member

@poettering poettering commented Jun 7, 2017

@chrisisbd The "Switching to fallback DNS server 8.8.8.8." message indicates that you have no DNS servers configured at all, in which case resolved will use compiled-in fallback servers because it tries hard to just work also if you have a locally misconfigured system

@chrisisbd
Copy link

@chrisisbd chrisisbd commented Jun 7, 2017

No, I have a working DNS on the LAN which (when I use it from xubuntu 16.04 systems) works perfectly.

The relevant part from 'systemd-resolve --status' is:-

ink 3 (wlp4s0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 192.168.1.2
8.8.8.8
DNS Domain: zbmc.eu

Most of the time local names resolve OK on the 17.04 system too but it (randomly?) falls back to using the 8.8.8.8 server for no obvious reason.

@amazon750
Copy link

@amazon750 amazon750 commented Jun 8, 2017

Hi Lennart, thanks for all of your work so far. I'm trying to keep using systemd, but you can add me to the list of people for whom the old behaviour seemed to be standardised and useful, and the new behaviour seems like a regression.

the assumption is always that all DNS servers configured provide the same dataset.

That assumption doesn't seem universal. I too have local names that aren't in public DNS, and some local overrides for external names, neither of which work if the failover happens (I only have a secondary server listed for the same reason as these other fellas: to keep internet access working more reliably for the rest of the local users if the primary fails). Under the old system, with the same resolv.conf and the same primary DNS server, things worked as I designed nearly 100% of the time. Now, with systemd, it's become quite unreliable. I hadn't needed to do per-domain request routing before, but I'd be fine with that solution. I also like the suggestion of a switch to choose which behaviour the local admin prefers. Anything would be better, I've been reduced to editing /etc/hosts to relieve some frustration, which I haven't otherwise done in years.

And I think for the latter it would be better to do proper per-domain request routing, for which an RFE is file in #5573 for example

Actually, on thinking about it further, that isn't as good. I would still prefer to use my internal DNS as primary for everything, and have it forward requests that it can't answer. Then again, maybe my preference is a bad practice, and won't be supported. But as mentioned, this all used to work, now it doesn't. If that's by design and won't be changed, that's unfortunate.

@lifeboy
Copy link

@lifeboy lifeboy commented Jun 23, 2017

This is a problem, @poettering. The behaviour is a major change from the expected way and doesn't work in practice. If I specify 3 nameservers, the expectation that the first is always queried first, is settled. You can't change that now unilaterally.

Consider this scenario:

I have a VPN connection to a development environment where I have VM's running various tests. On the gateway on that cluster I run a DNS forwarder (dnsmasq) on pfsense. (192.168.121.1) Here I override the public DNS to resolv to a server on the LAN. This is not uncommon to do and in many corporate environments similar scenarios exist. In addition to the overriding of existing public DNS entries, I also add my own inhouse entries for my test servers.
Now, in addition to this development cluster, we run various production clusters on a similar basis. (192.168.0.1) Again, a DNS forwarder allow the resolution of a domain to LAN address instead of the public address.

Since we don't work in one location and precisely therefor that we use VPN to connect to the various clusters, we need the expected behaviour: Always try to resolve in this order:
192.168.121.1
192.168.0.1
8.8.8.8

What happens with systemd-resolved is this:
Try to resolve abc.com from 192.168.121.1. It resolves. Open tools and work on servers.
In the course of time, some entry for xyz.com doesn't resolve from 192.168.121.1. It does resolve from 192.168.0.1 and xyz.com is not accessible. However, quite soon after that abc.com is not found any more. This is because 192.168.0.1 doesn't have records for abc.com.

The only way to restore this is to clear the dns cache and restart systemd-resolved .

This is not acceptable and at the least we need a way to prevent this automatic jumping to a dns server lower down in the priority list.

@mourednik
Copy link

@mourednik mourednik commented Jun 25, 2017

Hey guys. The only fix appears to be "install Linux without systemd" or install BSD.

I'm not trolling. This is not a joke.

@systemd systemd locked and limited conversation to collaborators Jun 26, 2017
@systemd systemd unlocked this conversation Jul 9, 2017
@keszybz
Copy link
Member

@keszybz keszybz commented Jul 9, 2017

Hm, we could allow DNS= configuration to specify two tiers of servers (e.g. with the syntax DNS=1.2.3.4 -8.8.8.8 -8.8.4.4), where those minus-prefixed servers would only be used if the non-minus-prefixed servers fail. Not sure about the details — the way I think could work would be to: first, round-robin on the first tier servers, and then fall back to the second tier once all of the first-tier servers have accumulated enough failures, like maybe 5 out of last 10 queries. And after some timeout, let's say 10 minutes, we should switch back.

Of course such a setup is not useful for merging zones (as @poettering wrote) in any reliable way, but it makes it easier to achieve "soft failure", where some local names stop working but the internet is not fully broken when the local nameserver goes down. Also, thanks to automatic switching back after a delay, things would "unbreak" automatically.

@lifeboy
Copy link

@lifeboy lifeboy commented Jul 14, 2017

@keszybz keszybz removed the not-a-bug label Jul 15, 2017
@keszybz
Copy link
Member

@keszybz keszybz commented Jul 15, 2017

This was already explained above (#5755 (comment)), but I'll try again:

  • we rotate among servers when the current server is not responding to provide reliability.
  • we remember which server is "good" so that there's no initial delay.

Contrary to what you wrote, DNS clients do not cache answers in general. Actually, when programs are short-lived, they cannot cache answers even if they wanted to; every time a program is restarted is starts with a clean slate. The place where caching is performed is inside of systemd-resolved (or in another external cache, like nscd, sssd, etc., but with systemd-resolved running the idea is that you don't need those).

With DNSSEC, the delay from nonresponding name server becomes even more noticeable. We might want to adjust caching details, but it's a central feature of systemd-resolved functionality and it's not going away. (Both the memory of "last good" server, and previously queried resource records). So if you want something to change to accommodate your use case, help design the solution like proposed (#5755 (comment)) so that it works for you.

@keszybz keszybz reopened this Jul 15, 2017
@lifeboy
Copy link

@lifeboy lifeboy commented Jul 16, 2017

I think @keszybz's workaround is not a good idea. It's still not a solution that keeps the established behaviour and add strange new "features" for that that wish to enable them. Why does @poettering insist on breaking things that work just fine?

I'm being forced off systemd more and more and I now see that's a good think. The more people move away, the better.

@keszybz
Copy link
Member

@keszybz keszybz commented Jul 16, 2017

Well, we keep telling you why, and you keep asking.

@kroeckx
Copy link

@kroeckx kroeckx commented Sep 12, 2017

Since glibc doesn't do any checking of DNSSEC, all my the IP addresses in my resolv.conf have become the addresses of servers I run myself and are checking DNSSEC. So I would really like to avoid some fallback to some default server over which I have no control.

@darkstar
Copy link

@darkstar darkstar commented Sep 12, 2017

Maybe it just boils down to a timeout that makes systemd-resolved switch to the next server quicker than the usual resolver. That could explain that some people are seing a switch to the second server even though lookups with dig/nslookup work just fine. If so, it can probably be fixed or worked around.

But I think most people don't understand the fact that, as was already stated, the DNS servers are supposed to be exactly equivalent. And if they are equivalent, then it doesn't matter which one you choose. If you want reliable DNS, you have to provide one or more reliable DNS servers. Or, provide only one server (and deal with the occasional disruption if a single lookup fails) and let that server handle the forwarding to 8.8.8.8.
Yes, it might be annoying for home users that have only one DNS server. But there are already tons of other options that can help in such a setting (nscd, sssd, etc.) and there's no reason not to use them instead of systemd-resolved.

@Harleqin
Copy link

@Harleqin Harleqin commented Sep 12, 2017

Where does the assumption that all DNS servers are supposed to be equivalent come from?

You see, they are not.

@mthorpe7
Copy link

@mthorpe7 mthorpe7 commented Sep 13, 2017

@Harleqin - it comes from RFC 1034 and RFC 1035:

The strategy is to cycle around all of the addresses for all of the servers with a timeout between each transmission.

its fairly explicit that the resolver can determine the order:

To complete initialization of SLIST, the resolver attaches whatever history information it has to the each address in SLIST. This will usually consist of some sort of weighted averages for the response time of the address, and the batting average of the address (i.e., how often the address responded at all to the request).

@systemd systemd locked and limited conversation to collaborators Sep 13, 2017
@systemd systemd deleted a comment from tebruno99 Sep 13, 2017
@systemd systemd deleted a comment from ctrix Sep 13, 2017
@systemd systemd deleted a comment Sep 13, 2017
@poettering
Copy link
Member

@poettering poettering commented Sep 13, 2017

Sorry, but given how the quality of discussion has degraded and the number of inappropriate comments I had to delete has increased I have now locked this issue. I will unlock this again in a few days when things have become quieter. Thank you for understanding.

@tmccombs
Copy link

@tmccombs tmccombs commented Nov 26, 2019

The problem is that all configured DNS servers are assumed to be equivalent

Does that include response time?

Specifically, if I have two dns servers, and the first one is faster than the second (for example, because it is closer), if the first one fails temporarily will resolved ever switch back to the first one, or will it continue using the second, slower server even when the first one comes back up?

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented Nov 27, 2019

@tmccombs For what I understand from this thread, if your first (fast) server fails, your system will start using the second (slow) server until this last one fails. So no, when the first one comes back you will still be using the second slow one.

I forgot about this issue during these years because I just did systemctl disable systemd-resolved.service after this issue was locked for discussion.

In my opinion, adding some sort of priority to the nameservers makes sense and gives more power to the user. However, to my understanding the implementation of this is more complex as it would require to:

  • add some sort of monitorization to the failed nameservers so you can start using them when they are available. So you may have periodically query the failed DNS in the background and then report if it is available or not. Probably too complex for a real solution.

OR

  • retry the failed nameservers each N queries and if a query to a failed nameserver succeeds then mark it as default server again. This would be easier to implement but it would add slowness to DNS resolution while there is a nameserver down and specially if there are several down.

I would love to see the second option implemented, specially if N is configurable in a way that N=0 means it works like it is implemented right now (never retry fallen servers) and N>0 means to retry each N request so you can decide the impact that a fallen DNS server will have in your DNS resolution timings. When the system retries (request_number >= N), follows the order in the nameservers list until one responds and the first to respond is marked as the default.

In any case, it is up to @poettering to decide so... here we are :).

@spamcop
Copy link

@spamcop spamcop commented Nov 27, 2019

can you please add option to systemd-resolved and make it default to actually be compatible with "man resolv.conf"

If there are multiple servers, the resolver library queries them in the order listed.

until then, some environments have to unfortunately use alternatives to systemd-resolved as there is not possible to make those DNS servers equivalent, its just reality

@tmccombs
Copy link

@tmccombs tmccombs commented Nov 28, 2019

retry the failed nameservers each N queries and if a query to a failed nameserver succeeds then mark it as default server again. This would be easier to implement but it would add slowness to DNS resolution while there is a nameserver down and specially if there are several down.

This sounds reasonable to me. It also means that with N=1 the old behaviour of resolv.conf is restored.

A couple of variations that might be worth looking at:

  • Using a time instead of a count of queries. For example, retry a fallen DNS server if it has been more than 5 seconds since it was marked down (possibly with exponential backoff)
  • Make the request to the fallen DNS server in parallel with a request to the known-good dns server to minimize the latency if the desired DNS server is still down.
@charlesritchea
Copy link

@charlesritchea charlesritchea commented Mar 27, 2020

@poettering I've tried reading this whole thread without going into cardiac arrest, but I still don't understand how to solve a fundamental problem of VPN users. What is the recommended method / workaround whatever of using multiple DNS with different behavior, one for private resources, one for public, that used to work on the old resolver because of rotation?

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented Mar 27, 2020

@charlesritchea I am not sure that this issue is related to your problem. This issue is related to how systemd-resolved behaves with several configured DNS servers, however, when connecting to a VPN, usually the VPN client changes the DNS servers or even a DHCP server inside the VPN can send DNS servers, depending on the VPN client and server configuration. So basically the nameservers will be replaced and the ones from the VPN will be used.

The only issue I see here is that systemd-resolved presumes that the DNS that it receives will not have any important order. So basically it presumes that the whole world will behave as it wants even if during the past decades the DNS resolution worked with DNS server prioritation.

@poettering I understand that the implementation follows the RFCs but, don't you think that, as this adds a substantial change to how things have been working for decades, a configurable option can be added to make it work as it always did? Even if the default option is to behave like it is right now.

@charlesritchea
Copy link

@charlesritchea charlesritchea commented Mar 27, 2020

@diego-treitos My particular VPN, OpenVPN, relies on this hacky update-systemd-resolved script to add the DNS entries to /etc/resolv.conf; however, the order of the nameserver entries appears to matter, which I assumed was this problem. I've had to tweak the update-systemd-resolved to prepend the VPN hostnames instead of appending. This led me back to this discussion. And after reading it led me to my comment.

@diego-treitos
Copy link
Author

@diego-treitos diego-treitos commented Mar 27, 2020

@charlesritchea If it adds them (anywhere) instead of replacing them, then this issue affects that scenario too. Thanks for the clarification and for pointing it out.

Too bad that nobody in the systemd project cares. For now the first thing I do after installing a distro with systemd is systemctl mask systemd-resolved.

@ZigMeowNyan
Copy link

@ZigMeowNyan ZigMeowNyan commented Apr 16, 2020

I'd really like for this stateful switching behavior to be optional. Or for there to at least be some tiered servers so users can prioritize that at least one server from a tier consistently be contacted before servers in the next tier.

We should not expect DNS servers to contain the same information, or for DNS data to always be propagated to all servers. There are legitimate reasons for this to not be the case. Just as there are legitimate reasons to include DNS servers without knowledge of the internal systems for name resolution.

I've seen a lot of people arguing about particular implementations of the existing specification. I think they're correct in that implementing the specification doesn't require mandatory server rotation. But I don't think it's the strongest argument available.

I think that, in this case, the team has failed to understand that reality is itself a form of specification. And interoperability with other servers and applications may require conformance to the de facto "specification" of how everyone expects it to behave. If your specs conflict, you should either implement both or follow the expectation rather than the outdated document. When attempting to specification document to be more secure or performant, this behavior should be switchable or configurable. And as long as we don't have an implementation that solves all the related problems caused by divergent behavior, we shouldn't force that behavior on the users. Refusing to do so can make a tool incompatible and unusable.

When it comes to networks, compatibility is more important than "correctness". And network protocols involve millions of applications, networks, and machines. We're essentially forcing users of resolved to change their network topology out of stubbornness with a possible result of millions of lost labor hours. I don't see this break in compatibility as a net gain, but as a net loss.

@skuep
Copy link

@skuep skuep commented May 2, 2020

You know the feeling? You have been debugging all day to no avail, find an issue that is exactly your problem, read the first comment and think "oh thats from 2017, that can't be my problem. But let's read it anyway". Now I am at the bottom of this page and I cannot believe it.

I am trying to get Strongswan VPN going via the NetworkManager applet and for the life of me could not figure out why the split-DNS setup is not working. Now I know why, something I have taken for granted all these years is deliberately broken :-(
Seriously, I get some (VPN internal) DNS servers pushed to my machine, i can see them being added in journalctl -feu NetworkManager
After establishing the VPN connection, I can even see them in systemd-resolve --status added to my wifi interface, which I am currently using. That is how ipsec tunneling works.

Global
          DNS Domain: <redacted>
                      home
...
Link 3 (wlp61s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.8.0.4 <- 1st DNS server pushed via VPN
                      10.8.0.22 <- 2nd DNS server pushed via VPN
                      10.4.8.4 <- 3rd DNS server pushed via VPN
                      192.168.2.1 <- my home DNS server
          DNS Domain: <redacted>
                      ~.

I was under the assumption, that the pushed DNS servers are now of higher priority, because they come first. However they are not and this seems to be intended behavior.
How is split-DNS in this very common use-case supposed to work? Someone must have had this use-case in mind!
By the way, when I restart systemd-resolved after establishing the tunnel, everything works fine. I guess that is because systemd-resolved now starts it's "game" from zero and picks the first DNS server, which is the one pushed via VPN.

I have another setup using openvpn that I am starting via command-line (instead of network-manager) where I update the pushed DNS server using resolvconf. That has worked very well over the years, but I wanted to make things easier by using the NetworkManager GUI and an IPsec-style tunnel. Now very ironically the latter now took me a day to a half-working solution, while the openvpn script I wrote took me an hour.

$ ls -al /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 Sep 18  2018 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search <redacted> home

Update: I now disabled systemd-resolved (it was always not very welcome with defaulting to hardcoded DNS servers...) and switched to dnsmasq as system resolver, which correctly honors the VPN DNS server.

@bronger
Copy link

@bronger bronger commented May 20, 2020

Let us do something constructive and helpful. How does a workaround work? How do I tell NetworkManager to use the DHCP-provided DNS'es for resolv.conf instead of 127.0.0.53?

As for Ubuntu 20.04, I not only had to mask and stop systemd-resolved and remove the symlink at /etc/resolv.conf, I also had to explicitly set dns=default in /etc/NetworkManager/NetworkManager.conf (and of course restarting NetworkManager), although https://developer.gnome.org/NetworkManager/stable/NetworkManager.conf.html asserts that removing the symlink should be enough.

If this procedure differs in other OS’es, please tell so. Issue threads are a valid place for workarounds after all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.