"Override local DNS" does not actually stop querying local DNS

[ornate-lemon-horse]

What is the issue?

When a global DNS server is set in the admin interface and "Override local DNS" is enabled, the system still contacts the local DNS server, in addition to the DNS server specified in the admin interface. I expected that when the "Override local DNS server" setting is enabled, the local DNS server would not be used. This occurs in both a web browser (Firefox) and ping.

Steps to reproduce

Tailscale set to allow incoming connections, allow subnets, allow DNS, and run unattended
In admin panel, have a split DNS entry, set global DNS to Cloudflare, and enable "Override local DNS"
Open Wireshark, start capturing, and use filter ip.addr == 1.1.1.1 || ip.addr == 1.0.0.1 || dns
Open Firefox (with its own DoH setting disabled) and browse to a website. Or, run ping google.com
In Wireshark capture, observe simultaneous DNS activity to Cloudflare over DoH (expected) and system default resolver over port 53 (unexpected)

Are there any recent changes that introduced the issue?

No response

OS

Windows

OS version

20H2, build 19042.1645

Tailscale version

1.22.2

Bug report

BUG-7533d3f5ffa515b26e2432d873cb78d0f7f23890d82081c7dd698dc1374a109e-20220422032618Z-9baa754b87e7df4e

[dbz2k]

Even though it queries the local dns I do notice that pi-hole still works, and block ads.

[lincolnmantracer]

Since it's not mentioned in the issue, I wanted to point out that this actually breaks resolution of internal services for our org when it happens: the local or ISP resolver returns a negative result, and that's the only response the user sees:

Get "https://internal.example.com": dial tcp: lookup internal.example.com on 192.168.86.1:53: server misbehaving

If I had to guess, is it racing and returning a response from whichever resolver answers first?

[SergeantSerk]

Same here, running IPLeak and DNSLeakTest on Windows 10, I can see my UK IP address (where I am running my own home server as an exit node) but DNS is showing both UK and Turkey (I am currently abroad).

Interestingly, I do not observe the same behaviour on my iPhone iOS 15.5 (configured to use my UK server as an exit node too) however many times I've tried, both on WiFi and LTE.

This is with "Override local DNS" enabled with Quad9 and Cloudflare IPv4 and IPv6 DNS addresses.

[OliverCole]

I see the opposite to @SergeantSerk - traffic on the Windows 10 exit node (v1.34.2) goes to the Tailscale overridden DNS servers, but on my iPhone (v1.34.1, 16.1), traffic goes to the exit node and hits both the Tailscale overridden DNS servers and the ISP servers.

[sailorfrag]

I've done some investigation into this, Microsoft documentation says that if the preferred DNS server doesn't provide a response within 1 second, it will try DNS servers on all network interfaces. While Tailscale's DNS forwarding could probably use further refinement, even if it were perfect there will be occasional times when the configured DNS server will take too long to respond and Windows will trigger this behavior.

I don't see an obvious way to disable this behavior but when "Override local DNS" is enabled, that is what we would like to do. We will need to investigate further as to what are options are here. For example, deleting all DNS settings from other interfaces may be possible but is likely going to have undesired side effects.

When using an exit node with LAN access disabled, Tailscale blocks non-Tailscale outgoing traffic (this is standard procedure for VPN software that expects to control all Internet access) which may mitigate the issue.

[wingcomm]

Pretty Microsoft is gonna throw its weight around and prevent this from working eventually.

https://techcommunity.microsoft.com/t5/networking-blog/announcing-zero-trust-dns-private-preview/ba-p/4110366