100.100.100.100 handles reverse DNS requests even though override local DNS is enabled

[beningodfrey4]

What is the issue?

I run my own CoreDNS server within the tailnet with one of the zones serving records of all the tailnet devices both forward and backward. It used to work perfectly while enabling override local DNS, using search domains for the zones and disabling MagicDNS.

With the introduction of tailnet names *.<tailnet-name>.ts.net, reverse lookups aren’t forwarded to my overridden DNS server but handled internally by 100.100.100.100 to resolve to <hostname>.<tailnet-name>.ts.net instead of my DNS server’s authoritative zone.

This breaks Samba connections as the samba server accepts connections only from my zone, which it verifies with a reverse lookup, that now sadly goes somewhere else.

I know you guys are working towards DNS that just works:tm:, but would it be possible to ensure that “override local DNS” includes reverse DNS lookups as well? Or better yet, make tailnet name/auto-DNS opt-in, just like MagicDNS?

Steps to reproduce

1. Run a DNS server in some device on the Tailscale interface serving records of all the devices in the tailnet in some zone.
2. Add the IP and zone to split-DNS, add the IP to global nameservers and enable "override local dns".
3. Running dig @100.100.100.100 -x <ip> returns <ip>.<tailnet-name>.ts.net
4. Whereas running dig @<ip> -x <ip> returns the expected <ip>.<zone>

Are there any recent changes that introduced the issue?

I believe that it was caused by recent changes to 100.100.100.100 as a local DNS server of sorts and the tailscale fun names stuff.

OS

Linux, Windows, iOS, Android

OS version

Arch Linux, Windows 10, iOS 16.4, Android 12

Tailscale version

All devices run > 1.38

Other software

CoreDNS
Samba

Bug report

No response

[bradfitz]

Hah. Somehow I immediately think of https://xkcd.com/1172/

But more serious: I don't recall any recent change we made that should've changed this behavior. 100.100.100.100 has always answered authoritatively if it got the PTR query for a Tailscale IP.

Whether/how we program the OS's DNS configuration varies by OS & what else is running on Linux (https://tailscale.com/blog/sisyphean-dns-client-linux/ etc ... but that's a bit dated at this point)

On what OS/distro/version/other-DNS-stuff do you see the behavior you don't like?

[beningodfrey4]

Ha, I do understand that it isn't a standard use-case.

Interesting, I do remember it working late 2022 when I set it up, but I don't recall if Tailscale pushed the DNS server's IP itself, or 100.100.100.100 at that time. Is there any way to force upstream DNS instead of 100.100.100.100?

I observed it first on iOS 16.4 running Tailscale 1.38.2. The above "tests" were run on Arch Linux, with 1.38.4 on both the client machine and the machine with the DNS server.

[beningodfrey4]

So I did some messing around.

Disabling override local DNS made Tailscale push the IP itself instead of 100.100.100.100. Then reverse lookups failed completely both on iOS and Linux, not sure why. So I added another split-DNS search domain with 100.in-addr.arpa that finally worked around it and allowed my DNS server to handle PTR queries. Tested on both iOS and Linux.

But now since override local DNS is disabled, every non-tailscale IP DNS request goes to the default network interface and I no longer have network-wide adblock 🤦.

So I believe the issue title still stands, if override local DNS is enabled (among several other reasons), 100.100.100.100 is pushed as DNS server that handles PTR queries before checking with the Tailscale DNS settings. This now makes me wonder about #1543, and if there's a possibility of ordering all DNS rewrites / reroutes before 100.100.100.100 forwards it to global DNS?

[bradfitz]

Maybe what we can do is say that if you have the in-addr.arpa split DNS route, then we use those and forward to them from 100.100.100.100 instead of answering.

Technically, you actually want:

                    ~100.100.in-addr.arpa ~101.100.in-addr.arpa
                    ~102.100.in-addr.arpa ~103.100.in-addr.arpa
                    ~104.100.in-addr.arpa ~105.100.in-addr.arpa
                    ~106.100.in-addr.arpa ~107.100.in-addr.arpa
                    ~108.100.in-addr.arpa ~109.100.in-addr.arpa
                    ~110.100.in-addr.arpa ~111.100.in-addr.arpa
                    ~112.100.in-addr.arpa ~113.100.in-addr.arpa
                    ~114.100.in-addr.arpa ~115.100.in-addr.arpa
                    ~116.100.in-addr.arpa ~117.100.in-addr.arpa
                    ~118.100.in-addr.arpa ~119.100.in-addr.arpa
                    ~120.100.in-addr.arpa ~121.100.in-addr.arpa
                    ~122.100.in-addr.arpa ~123.100.in-addr.arpa
                    ~124.100.in-addr.arpa ~125.100.in-addr.arpa
                    ~126.100.in-addr.arpa ~127.100.in-addr.arpa
                    ~64.100.in-addr.arpa ~65.100.in-addr.arpa
                    ~66.100.in-addr.arpa ~67.100.in-addr.arpa
                    ~68.100.in-addr.arpa ~69.100.in-addr.arpa
                    ~70.100.in-addr.arpa ~71.100.in-addr.arpa
                    ~72.100.in-addr.arpa ~73.100.in-addr.arpa
                    ~74.100.in-addr.arpa ~75.100.in-addr.arpa
                    ~76.100.in-addr.arpa ~77.100.in-addr.arpa
                    ~78.100.in-addr.arpa ~79.100.in-addr.arpa
                    ~80.100.in-addr.arpa ~81.100.in-addr.arpa
                    ~82.100.in-addr.arpa ~83.100.in-addr.arpa
                    ~84.100.in-addr.arpa ~85.100.in-addr.arpa
                    ~86.100.in-addr.arpa ~87.100.in-addr.arpa
                    ~88.100.in-addr.arpa ~89.100.in-addr.arpa
                    ~90.100.in-addr.arpa ~91.100.in-addr.arpa
                    ~92.100.in-addr.arpa ~93.100.in-addr.arpa
                    ~94.100.in-addr.arpa ~95.100.in-addr.arpa
                    ~96.100.in-addr.arpa ~97.100.in-addr.arpa
                    ~98.100.in-addr.arpa ~99.100.in-addr.arpa

... at least until systemd/systemd#21526 is fixed.

But maybe we can make 100.in-addr.arpa an alias to expand to that, as that's almost certainly what people actually meant.

/cc @danderson

[beningodfrey4]

That works as well, a condition to check for *.in-addr.arpa from the list of search domains before answering PTR lookups would work fine.

Meanwhile, routing all domains with ~. as a search domain instead of enabling override local DNS also doesn't work, as the client gets the ASCII code \126 and the dot is removed.

Going back to 100.100.100.100 having to decide between handling and forwarding, my 2 cents is to separate concerns with a tab like "DHCP" next to the DNS tab in the admin console. Then advertising upstreams and domains would be handled by that "DHCP" tab. Now the DNS tab is a proper DNS resolver and can do MagicDNS, tailnet naming, custom records, forwarding, blocklists etc. Obviously, enabling any feature in the DNS tab would propagate only if the DHCP tab has upstream set to 100.100.100.100.

As it stands, there's no way of knowing if a client would get 100.100.100.100 or other upstreams directly and there could always be drift / edge-cases / unexpected behavior between clients like this issue.