Tailscale still uses "direct" mode for dns when resolv.conf is linked to /run/systemd/resolve/resolv.conf

[Cnly]

What is the issue?

I'm on Ubuntu 22.04 using NetworkManager and systemd-resolved. I went over the logic described in the figure in https://tailscale.com/blog/sisyphean-dns-client-linux/ and thought that tailscale should arrive at "Use systemd-resolved."

However in the log it says it's using the "direct" mode, and resolv.conf is overwritten by tailscale. This seems to create DNS loops: Every time I send a DNS request, tcpdump shows thousands of them; then after a while DNS would stop working altogether, with lots of "dns udp query: request queue full" in the logs.

After much debugging, I found that my /etc/resolv.conf is a link to /run/systemd/resolve/resolv.conf, not /run/systemd/resolve/**stub-**resolv.conf as wanted by tailscale's code. Changing the link fixes the issue. But overall, I think tailscale should detect systemd-resolved even if the stub server isn't being used?

Steps to reproduce

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf and restart tailscale.

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu 22.04

Tailscale version

1.38.2

Other software

NetworkManager, systemd-resolved

Bug report

BUG-96c93e336ba3e0bb3740a14f9fa491e23a7b037a64e15ea7a9f21a6326e55cfb-20230323124211Z-1bbbee80842ddf94

[DentonGentry]

The template requests a tailscale bugreport. Without it we cannot look up telemetry and will not make progress on diagnosing this.

[Cnly]

@DentonGentry Sorry, it wasn't marked as required. I've updated the post to include it. BTW is there any way we can also see debug logs, etc.? I don't remember seeing relevant CLI options.

[DentonGentry]

BTW is there any way we can also see debug logs, etc.?

journalctl -u tailscaled

https://tailscale.com/kb/1011/log-mesh-traffic/?tab=linux

[Cnly]

@DentonGentry Thanks for the quick reply, but I've been using the command you mentioned, and that's where I got the "dns udp query: request queue full" lines.

In the page you linked to, it says

Some logs are centrally collected by Tailscale for debugging. [...] Right now, logs are only accessible locally on each node.

How can we access them? Does tailscaled's default log level writes those logs to systemd journal?

And these dbg calls in the code look particularly helpful. Is there any way to see them?

tailscale/net/dns/manager_linux.go

Line 126 in 731688e

dbg("rc", "missing")

[DentonGentry]

Does tailscaled's default log level writes those logs to systemd journal?

Yes.

And these dbg calls in the code look particularly helpful. Is there any way to see them?

They should appear in journalctl.

tailscale/net/dns/manager_linux.go

Lines 86 to 91 in 731688e

	defer func() {
	if ret != "" {
	dbg("ret", ret)
	}
	logf("dns: %v", debug)
	}()

[wont-work]

I'm also affected by this.

Arch Linux, using systemd-networkd and systemd-resolved (no NetworkManager)

% tailscale bugreport
BUG-9dd5fd6fc648625d3b482f3893ac8b00c6cce97b2749ee57d4def094b62e7e19-20230717111017Z-c7dcb3a7ede7f095