Fix DNS redirection when port 53 is already in use

[Laotree]

Before posting

• I searched existing discussions and issues, including closed ones, and checked the relevant docs.
• I believe this is a product bug rather than a configuration or setup question.
• I can reproduce this issue, or for intermittent issues I've included trigger, frequency, and timing details below.
• I removed or anonymized sensitive data from logs, screenshots, and configuration.

Affected area

Client / Agent, DNS

Deployment type

NetBird Cloud

Operating system or environment

Linux

NetBird version and upgrade status

0.78.0

Did this work before?

No, this never worked

Regression details

No response

Summary

Proposal:

When Nameservers: 0/0 Available, NetBird should not override the host's DNS configuration. The effective default should be equivalent to disableDns=true until a usable NetBird nameserver is available.

Current behavior

When NetBird DNS is enabled (disableDns=false), but the peer has no available NetBird nameservers, the client can still configure and use the NetBird DNS resolver.

In this situation, netbird status -d reports:

Nameservers: 0/0 Available

However, DNS queries may still be directed to the NetBird DNS resolver instead of the system's normal DNS configuration.

For example:

$ dig netbird.io
;; communications error to 100.101.x.x#53: timed out
;; communications error to 100.101.x.x#53: timed out
;; communications error to 100.101.x.x#53: timed out

;; no servers could be reached

After disabling NetBird DNS:

$ netbird down
Disconnected

$ netbird up --disable-dns=true
Connected

DNS immediately works again through the system resolver:

$ dig netbird.io

;; ANSWER SECTION:
netbird.io. 50 IN A 216.150.1.1

;; SERVER: 8.8.8.8#53(8.8.8.8)

Expected behavior

If NetBird reports:

Nameservers: 0/0 Available

NetBird should not take over the system DNS configuration.

The existing system DNS configuration should remain in use until at least one NetBird nameserver becomes available.

Steps to reproduce

when Nameservers: 0/0 Available
netbird down && netbird up --disable-dns=true
dig netbird.io

Environment and topology

N/A

Self-hosted details, if available

No response

Logs, status output, or debug evidence

N/A

Related issues or discussions

No response

Impact

No response

Additional context

No response

[lixmal]

Your expectation is wrong, NetBird still needs to install its nameserver to resolve peer hostnames. For you specific problem can you provide a debug bundle please? netbird debug for 1m -AU. Run your query during that 1 minute window

[Laotree]

The main problem I'm trying to report is not that NetBird installs its nameserver.

The problem is that once NetBird takes over the system DNS, normal DNS resolution also fails when there are no available NetBird nameservers (Nameservers: 0/0 Available).

In fact, the debug command itself demonstrates the problem:

upload failed: get presigned URL:
Get "https://upload.debug.netbird.io/upload-url?...":
dial tcp: lookup upload.debug.netbird.io on 100.101.192.102:53:
read udp 100.101.192.102:57694->100.101.192.102:53:
i/o timeout

So even netbird debug for 1m -AU cannot upload the debug bundle because upload.debug.netbird.io cannot be resolved through the NetBird nameserver.

I understand that NetBird needs to install its nameserver for peer hostname resolution. My concern is that when there are 0/0 available nameservers, NetBird DNS should not make the host's normal/external DNS resolution completely unavailable.

The desired behavior would be to keep NetBird's peer hostname resolution while allowing normal DNS queries to continue working when no NetBird nameserver is available.

[Laotree]

netbird.debug.2688364740.zip
here is the debug file, please feel free for more actions

[lixmal]

Thanks for the bundle, it shows what is happening, and the nameserver count is not the cause.

Something else on that host already owns port 53, so the client could not bind its resolver on any of the addresses it tries:

binding dns UDP on 100.101.192.102:53 is not available: bind: address already in use
binding dns UDP on 127.0.0.1:53 is not available: bind: address already in use
binding dns UDP on 127.0.0.153:53 is not available: bind: address already in use
load eBPF DNS forwarder, watching addr: 100.101.192.102:53, redirect to port: 5053
added DNS TCP DNAT rule: 100.101.192.102:53 -> 100.101.192.102:5053

It then falls back to port 5053 and redirects port 53 to it. That redirect only installed a firewall rule for TCP; UDP was left to an eBPF program attached to the loopback device, and on your host that program is not taking effect. So /etc/resolv.conf points at 100.101.192.102 while nothing answers there over UDP, even though the resolver itself is healthy and forwarding to your original nameserver. That is also why netbird debug for 1m -AU could not resolve the upload endpoint, and why --disable-dns=true restores resolution: it puts your original nameservers back into resolv.conf.

Two things worth confirming on that host:

• sudo ss -lunp | grep ':53', to see which daemon holds port 53
• dig @100.101.192.102 netbird.io versus dig +tcp @100.101.192.102 netbird.io. TCP should answer through the existing rule while UDP times out.

On the proposal itself: with no nameserver groups the client still needs its own resolver for peer names, and it forwards everything else to the nameservers that were in resolv.conf before it started. Your bundle shows that part working correctly, it registered 8.8.8.8 as the upstream. Disabling DNS whenever Nameservers: 0/0 would take peer name resolution away from every setup without a nameserver group, so the fix belongs in the redirect instead.

PR: #7439. It installs UDP and TCP rules for the port 53 redirect and drops the eBPF program, which could not work at all on kernels older than 5.7 or when another XDP program is attached to the loopback device.

Client builds from that PR are linked here if you would like to try one on the affected host: #7439 (comment)

[Laotree]

Thanks for the detailed analysis. I did some additional tests on the affected host, and the behavior is now quite clear.

The host is running dnsmasq, which owns both UDP and TCP port 53:

udp  UNCONN  0.0.0.0:53  users:(("dnsmasq",pid=1775222,fd=4))
tcp  LISTEN 0.0.0.0:53  users:(("dnsmasq",pid=1775222,fd=5))

So NetBird indeed cannot bind port 53 directly.

The original upstream resolver works normally:

dig @8.8.8.8 netbird.io

returns NOERROR with an A record.

NetBird is listening on the fallback port as expected:

udp  UNCONN  100.101.192.102:5053  users:(("netbird",pid=2016398,fd=36))
tcp  LISTEN  100.101.192.102:5053  users:(("netbird",pid=2016398,fd=37))

However, querying the fallback resolver directly also times out:

dig @100.101.192.102 -p 5053 netbird.io

To investigate further, I ran tcpdump while making that query. The UDP request definitely reaches the NetBird resolver:

13:42:01.932707 lo In IP 172.16.1.51.49846 > 100.101.192.102.5053: UDP, length 51
13:42:06.935541 lo In IP 172.16.1.51.49886 > 100.101.192.102.5053: UDP, length 51
13:42:11.939482 lo In IP 172.16.1.51.60170 > 100.101.192.102.5053: UDP, length 51

But there is no corresponding UDP response from 100.101.192.102:5053.

For comparison:

dig @100.101.192.102 netbird.io

times out over UDP, while the TCP query to port 53 returns:

status: REFUSED
WARNING: recursion requested but not available

So at this point my understanding is:

1. dnsmasq occupies port 53, preventing NetBird from binding it.
2. NetBird correctly falls back to port 5053.
3. NetBird is actually listening on 100.101.192.102:5053 for both UDP and TCP.
4. UDP DNS queries can reach the 5053 socket, but no UDP response is generated.
5. The original upstream DNS (8.8.8.8) works normally.
6. With --disable-dns=true, DNS resolution works again because the original nameservers are restored to /etc/resolv.conf.

This makes the UDP redirect issue in #7439 still relevant, but the direct test against port 5053 suggests there may also be something worth investigating in the resolver's UDP handling when it is running on the fallback port.

I'm happy to test a build from #7439 on this host if that would help.