Catch-all NameServerGroup shadows fallback handler on embedded proxy and routing peers

[renne]

NetBird version and upgrade status

• Client: 0.76.0
• Management (netbird-server): v24.04 image (built 2026-07-29, includes PR [management] Force routing-peer DNS resolution for reverse-proxy domain targets #6872 by @lixmal)
• Reverse Proxy: netbirdio/reverse-proxy:latest (pulled 2026-07-29)
• Routing peer: netbirdio/netbird:latest v0.76.0
• Upgrade status: reproduced on current version

Regression details

Not a regression. This never worked — the catch-all NameServerGroup has always been incompatible with reverse-proxy backend resolution on embedded proxy peers. We kept the catch-all NameServerGroup disabled since discovering this.

Summary

A primary (catch-all) NameServerGroup at PriorityDefault (1) permanently shadows the fallback handler at PriorityFallback (-100) on embedded proxy peers and routing peers. When the catch-all NameServerGroup is enabled, NXDOMAIN from public upstream resolvers terminates the DNS handler chain before it reaches the fallback handler (original system DNS, e.g. Docker's embedded DNS at 127.0.0.11). This prevents reverse proxy peers from resolving backend hostnames (e.g. *.internal container names) that are only resolvable via the fallback, causing 502 Bad Gateway for all reverse-proxied services.

Current behavior

When a catch-all NameServerGroup (primary=true, no match domains) is enabled and distributed to peers running embedded NetBird clients (reverse proxy containers) or routing peers:

1. The embedded NB client's handler chain registers the catch-all at PriorityDefault (1) with pattern . (root zone)
2. The fallback handler (original nameservers from /etc/resolv.conf, e.g. Docker DNS 127.0.0.11) registers at PriorityFallback (-100), also with pattern .
3. When the proxy resolves a backend hostname like vault.internal:
   • No handler at PriorityDNSRoute (100) matches — the DnsInterceptor is only registered on client peers, not on embedded proxy peers or the routing peer itself
   • PriorityLocal (75) doesn't match — *.internal is not a custom/synthesized zone
   • PriorityUpstream (50) doesn't match — no domain-matched NameServerGroup for internal
   • PriorityDefault (1) matches (catch-all, pattern .) → queries public DNS → returns NXDOMAIN
   • The upstream handler clears the Zero bit (rm.MsgHdr.Zero = false) so NXDOMAIN is terminal
   • PriorityFallback (-100) is never reached
4. The proxy cannot resolve the backend → 502 Bad Gateway

Evidence from proxy logs

docker-host proxy:
  err=dial: lookup backend-app.internal. on 100.106.255.254: no such host

VPS proxy:
  err=dial: lookup vault.internal. on 100.106.255.254: no such host

Both proxies use 100.106.255.254 (embedded NB DNS) as their resolver. The embedded NB client receives the catch-all NameServerGroup automatically — embedded proxy peers are not members of any user-assigned group, but the management server pushes all enabled NSGs to them regardless of group membership (by design, so they can resolve any backend hostname).

Evidence from routing peer handler chain (debug logs)

registering handler Upstream [public DNS resolvers] with priority 1 for [.]
registering handler Upstream [127.0.0.11:53] with priority -100 for [.]

No handler at priority 100 (DnsInterceptor) exists on the routing peer. Before the catch-all is enabled, *.internal falls through to the fallback at -100 (Docker DNS) and resolves correctly. After the catch-all is enabled, NXDOMAIN at priority 1 terminates the chain.

Test results

With catch-all NameServerGroup disabled (current production state):

vault.internal @client-NB-DNS  → 172.0.5.10 ✓ (via DnsInterceptor → forwarder → fallback loop → Docker DNS)
vault.example.com              → 200 ✓
example.com                    → 302 ✓

With catch-all NameServerGroup enabled:

vault.internal @client-NB-DNS  → NXDOMAIN ✗ (catch-all terminates before fallback)
vault.example.com              → 502 ✗ (proxy can't resolve vault.internal backend)
dms.example.com                → 502 ✗ (proxy can't resolve paperless.internal backend)
family.example.com             → 502 ✗ (proxy can't resolve family.internal backend)
example.com                    → 302 ✓ (public service, backend still cached/reachable)

Expected behavior

A catch-all (primary) NameServerGroup should not permanently prevent the fallback handler from resolving domains that the catch-all cannot answer authoritatively. Specifically:

• Domains that are only resolvable via the original system DNS (e.g. Docker container names, Kubernetes service names, .local mDNS) should fall through the catch-all to the fallback handler when the catch-all returns NXDOMAIN
• OR: the fallback handler should take priority over the catch-all for domains not matched by any configured zone, NameServerGroup, or DNS route
• OR: a mechanism should exist to exclude specific domains or domain suffixes from the catch-all NameServerGroup

Steps to reproduce

1. Set up a self-hosted NetBird deployment with reverse proxy (netbirdio/reverse-proxy) running as a Docker container with an embedded NetBird client
2. Configure reverse-proxy services with domain targets (e.g. vault.internal) that are resolvable only via Docker's embedded DNS (127.0.0.11)
3. Ensure RoutingPeerDNSResolutionEnabled is enabled (account setting) and the routing peer's DNS forwarder is running
4. Create a primary (catch-all) NameServerGroup (no match domains) with public DNS servers, assigned to any group (e.g. "All")
5. Wait for config propagation — the management server automatically pushes all enabled NSGs to embedded proxy peers regardless of group membership
6. Access any reverse-proxied service

Result: 502 Bad Gateway — the proxy cannot resolve backend hostnames.

7. Disable the catch-all NameServerGroup and wait for propagation

Result: All services work again — backends resolve via the fallback handler (Docker DNS).

Environment and topology

• Peer A (client): Linux client, NetBird 0.76.0, kernel WireGuard.
• Peer B (routing peer): Docker container on docker host, NetBird 0.76.0. Routing peer for the Docker network with domain resources (*.internal). DNS forwarder on port 22054.
• Peer C (docker-host proxy): netbirdio/reverse-proxy:latest with embedded NB client. NB_PROXY_PRIVATE=false. Serves public services. Embedded peer (proxy_meta_embedded=1) — receives all enabled NSGs automatically, not via group membership.
• Peer D (VPS proxy): netbirdio/reverse-proxy:latest with embedded NB client. NB_PROXY_PRIVATE=true. Serves all services. Embedded peer (proxy_meta_embedded=1) — receives all enabled NSGs automatically.
• Management server: netbirdio/netbird-server:latest on VPS (public IP). v24.04 image built 2026-07-29.
• Networks: Docker network (172.0.5.0/24) — Docker network with all backend containers. NetBird overlay 100.106.0.0/16.
• DNS zones: Custom zone example.com with wildcard *.example.com → 192.168.178.64. Synthesized private-service zones for private=1 services → proxy NB IPs.
• NameServerGroups: Catch-all (public DNS resolvers, assigned to an internal group for regular peers; automatically pushed to embedded proxy peers regardless of group membership; disabled in production), LAN resolver (group=All, enabled).
• No other VPN software. No firewall or endpoint security software relevant.

Self-hosted details

• Deployment method: Docker Compose (custom)
• Management/signal/relay: netbirdio/netbird-server:latest (combined binary, v24.04 image)
• Reverse proxy: netbirdio/reverse-proxy:latest (two instances: docker-host NB_PROXY_PRIVATE=false, VPS NB_PROXY_PRIVATE=true)
• IdP: self-hosted OIDC (Nextcloud AIO)
• STUN/TURN: embedded in combined server
• Routing peer DNS forwarder: confirmed running on port 22054 with 11 domain entries

Logs, status output, or debug evidence

Routing peer handler chain (debug logs, catch-all enabled)

registering handler MgmtCacheResolver with priority 150 for [netbird.example.com]
registering handler LocalResolver with priority 75 for [example.com.]
registering handler Upstream [LAN resolver NameServerGroup] with priority 50 for [lan]
registering handler Upstream [public DNS] with priority 1 for [.]
registering handler Upstream [127.0.0.11:53] with priority -100 for [.]

Note: No handler at priority 100 (DnsInterceptor). The DnsInterceptor is only registered on client peers, not on the routing peer or embedded proxy peers.

Proxy error logs (catch-all enabled)

docker-host proxy:
  proxy error: status=502 err=dial: lookup backend-app.internal. on 100.106.255.254: no such host

VPS proxy:
  proxy error: status=502 err=dial: lookup vault.internal. on 100.106.255.254: no such host

DNS resolution comparison

Catch-all DISABLED:
  vault.internal @routing-peer → 172.0.5.10 (NOERROR, via fallback → Docker DNS)
  vault.internal @client-NB    → 172.0.5.10 (NOERROR, via DnsInterceptor → forwarder → fallback loop → Docker DNS)

Catch-all ENABLED:
  vault.internal @routing-peer → NXDOMAIN (via catch-all → public DNS → NXDOMAIN, terminal)
  vault.internal @client-NB    → NXDOMAIN (forwarder loops back, gets NXDOMAIN from catch-all)

Related issues or discussions

• Upstream nameserver NXDOMAIN overrides DNS-based network route resolution #5647 (open): Upstream nameserver NXDOMAIN overrides DNS-based network route resolution. Same family (catch-all NXDOMAIN terminates chain), but describes the problem on CLIENT peers where a DnsInterceptor at priority 100 should handle the domain. Maintainer (lixmal) states routing peer DNS "doesn't have inherent priority" and suggests domain-matched NameServerGroups or custom DNS zones. This is different: our problem is on PROXY and ROUTING peers where there is no DnsInterceptor at all, and the fallback at -100 is the only resolver for backend hostnames.
• [management] Force routing-peer DNS resolution for reverse-proxy domain targets #6872 (merged 2026-07-28): Force routing-peer DNS resolution for reverse-proxy domain targets. Forces RoutingPeerDNSResolutionEnabled on for embedded proxy peers and routing peers involved in reverse-proxy services. Ensures the DNS forwarder starts, but does NOT address the catch-all shadowing the fallback — the forwarder still needs the fallback to resolve via Docker DNS.
• Nameservers only work if "Match Domains" is specified #5569 (open): Nameservers only work if "Match Domains" is specified. Related complaint about catch-all NameServerGroup behavior on mobile clients.

Impact

• Affected users/peers: All self-hosted users running reverse proxy with Docker/Kubernetes backend hostnames and a catch-all NameServerGroup. This is a common architecture for self-hosters.
• Business/production impact: Catch-all NameServerGroup is a natural configuration for general DNS resolution (non-NetBird domains). The incompatibility with reverse proxy backend resolution means users must choose between general DNS and working reverse proxy — they cannot have both.
• Workaround available: Keep the catch-all NameServerGroup disabled and rely on the fallback handler (system DNS) for all non-zone/non-NameServerGroup domains. There is no way to exclude embedded proxy peers from receiving a catch-all NameServerGroup — they receive all enabled NSGs automatically regardless of group membership, so the only option is to disable the NameServerGroup account-wide.

Additional context

Root cause analysis (from source code)

The DNS handler chain in client/internal/dns/handler_chain.go dispatches queries in priority order. The shouldContinue fallthrough mechanism uses NXDOMAIN-with-Zero-bit-set as a "try next handler" signal. However, the upstream handler (client/internal/dns/upstream.go) actively clears the Zero bit:

// Clear Zero bit from external responses to prevent upstream servers from
// manipulating our internal fallthrough signaling mechanism
rm.MsgHdr.Zero = false

This means NXDOMAIN from a catch-all NameServerGroup is always terminal — the chain stops and the fallback handler is never reached. This is by design (to prevent external DNS servers from triggering internal fallthrough), but it has the side effect of permanently shadowing the fallback handler for all domains the catch-all doesn't know about.

The priority gap

PriorityDNSRoute  = 100   // DnsInterceptor (client peers only)
PriorityLocal     = 75    // Custom/synthesized zones
PriorityUpstream  = 50    // Domain-matched NameServerGroups
PriorityDefault   = 1     // Catch-all NameServerGroup (primary=true)
PriorityFallback  = -100  // Original system DNS (Docker DNS, etc.)

The 99-priority gap between PriorityDefault (1) and PriorityFallback (-100) is where the catch-all sits. Any domain not matched by a higher-priority handler hits the catch-all, gets NXDOMAIN, and terminates — the fallback is permanently shadowed.

On client peers this is fine (DnsInterceptor at 100 handles routing-peer domains before the catch-all). On embedded proxy peers and routing peers there is no DnsInterceptor, so the fallback at -100 is the only resolver for backend hostnames, and the catch-all kills it.

Proposed solution

Option A: NXDOMAIN fallthrough for root-zone upstream handlers (minimal change)

When a root-zone (pattern .) upstream handler at PriorityDefault returns NXDOMAIN, set shouldContinue = true instead of treating it as terminal. This would allow the chain to fall through to the fallback handler at PriorityFallback.

Risk: This changes the semantics of NXDOMAIN for all catch-all NameServerGroups. If the catch-all is intentionally authoritative for NXDOMAIN (e.g. for DNS filtering/blocking), fallthrough would undermine that. Mitigation: only apply fallthrough when a PriorityFallback handler exists for the same pattern, or make the behavior opt-in via an NameServerGroup flag (e.g. "fallthrough on NXDOMAIN").

Option B: Register fallback handler above catch-all for embedded/routing peers

On embedded proxy peers and routing peers, register the fallback handler at a priority above PriorityDefault (e.g. PriorityUpstream - 1 = 49) instead of PriorityFallback (-100). This ensures the fallback is reached before the catch-all.

Risk: This changes the fallback's position in the chain for all peers, potentially affecting other use cases where the catch-all should take priority over system DNS. Mitigation: only adjust the fallback priority for peers that have no DnsInterceptor (embedded/routing peers), or make it conditional on the peer type.

Option C: Domain exclusion list for catch-all NameServerGroups

Allow catch-all NameServerGroups to specify a list of excluded domains/suffixes (e.g. internal, .local, .svc.cluster.local). Queries for excluded domains would skip the catch-all and fall through to the fallback.

Risk: Adds configuration complexity. Mitigation: This is the most flexible approach and aligns with the existing "match domains" feature — it's the inverse (exclude domains).

Option D: Register DnsInterceptor on routing peers for their own domain resources

Currently the DnsInterceptor is only registered on client peers. If it were also registered on routing peers for their own domain resources (at PriorityDNSRoute = 100), the routing peer's DNS forwarder would intercept *.internal before the catch-all, breaking the loop and resolving directly.

Risk: This changes the routing peer's DNS architecture. The forwarder would need to resolve via Docker DNS directly (127.0.0.11) instead of looping through the NB DNS chain. Mitigation: The forwarder already uses net.DefaultResolver which reads /etc/resolv.conf; if /etc/resolv.conf pointed to the original Docker DNS instead of the NB DNS, the loop would be broken.

Compounding factor: the forwarder loop on the routing peer

Any solution must account for a secondary problem on the routing peer itself. NetBird rewrites the routing peer's /etc/resolv.conf from the original Docker DNS (127.0.0.11) to its own NB DNS address (the routing peer's NetBird IP). The DNS forwarder uses net.DefaultResolver, which reads /etc/resolv.conf — so the forwarder queries its own NB DNS on port 53. On the routing peer, the NB DNS chain has no DnsInterceptor (priority 100) for *.internal, so the query hits the catch-all (priority 1) or falls through to the fallback (priority -100, which is the original Docker DNS). Before the catch-all is enabled, this loop works: the forwarder → NB DNS → fallback → Docker DNS → resolves. With the catch-all enabled, the loop produces NXDOMAIN: forwarder → NB DNS → catch-all → public DNS → NXDOMAIN → terminal. Solutions that only fix the priority ordering on proxy peers (Options A-C) would not fix the routing peer unless they also ensure the catch-all does not terminate the chain before the fallback is reached.