Self-hosted (combined image + reverse proxy): all peers stuck on Relayed, RemoteCandidate always 0

[ilvoen]

Before posting

• I searched existing discussions and issues, including closed ones, and checked the relevant docs.
• I believe this is a product bug rather than a configuration or setup question.
• I can reproduce this issue, or for intermittent issues I've included trigger, frequency, and timing details below.
• I removed or anonymized sensitive data from logs, screenshots, and configuration.

Affected area

Peer connectivity, Relay / Signal / NAT traversal, Self-hosting / Deployment

Deployment type

Self-hosted - quickstart script

Operating system or environment

Linux, Android, iOS, Docker

NetBird version and upgrade status

• Netbird Combined Server 0.74.2 (and 0.74.0)
• Linux Client 0.74.2
• Docker Client 0.74.2
• Android Client 0.5.0
• IOS & iPadOS Client 0.3.1

Did this work before?

Not sure

Regression details

No response

Summary

All peer connections are permanently stock on Relayed connection type across all peers which are connected to the self-hosted instance. RemoteCandidatestays at 0for every peer at every routing / exit node, on two independent hosts / providers, despite STUN working correctly and confirmed reachable. Traffic works (and functions correctly), but true P2P/ICE never establishes for any peer pair.

Current behavior

P2P Connections won‘t establish.

Expected behavior

P2P connections between the peers.

Steps to reproduce

1. Deploy combined NetBird server (v0.74.2) behind Traefik v3, with a dedicated router forwarding /signalexchange.SignalExchange/* and /management.ManagementService/* via h2c (gRPC over cleartext HTTP/2).

2. Connect several peers: at least one routing/exit-node peer (netbird-client Docker image, bridge network mode, public IP, no NAT) plus regular peers on independent networks (mobile data, home WiFi).

3. Confirm Lazy Connections is off ("Lazy connection: false" in netbird status --detail).

4. Restart the netbird-server container. Peers briefly show P2P/Direct right after restart.

5. Wait a few minutes to hours, no further restarts.

6. Run netbird status --detail: all peers now show Relayed, ICE candidate (Local/Remote): -/-. Debug state dump shows RemoteCandidate: 0 for every peer, RemoteAnswer: 1, SentOffer climbing (retry loop).

7. Manually reconnect an affected peer (netbird down && netbird up) — it reconnects fine but stays Relayed, RemoteCandidate still 0.

8. STUN itself confirmed reachable/working throughout (manual STUN binding test succeeds). Result: once relayed, peers never recover P2P. Reproduced independently on two exit-node hosts on different providers/networks.

Environment and topology

• NetBird version: 0.74.2 (server and all clients consistently on this version)
• Deployment: self-hosted, combined image (netbirdio/netbird-server, management + signal + relay + STUN in one container), SQLite backend
• Reverse proxy: Traefik v3, TLS-terminating, with:
  • Dedicated router for /signalexchange.SignalExchange/* and /management.ManagementService/* using h2c scheme to the backend (gRPC over cleartext HTTP/2)
  • Separate router for /relay, /ws-proxy/, /api, /oauth2
• Topology: 2 independent exit/routing-peer hosts on different providers, both running the netbird-client Docker image in bridge network mode (not host networking, due to Traefik occupying host ports 80/443)
• Other peers: iOS, iPadOS, Android (GrapheneOS) mobile clients on various independent networks (home WiFi, mobile data)

Self-hosted details, if available

No response

Logs, status output, or debug evidence

N/A

Related issues or discussions

No response

Impact

No response

Additional context

Symptoms

netbird status —-detail on both exit-node containers shows, for every peer, every time:

Connection type: Relayed
ICE candidate (Local/Remote): -/-
ICE candidate endpoints (Local/Remote): -/-

Internal peer state dump (client/internal/internal/state_dump.go) confirms this is not just a display issue:

Status: Connected, SentOffer: 60, RemoteOffer: 2, RemoteAnswer: 1
RemoteCandidate: 0, P2PConnected: 0, SwitchToRelay:0,
WGCheckSucess: 65, RelayConnected: 3, LocalProxies: 3

RemoteCandidate: 0 for every single peer, constantly, on both independent exit-node hosts. SentOffer climbs very high (60+) for peers with long uptime, indicating a stuck retry loop rather than a one-time negotiation failure.

Traffic is fully functional over relay (WireGuard handshakes succeed, exit-node routing works, internet access via 0.0.0.0/0 route works), so this is purely an ICE/P2P establishment issue, not a connectivity outage.

What has been ruled out

• STUN reachability: Confirmed working - manual STUN binding request from an external host returns a correct XORMappedAddressresponse with the server‘s public IP. Server logs also show real STUN Binding requests/responses from multiple peer IPs during normal operation.
• Host firewall / NAT / port mapping: WireGuard UDP port (51820) is now explicitly published via Docker and allowed in UFW on the primary host. No change in behavior after this fix.
• NAT type as sole causes: Two independent exit-node hosts, both with public IPs (not behind NAT themselves), show identical RemoteCandidate: 0behaviour - this routes out a NAT-type/CGNAT explanation for at least those two peers.
• Reverse proxy security headers: Removed a global secHeadermiddleware from Traefik entrypoint (potential interference with the long-lived gRPC bidirectional stream) - no change.
• Lazy Connections: Explicitly disabled in the dashboard; netbird status —detailconfirms Lazy connection: falseon all peers. No change.
• Version mismatch: Server and all clients confirmed on latest versions.
• TrustedHTTPProxies misconfiguration: An earlier, separate issue (incorrect trusted proxy CIDR causing ALL peers to be relayed due to spoofed client IPs from Traefik) was found and fixed. After the fix, peers briefly archive P2P connections immediately after a fresh server restart, but degrade back to Relayed / RemoteCandidate: 0within minutes to hours, and stay that way through subsequent reconnects - suggesting this is a separate, second issue layered on top of the (now fixed) proxy-trust bug.

Observed pattern

• Immediately after a fresh netbird-server restart, some peers briefly show P2P / Direct: true.
• Within minutes to a few hours, all peers degrade to Relayed and stay there - including after individual peer reconnects (netbird down && netbird up), which do not recover P2P.
• This has been reproduced independently on two exit-node hosts (different provider, different networks, different public IPs), both in Docker bridge mode.

Question

Is there a known issue in 0.74.x (or a documented interaction with combined-image+reverse-proxy deployments) where ICE candidate messages specifically fail to propagate the Signal exchange after some period of uptime, while Offer/answer messages and STUN itself continue to work?

[PizzaLovingNerd]

Thanks for the detailed write-up!

I’d focus on the Signal path first. RemoteCandidate: 0 usually means the peers are not receiving ICE candidates from each other. That is different from “STUN works but the direct path fails” - in that case I’d expect candidates to exist, then ICE checks to fail.

A few concrete things I’d test:

1. Bypass Traefik for one test

Expose the NetBird server directly on a temporary test hostname/port, or run a temporary clean quickstart instance without Traefik, then enroll two test peers against that URL.

If P2P starts working there, the issue is almost certainly in the reverse proxy path.

2. Check the Traefik routing split

For the combined server, make sure the gRPC paths go to the backend with h2c:

  - traefik.http.routers.netbird-grpc.rule=Host(`netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`))
  - traefik.http.services.netbird-grpc.loadbalancer.server.scheme=h2c

And keep the normal HTTP/WebSocket paths separate:

  - traefik.http.routers.netbird-http.rule=Host(`netbird.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/relay`) || PathPrefix(`/ws-proxy/`) || PathPrefix(`/oauth2`) || PathPrefix(`/`))

The exact labels may differ, but the important part is that Signal gRPC is not accidentally handled by the plain HTTP/WebSocket router.

3. Confirm STUN is exposed outside Traefik

From another machine:

nc -uvz netbird.example.com 3478

Or use any STUN test tool against:
netbird.example.com:3478

Traefik usually should not be in this path. UDP 3478 needs to be published on the host and allowed by the host/cloud firewall.

4. Check the Docker peer’s actual WireGuard port

Inside the Docker client:

netbird status --detail

Look for the WireGuard listen port. Then confirm that exact UDP port is published and allowed. Publishing 51820/udp only helps if the client is actually listening on 51820.

As a quick isolation test, run one routing peer with host networking:
network_mode: host

If P2P starts working with host networking, the problem is likely Docker port/NAT behavior on the routing peer, not NetBird Signal itself.

5. Capture one failed renegotiation

On one affected client:

netbird down
netbird up
netbird status --detail

Then grab logs while SentOffer increases but RemoteCandidate stays 0:

journalctl -u netbird --since "10 minutes ago" --no-pager

For Docker clients:

docker logs netbird-client --since 10m

And on the server:

docker logs netbird-server --since 10m

I’d share redacted Traefik config, config.yaml, and those client/server logs. The interesting bit is whether candidate messages are being sent by one peer but never received by the other, or whether they are not being generated/sent at all.

The restart behavior is the odd part. If direct connections briefly work after restarting netbird-server and then all pairs decay to relayed with RemoteCandidate: 0, that sounds worth investigating, but I’d first rule out the long-lived Signal gRPC stream through Traefik and the Docker routing peer’s UDP exposure.

[ilvoen]

Hei thanks for the detailed answer.

I‘m really at home, that’s why I haven‘t been able to answer.
Recently I tested everything again.
What I noticed:

1. In my network at home, most problems seem to disappear. Most peers started to connect over P2P most of the time. This lets me to believe, that the network where I was was the problem and blocked something.
2. The mobile devices over carrier nwtrwork still only get Relayed connection and no ICE, but this is to expected I think.
3. The netbird client run in Docker is not able to get an P2P connection and ICE at all, when not using network_mode: host. That‘s while hard configuring the WireGuard port and mapping it in docker compose. Also Opened it on all Firewalls.

[sticklizard]

I'm hitting the same symptom on a very similar deployment, with one additional root cause traced and fixed — but the core issue you describe persists even after that fix.

Environment:

NetBird Combined Server v0.74.7, self-hosted, SQLite backend
Windows clients 0.75.1 (both updated during troubleshooting; identical behavior on 0.73.2)
Reverse proxy: Traefik v3.6, TLS-terminating, same topology as described above — dedicated router for /signalexchange.SignalExchange/* and /management.ManagementService/* via h2c, separate router for /relay, /ws-proxy/, /api, /oauth2
Two peers on independent networks (cable ISP, and a cellular 5G home internet connection), neither behind CGNAT — confirmed directly by comparing each router's own WAN IP against an external "what's my IP" check; both showed a stable, directly-assigned public IP with no mismatch
reverseProxy.trustedHTTPProxies correctly set to the Traefik container's static IP (matches the documented example exactly)

A distinct bug I found and fixed along the way:
A host-level iptables POSTROUTING rule — not one of Docker's own auto-generated rules, but a pre-existing unconditional SNAT --to-source catch-all with no source/destination/interface restriction — was rewriting the source address of externally-arriving STUN requests as they were forwarded into the container via the Docker bridge network. This happened regardless of userland-proxy setting (tested both ways) and regardless of which host IP the port was published on. The result: the STUN server's XORMappedAddress response reported the wrong (internal bridge-facing) address instead of the real client's public IP — confirmed via tcpdump at the packet level and via the server's own debug logs.

Fixed with a targeted exemption inserted ahead of that catch-all rule, scoped to the container's published port. After the fix, confirmed via both tcpdump and docker compose logs grep for xormapped that STUN now returns the correct, real public IP/port for external clients — verified independently with a browser-based WebRTC Trickle-ICE test (bypassing the NetBird client entirely) as well as by the netbird-server debug log during real client traffic.

Despite that fix, the original symptom is unchanged:

netbird status -d on both peers: Connection type: Relayed, ICE candidate (Local/Remote): -/-, even immediately after a fresh netbird down && netbird up on both peers simultaneously
This is with STUN now confirmed correct and trustedHTTPProxies confirmed correct — the two things I'd expect to be the most likely causes
Same pattern you describe: works fine over relay, WireGuard handshakes succeed, just never establishes P2P/ICE

Question: given that STUN reachability/correctness and reverse-proxy trust are both ruled out (in my case verified at the packet level, not just inferred), is there a known gap in how ICE candidate messages propagate through the Signal exchange specifically for combined-image + external-reverse-proxy deployments? Happy to grab a debug bundle or any specific logs if that would help narrow it down.