NetBird update breaks routing peer connection #6639
Before posting
TopicSelf-hosting Deployment typeSelf-hosted - quickstart script Operating system or environmentLinux, Docker NetBird versionclient 0.73.0, management 0.74.0 QuestionWe have a self-hosted NetBird setup with access to internal resources (including the NetBird vm) through a routing peer. Previous updates worked flawlessly. The latest update from 0.73.0 to 0.74.0 seemingly broke the routing peer's connection despite it showing as online in the dashboard. I used a secondary backup VPN connection to access the routing peer and force a reconnect (and updates its client to 0.74.0 as well in the process). Without that second VPN I would've been completely locked out and unable to restore access. My questions are:
Desired outcomeNo response What have you tried?No response Relevant environment detailsNo response Logs or outputNo response Additional contextNo response |
Replies: 8 comments 6 replies
|
I encountered the same issue — in the new v0.74.0 version, lazy connection became enabled by default. This was causing problems with connections going through the routing peer.
|
|
I can also confirm this error. After the update, all clients are displayed as However, SSH connections can no longer be established, and services accessed via the Netbird reverse proxy are no longer reachable (Error 502). The Netbird clients (version 0.74.0) work fine when used with the Netbird server stack 0.73.2, Management, and Dashboard v2.90.2. The errors only occur after updating the server stack to Management version v0.74.0. The workaround suggested by @lnyamsurenn-rgb might certainly work, but unfortunately it’s not practical with 300 clients. ;-) P.S. Still, I'd like to thank the developers at Netbird for this excellent software. |
|
Can confirm the same thing @securebitsorg is describing. Did a bit of isolation testing on my end that might add some useful detail. My setup: self-hosted combined server (management+signal+relay+STUN) + reverse-proxy, both normally pinned together. Upgraded both to 0.74.0, reverse-proxy services started returning 502, and peers still showed "connected"/online in the dashboard the whole time, so it wasn't visible as a health issue on the surface. Logs from the reverse-proxy peer: This happened even with What I found when testing:
So similar to what @securebitsorg found, it seems tied to the server side rather than the client. On the lazy-connections idea from @lnyamsurenn-rgb, the release notes mention that's only enabled by default for new accounts, so probably not it for existing setups, but not 100% sure. For what it's worth, one thing I noticed browsing the 0.74.0 changelog is PR #6626, which changes the order WG watcher does its initial handshake read relative to the peer being added to the interface. My logs also show Happy to share more logs if it'd help track this down. |
|
I am having similar odd issues with my iphone (as a client) connecting to my VPS netbird server after upgrading to 0.74. The connection type states “Relay” with no ICE candidate. I have an exit route through my VPS and a route for my local network. I am not able to access my local network when I am remote, however, can still access the internet via the VPS exit route. Everything was working fine until upgrading to 0.74. |
|
Having Lazy Connection enabled by default with the 0.74.0 server update is causing very frequent disconnections on connected peers. Also, a large number of "Server failed" errors are starting to occur on the DNS side. There are no issues on gateway machines with client version 73.2, only the connections on the 74.0 version I updated are causing a lot of problems. Everything was working fine until switching to version 0.74. |
|
Hey folks, can you validate if 0.74.1 fixes the issue? We found a bug where relays were not sent to peers and fixed in this version |
|
I am experiencing the exact same "502 Bad Gateway" issues mentioned in this thread since the 0.74.x updates. I'm just a home-labber playing around with some Raspberry Pis, so I'm not a networking expert, but I want to share a critical piece of debugging information that I found. It might help you guys pinpoint the exact root cause! Since I am using the NetBird Managed Cloud, I cannot simply downgrade the management server to 0.73.2 as suggested above. My local clients (Raspberry Pis, Linux/arm64) are running on v0.74.4. My Setup (Target Type: Peer):
The Debugging & The "Smoking Gun": However, I noticed a very specific pattern regarding which services fail:
Conclusion: From my understanding, Stirling PDF relies on standard, stateless HTTP requests. ntfy and Nextcloud, on the other hand, heavily rely on WebSockets and Server-Sent Events (SSE) for push notifications (long-lived connections). Could it be that the current NetBird L7 Proxy implementation (since 0.74.x) struggles to handle WebSockets or long-lived connections? It seems to time out and corrupt the routing path for those specific services, while standard stateless services continue to pass through without issue. I’m still very new to this field and learning every day, so I hope my amateur sleuthing helps the team to isolate the bug! Let me know if you need any further logs. |
|
I hit the same issue after updating my Raspberry Pi routing-peer clients to Peers remained shown as connected, but all reverse-proxy services returned 502/timeouts and no proxy traffic reached either routing peer. The recovery that fixed it was:
After that, my reverse-proxy services recovered immediately. |
Hey folks, can you validate if 0.74.1 fixes the issue? We found a bug where relays were not sent to peers and fixed in this version