Lazy connections are inherently unreliable

[Spiritreader]

Before posting

• I searched existing discussions and issues, including closed ones, and checked the relevant docs.
• I believe this is a product bug rather than a configuration or setup question.
• I can reproduce this issue, or for intermittent issues I've included trigger, frequency, and timing details below.
• I removed or anonymized sensitive data from logs, screenshots, and configuration.

Affected area

Client / Agent

Deployment type

Self-hosted - advanced/custom deployment

Operating system or environment

Linux, macOS, Windows, Android, iOS, FreeBSD, OpenWRT, Docker, Kubernetes, Synology, Browser, Other / not sure

NetBird version and upgrade status

• Client: 0.74.2
• Signal: 0.74.2
• Relay: 0.74.2
• Management: 0.74.2
• Dashboard: 2.90.3

Did this work before?

No, this never worked

Regression details

No response

Summary

When lazy connections are enabled, I'm constantly hitting the problem that clients do not connect anymore after some time, especially though after upgrading netbird.

For example if I update netbird on my local machine, I can no longer connect to for example 3-4 nodes. I then have to remote into these nodes via other means and restart them too, because connections will never be re-established. Netbird shows either "Connecting" or "Relayed" on these peers, but there is no active connection, only a connection update, no wireguard handshake

Current behavior

Lazy connections under some conditions will never be re-established.

Expected behavior

Lazy connections allow peers to re-connect if the connection is dropped.

Steps to reproduce

It's a bit hard to reproduce since it's not eniteryl deterministic and doesn't always happen, but

1. Leave netbird running for a couple of days with lazy connections
2. upgrade a client to a newer version
3. observe some connections never re-establishing

It also happens sometimes when just waking a machine from sleep.

Environment and topology

Peers are on different networks, connected via the internet, in IPv4 and IPv6 P2P and relay connections.

Self-hosted details, if available

Deployment version is via docker compose, legacy split setup

Logs, status output, or debug evidence

Peers detail:
 suc-docker.netbird.anon:
  NetBird IP: 100.64.192.207
  NetBird IPv6: fd8c:7e8f:50bf:1ab4:ef6c:c130:8062:83ce
  Public key: anonymized
  Status: Connecting
  -- detail --
  Connection type: -
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 minute, 36 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

Related issues or discussions

No response

Impact

Connection between peers does not work when lazy connections are enabled

Additional context

My hunch is that it has something to do with peer activation. I tried to dig around a bit and it appears that there is nothing that actually guarantees that a peer where foundequals true is actually a peer that does not need to be activated.

if found := e.lazyConnMgr.ActivatePeer(conn.GetKey()); found {
		if err := conn.Open(ctx); err != nil {
			conn.Log.Errorf("failed to open connection: %v", err)
		}
	}

netbird/client/internal/engine.go

Line 1839 in 3aa6c02

e.connMgr.ActivatePeer(e.ctx, conn)

netbird/client/internal/conn_mgr.go

Line 241 in 3aa6c02

func (e *ConnMgr) ActivatePeer(ctx context.Context, conn *peer.Conn) {

But I didn't have the time to compile clients myself and test this.