Skip to content

NetBird reports "Connected" but brev SSH gateway path is stale after a network interruption (requires manual systemctl restart netbird) #426

Description

@theFong

Summary

After a transient network interruption on a brev-managed instance, remote access via brev shell <instance> / ssh <instance> (through the brev gateway) fails with:

kex_exchange_identification: Connection closed by remote host
Connection closed by <gateway-ip> port <port>

…even though the NetBird agent on the instance reports a fully healthy connection (Management: Connected, Signal: Connected, all peers connected, service active + enabled). The gateway's forwarding/registration for the instance appears to go stale after the partition and is not refreshed on reconnect. A manual sudo systemctl restart netbird on the instance re-registers it and restores access immediately.

The key issue: the agent's self-reported health passes while the end-to-end path through the gateway is dead, so nothing triggers a re-registration.

Environment

  • brev-managed GPU instance (Ubuntu), NetBird daemon v0.74.7
  • NetBird service active and enabled (auto-starts on boot)

Steps to reproduce

  1. Instance is up and reachable via brev shell / ssh <instance>.
  2. Cause a transient network interruption — e.g. an upstream switch/power blip or a link flap on the primary NIC (in our case, briefly de-powering a shared network switch).
  3. Instance OS recovers and is reachable on its LAN / other overlays; NetBird reconnects on its own.
  4. netbird status on the instance reports healthy: Management: Connected, Signal: Connected, Peers count: N/N Connected.
  5. But ssh <instance> through the brev gateway still fails with Connection closed by remote host.
  6. sudo systemctl restart netbird → access restored immediately.

Root-cause hypothesis

The NetBird control plane (Management/Signal) reconnects after the partition, so the agent self-reports "Connected," but the gateway's port-forward / peer registration to the instance is left in a stale, half-open state. Because the agent's own health check passes, no re-registration is triggered and the stale mapping persists until the daemon is manually restarted.

Possible solutions

  1. Refresh the gateway port-forward on peer reconnect (gateway/control-plane side). When a peer's tunnel re-establishes after a partition, refresh the forwarding mapping instead of assuming the prior one is still valid.
  2. End-to-end liveness detection, not just control-plane status. Have the agent/CLI periodically verify reachability through the gateway (or to a known peer over the overlay) and trigger a re-register/reconnect on failure — rather than relying solely on Management/Signal: Connected, which can be green while the path is dead.
  3. Restart/re-register NetBird on network link-up. Ship or document a NetworkManager/networkd dispatcher hook that nudges NetBird when the primary interface transitions down→up, so a link flap forces a fresh registration.
  4. More aggressive reconnection/backoff after a partition, and confirm WireGuard persistent-keepalive is set on NAT'd peers so mappings survive / re-punch quickly.

Current workaround

sudo systemctl restart netbird on the affected instance. (Reachable in the meantime via a direct/LAN path or another overlay to run the restart.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions