Repository navigation
HA routing peers: orphaned TCP flows hang ~15 min after a switch, and the 0.83 external-chain passthrough disables the usual nftables workaround #8630
ryanmendoza-sketch
started this conversation in
Ideas & Feature Requests
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Use case
Self-hosted NetBird, two routing peers in an HA group serving the same masqueraded network route. Peers use it to reach a private cloud network (managed database endpoints, an internal load balancer).
When the route moves from routing peer A to routing peer B (failover, a restart or upgrade of A, a client restart), TCP flows that were opened through A keep sending packets, now through B. B has no conntrack entry for them. With
nf_conntrack_tcp_loose=1(the default), the mid-stream packet creates a new entry and is masqueraded with B's address. The destination does not know that 5-tuple and never answers, so the client retransmits untiltcp_retries2gives up (about 15 minutes with Linux defaults) before the application notices. We saw outages of about 16 minutes for database connection pools and log shippers after a single routing peer switch.The HA changes in 0.82 (#7689, #8192,
NB_HA_RELAYED_SWITCH_DELAY) cut down the unnecessary switches, which helps a lot. Unavoidable switches (failover, upgrades, reboots) still leave the existing flows hanging instead of failing fast.Workaround we use
On each routing peer, a small nftables table with a base chain on the forward hook. It answers with a TCP reset to any non-SYN packet from the WireGuard interface that opens a new conntrack entry:
Clients get the reset and reconnect: about 4 seconds instead of about 15 minutes in our tests.
What changes with the 0.83 client
Since #4899 the client adds passthrough rules to external input/forward chains, and the external chain monitor re-applies them whenever tables or chains change. With v0.83.1 on the routing peer (nftables backend, Ubuntu 24.04, kernel 7.0), the client inserted these two rules at the top of the chain above:
The
acceptends the evaluation of that chain, so the reset rule never matches and the workaround stops working. If we delete the rules, the monitor inserts them again within a second. We understand why #4899 exists (third-party chains with a drop policy blocking NetBird traffic). Here it has the opposite effect on a chain whose only job is to help NetBird HA routes.Request
Any of these would solve it, roughly from most to least useful for other users:
accept. In those chains the passthrough only needs to come after explicit drop/reject rules aimed at specific traffic, and inserting it at position 0 overrides them.For now we are moving the rule to a filter chain on the prerouting hook (after conntrack, limited to forwarded traffic with
fib daddr type != local), which the client does not touch today. That depends on behaviour that is not guaranteed to stay the same, so a supported option would be much better.Happy to test a build or give more details.
All reactions