eBPF WG proxy: injected packets dropped after ICE handover on Kubernetes nodes (EPERM)

[Spiritreader]

Before posting

• I searched existing discussions and issues, including closed ones, and checked the relevant docs.
• I believe this is a product bug rather than a configuration or setup question.
• I can reproduce this issue, or for intermittent issues I've included trigger, frequency, and timing details below.
• I removed or anonymized sensitive data from logs, screenshots, and configuration.

Affected area

Client / Agent

Deployment type

Self-hosted - advanced/custom deployment

Operating system or environment

Linux, Kubernetes

NetBird version and upgrade status

NetBird client	0.76.3, as a Talos system extension (peers seen on 0.76.3, 0.77.0, 0.78.1)
OS	Talos Linux 1.14.0, kernel 6.18.48-talos
Kubernetes	1.36.3, kube-proxy in nftables mode
CNI	Cilium, tunnel routing mode, kubeProxyReplacement: false
Nodes	6, two sites; the remote site is behind residential NAT and relies on relay

Did this work before?

Not sure

Regression details

No response

Summary

Disclaimer: I'm only somewhat confident in this bug report's causality as I have done this with Claude. At the very least the issue exists in this k8s scenario and the cause makes sense to me looking at the logs and iptables rules.

Whenever a connection drops to relay, the eBPF injection for the relay fails:

ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn:
     write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted

Rate on two nodes, sampled over a 6.7 minute window (2000 log lines each): 1203 and
1415 occurrences.

Happens every time when a peer transitions to relay

INFO peer/conn.go:508: ICE disconnected, set Relay to active connection
INFO peer/conn.go:431: set ICE to active connection
INFO peer/conn.go:466: configure WireGuard endpoint to: 10.244.1.103:51820

Socket creation succeeds, so CAP_NET_RAW is present. Only sendto fails.

Current behavior

client/iface/wgproxy/ebpf/wrapper.go (v0.76.3):

func NewPacketHeaders(localWGListenPort int, endpoint *net.UDPAddr) (*PacketHeaders, error) {
	ipv4 := &layers.IPv4{
		DstIP:    localHostNetIPv4,   // 127.0.0.1
		SrcIP:    endpoint.IP,        // <- the caller's endpoint

Two callers pass different endpoints:

Caller	endpoint.IP	Packet
AddTurnConn (ebpf/proxy.go:122)	net.ParseIP(loopbackAddr), 127.0.0.1	127.0.0.1 → 127.0.0.1
RedirectAs (ebpf/wrapper.go:190-215)	the peer's real endpoint	e.g. 10.244.1.103 → 127.0.0.1

The kubelet creates this rule in the filter table on every node
(pkg/kubelet/kubelet_network_linux.go, "block incoming localnet connections"):

chain KUBE-FIREWALL {
    ip saddr != 127.0.0.0/8 ip daddr 127.0.0.0/8 ct status dnat counter packets 157095 drop
}
chain OUTPUT {
    type filter hook output priority filter; policy accept;
    jump CILIUM_OUTPUT
    jump KUBE-FIREWALL
}

Packets from RedirectAs match it: non-loopback source, loopback destination, and
untracked. A NF_DROP verdict in the output hook is returned to the sender as
EPERM, which is the errno logged.

The steady-state case passes because NetBird's own rules exempt it, and those
counters show the mechanism working when the source is loopback:

chain netbird-raw-out {
    oifname "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 udp sport 51820 counter packets 516616 notrack
    oifname "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 udp dport 51820 counter packets 398045 notrack
}

Both rules require ip saddr 127.0.0.1, so they do not cover the redirected shape.

client/iface/wgproxy/udp/rawsocket.go has the same problem: NewSrcFaker is called
from WGUDPProxy.RedirectAs with the peer endpoint, so setting
NB_DISABLE_EBPF_WG_PROXY=true moves the error rather than avoiding it.

Every injected packet fails with sendto: operation not permitted, and the
kubelet's drop counter is the only rule in the output path matching that packet shape.

Expected behavior

Relay connection can be established successfully.
The injected packets reach the local WireGuard listener.

Steps to reproduce

1. Run the NetBird client on a Kubernetes node, so the kubelet's KUBE-FIREWALL rule
   is present. Confirm with
   nft list chain ip filter KUBE-FIREWALL.
2. Establish a peer connection that starts on relay and later gets a direct ICE path,
   for example by having one side behind a NAT that drops idle UDP mappings.
3. Watch the client log during the handover.

Environment and topology

• Peer A: NAT
• Peer B: NAT
• Firewalls: kubelet's iptables rules

Self-hosted details, if available

No response

Logs, status output, or debug evidence

2026-09-09T21:22:15.605Z INFO mgmt/mgmt.go:301: refreshed mgmt cache domain=turn.example.net type=A
2026-09-09T21:22:15.668Z INFO [peer: peer-A] peer/worker_ice.go:302: connection succeeded with offer session: <id>
2026-09-09T21:22:15.668Z INFO [peer: peer-A] peer/conn.go:431: set ICE to active connection
2026-09-09T21:22:15.668Z INFO [peer: peer-A] peer/conn.go:466: configure WireGuard endpoint to: 10.244.1.103:51820
2026-09-09T21:22:15.819Z INFO [peer: peer-B] peer/worker_ice.go:302: connection succeeded with offer session: <id>
2026-09-09T21:22:15.819Z INFO [peer: peer-B] peer/conn.go:431: set ICE to active connection
2026-09-09T21:22:15.819Z INFO [peer: peer-B] peer/conn.go:466: configure WireGuard endpoint to: <peer-public-ip>:10028
2026-09-09T21:22:16.079Z INFO [peer: peer-C] peer/worker_ice.go:302: connection succeeded with offer session: <id>
2026-09-09T21:22:16.079Z INFO [peer: peer-C] peer/conn.go:431: set ICE to active connection
2026-09-09T21:22:16.079Z INFO [peer: peer-C] peer/conn.go:466: configure WireGuard endpoint to: 10.244.6.228:51820
2026-09-09T21:22:16.180Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:16.180Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
        ... 1 further identical lines elided ...
2026-09-09T21:22:16.669Z WARN [peer: peer-A] peer/worker_ice.go:354: got an error while sending the punch packet, err: sendto: no buffer space available
2026-09-09T21:22:16.820Z WARN [peer: peer-B] peer/worker_ice.go:354: got an error while sending the punch packet, err: sendto: no buffer space available
2026-09-09T21:22:16.866Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:16.866Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
        ... 5 further identical lines elided ...
2026-09-09T21:22:17.081Z WARN [peer: peer-C] peer/worker_ice.go:354: got an error while sending the punch packet, err: sendto: no buffer space available
2026-09-09T21:22:18.258Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:19.074Z INFO [peer: peer-D] peer/conn.go:352: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected
2026-09-09T21:22:19.074Z INFO [peer: peer-D] peer/handshaker.go:136: received answer, running version 0.76.3, remote WireGuard listen port 51820, session id: <id>, remote ICE supported: true
2026-09-09T21:22:20.257Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:22.249Z INFO [peer: peer-C] peer/conn.go:508: ICE disconnected, set Relay to active connection
2026-09-09T21:22:22.254Z INFO [peer: peer-E] peer/worker_ice.go:302: connection succeeded with offer session: <id>
2026-09-09T21:22:22.254Z INFO [peer: peer-E] peer/conn.go:431: set ICE to active connection
2026-09-09T21:22:22.254Z INFO [peer: peer-E] peer/conn.go:466: configure WireGuard endpoint to: 192.168.1.133:59595
2026-09-09T21:22:22.298Z INFO [peer: peer-F] peer/worker_ice.go:302: connection succeeded with offer session: <id>
2026-09-09T21:22:22.298Z INFO [peer: peer-F] peer/conn.go:431: set ICE to active connection
2026-09-09T21:22:22.298Z INFO [peer: peer-F] peer/conn.go:466: configure WireGuard endpoint to: <peer-public-ip>:35603
2026-09-09T21:22:23.093Z INFO [peer: peer-C] peer/conn.go:383: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2026-09-09T21:22:23.093Z INFO [peer: peer-C] peer/handshaker.go:114: received offer, running version 0.76.3, remote WireGuard listen port 51820, session id: <id>, remote ICE supported: true
2026-09-09T21:22:23.142Z INFO [peer: peer-C] peer/conn.go:352: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected
2026-09-09T21:22:23.142Z INFO [peer: peer-C] peer/handshaker.go:136: received answer, running version 0.76.3, remote WireGuard listen port 51820, session id: <id>, remote ICE supported: true
2026-09-09T21:22:23.273Z INFO [peer: peer-C] peer/worker_ice.go:302: connection succeeded with offer session: <id>
2026-09-09T21:22:23.274Z INFO [peer: peer-C] peer/conn.go:431: set ICE to active connection
2026-09-09T21:22:23.274Z INFO [peer: peer-C] peer/conn.go:466: configure WireGuard endpoint to: 10.244.6.228:51820
2026-09-09T21:22:23.374Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:23.469Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
        ... 486 further identical lines elided ...
2026-09-09T21:22:24.274Z WARN [peer: peer-C] peer/worker_ice.go:354: got an error while sending the punch packet, err: sendto: no buffer space available
2026-09-09T21:22:24.275Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:24.430Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
        ... 3 further identical lines elided ...
2026-09-09T21:22:24.687Z INFO [peer: peer-A] peer/conn.go:383: OnRemoteOffer, on status ICE: Connected, status Relay: Connected
2026-09-09T21:22:24.687Z INFO [peer: peer-A] peer/handshaker.go:114: received offer, running version 0.76.3, remote WireGuard listen port 51820, session id: <id>, remote ICE supported: true
2026-09-09T21:22:24.694Z INFO [peer: peer-A] peer/conn.go:508: ICE disconnected, set Relay to active connection
2026-09-09T21:22:24.696Z INFO [peer: peer-B] peer/conn.go:383: OnRemoteOffer, on status ICE: Connected, status Relay: Connected
2026-09-09T21:22:24.696Z INFO [peer: peer-B] peer/handshaker.go:114: received offer, running version 0.78.1, remote WireGuard listen port 51820, session id: <id>, remote ICE supported: true
2026-09-09T21:22:24.703Z INFO [peer: peer-B] peer/conn.go:508: ICE disconnected, set Relay to active connection
2026-09-09T21:22:24.703Z INFO [peer: peer-B] peer/conn.go:892: reset wg endpoint
2026-09-09T21:22:24.703Z INFO [peer: peer-B] peer/wg_watcher.go:119: WireGuard watcher received peer reset, restarting handshake timeout
2026-09-09T21:22:25.428Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
2026-09-09T21:22:25.470Z ERRO ebpf/wrapper.go:276: failed to write out turn pkg to local conn: write to raw conn: write ip 0.0.0.0->127.0.0.1: sendto: operation not permitted
        ... 1 further identical lines elided ...

Related issues or discussions

No response

Impact

Workaround

Allow injected packets

iptables -I OUTPUT 1 -o lo -d 127.0.0.1 -p udp --dport 51820 -j ACCEPT

Use the client's WireGuard listen port if it is not 51820. The rule has to sit ahead
of the jump to the kubelet's chain. On our nodes the chain reads:

chain OUTPUT {
    type filter hook output priority filter; policy accept;
    jump CILIUM_OUTPUT
    jump KUBE-FIREWALL
}

An accept verdict reached before that jump ends traversal of the filter hook, so the
injected packet is delivered and both the relayed and the redirected paths work.

Two things to know about it. It narrows the kubelet's anti-spoof rule for one UDP
port on the loopback interface; since only locally generated packets reach the output
hook, nothing from the wire is admitted by it. And it does not survive a reboot on its
own. An immutable distribution such as Talos has no place in machine configuration for
an output rule, so keeping it means a privileged DaemonSet or equivalent that
re-asserts it, which is a component whose only job is this rule.

Additional context

Suggested Fixes

1. Have the firewall manager add an accept for NetBird's own injections in the filter
   output path, ahead of the kubelet's chain. NetBird already manages rules in that
   table (netbird-raw-out, netbird-acl-*), so this fits the existing shape; the
   current netbird-raw-out rules would need to match the redirected source too.
2. Keep the loopback source for injected packets and carry the peer identity in the
   port, as the eBPF port swap already does, so no packet ever has a non-loopback
   source with a loopback destination.
3. As a minimum robustness measure, treat EPERM from sendPkg as a signal to fall
   back to the loopback-sourced header, so a handover degrades to working relay
   instead of a silent blackhole.

[Spiritreader]

netbird-talos-unraid-ums-w-1-redacted.log

Here are the full redacted logs