FreeBSD drops / retries many packets when forwarding through a ZeroTier tunnel

[obrienmd]

Given #579 seems to be resolved by local.conf and is no longer an issue (though the auto-pathfinding not trying to use the tunnel would be preferred of course), I've been continuing load testing on FreeBSD, with the end goal being site to site tunnels using FreeBSD edge devices (OPNSense to be precise).

My current issue is that performance is terrible when client packets are routed through a FreeBSD ZeroTier tunnel. Basic logic follows, and testing notes are included at the end.

1. OPNSense to OPNSense iperf through a Zerotier tunnel looks great
2. Win client to Win client routed via OPNSense through a site-site Zerotier tunnel looks terrible
3. Win client to Win client routed via Linux through a site-site Zerotier tunnel looks good

CPU isn't running that hot on the router, mbufs look great, etc... I've played around with MTU, but I'm at a loss for how to move forward.

OPNSense router A - Zerotier over WAN - OPNSense router B
obrienmd results over 60ms gigabit link with 2CPU E5 or equivalent / 4GB on each OPNSense router:

[OPNSense A] iperf3 -s -B {{OPNSense A LAN IP}}
[OPNSense B] iperf3 -P 10 -c -t 180 -c {{OPNSense A LAN IP}} -B {{OPNSense B LAN IP}}
BW 250mbps, 10k retries (seems reasonable for # of packets)

[OPNSense A] iperf3 -s -B {{OPNSense A LAN IP}}
[OPNSense B] iperf3 -u -b 50M -c {{OPNSense A LAN IP}} -B {{OPNSense B LAN IP}}
Jitter 0.034ms, packet loss 0.25% (seems reasonable)

Windows Client A - LAN A - OPNSense router A - Zerotier over WAN - OPNSense router B - LAN B - Windows Client B
obrienmd results over 60ms gigabit link with 2CPU E5 or equivalent / 4GB on each OPNSense router:

[Win Client A] iperf3 -s
[Win Client B] iperf3 -P 10 -c -t 180 -c {{Win Client A LAN IP}}
BW 30mbps, not sure of retries as Win client doesn't report
Seems FAR too low based on firewall to firewall, and Win clients are 
not CPU/net constrained in the least.

[Win Client A] iperf3 -s
[Win Client B] iperf3 -u -b 50M -c {{Win Client A LAN IP}}
Jitter 0.018ms, packet loss 35.50%
Again, seems horrible based on the firewall to firewall results.

Windows Client A - LAN A - Linux router A (direct WAN) - Zerotier over WAN - Linux router B (direct WAN) - LAN B - Windows Client B
obrienmd results over 60ms gigabit link with 2CPU E5 or equivalent / 4GB on each Linux router:

[Win Client A] iperf3 -s
[Win Client B] iperf3 -P 10 -c -t 180 -c {{Win Client A LAN IP}}
BW 230mbps, not sure of retries as Win client doesn't report
Looks good to me!

[Win Client A] iperf3 -s
[Win Client B] iperf3 -u -b 50M -c {{Win Client A LAN IP}}
Jitter 0.018ms, packet loss 4.5%
Looks meh, but a heck of a lot better than going through OPNSense / FreeBSD routers!

Windows Client A - LAN A - Linux router A (via OPNSense GW) - Zerotier over WAN - (via OPNSense GW) Linux router B - LAN B - Windows Client B
obrienmd results over 60ms gigabit link with 2CPU E5 or equivalent / 4GB on each Linux router:

[Win Client A] iperf3 -s
[Win Client B] iperf3 -P 10 -c -t 180 -c {{Win Client A LAN IP}}
BW 35mbps, not sure of retries as Win client doesn't report
Bad again with OPNSense in path.

[Win Client A] iperf3 -s
[Win Client B] iperf3 -u -b 50M -c {{Win Client A LAN IP}}
Jitter 0.024ms, packet loss 27%
Bad again with OPNSense in path.

[obrienmd]

Added a bit more test data - putting OPNSense as just a WAN gateway on each side (beyond the Linux routers) kill throughput again. So, it not even OPNSense directly forwarding packets via a Zerotier tunnel, it's OPNSense anywhere in the path.

To verify - Suricata is off, as are all other "fancy" firewall things. iPerf between hosts behind OPNSense generally (with Zerotier not involved) is great.

[obrienmd]

I'm not really getting anywhere. Today:

I traced syscalls on the OPNSense zerotier-one process (though it seems to be network layer per last few posts) and saw:

1. Tons of gettimeofday syscalls, I think one for each packet
2. Quite a bit of zerotier-one RET recvfrom -1 errno 35 Resource temporarily unavailable

I thought OPNSense wasn't holding states for the firewall correctly, in that return traffic (9993/UDP back to an ephemeral port) was being blocked at the non-9993 side even though a state should have been opened from the originating "connection" (UDP, but a state should be opened anyway). I still think that's true, but opening up all src port 9993 traffic didn't really help timeouts / retransmissions / packet loss much.

I feel like I'm going in circles on this, and given it's a work (though side) project, I might have to fall off for a bit and fail back to OpenVPN or GRE/IPSec.