VPNs

There are a few VPN options to consider when connecting to peers. Part of the choice relies on whether or not a peer supports a particular protocol, but the other issue is which protocols I want to support. For a long time, OpenVPN was the standard for DN42, since it's relatively simple to configure (unlike IPsec, which is faster), but with the rise of Wireguard, that has become a de-facto standard. Tunneling protocols like GRE and SIT are in theory a simpler method, but they do not include any authentication features, and for that reason minimal use of these protocol is seen on the network.

A major part of testing was determining the best-performing VPN protocols, though another consideration is how easy-to-use those protocols are.

Results

No VPN

With no VPN, only a direct (para-virtualized) connection between two boxes, the maximum throughput is 25gbps.

Connecting to host 172.16.42.2, port 5201
[  5] local 172.16.42.1 port 57730 connected to 172.16.42.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.97 GBytes  25.5 Gbits/sec    0   3.08 MBytes
[  5]   1.00-2.00   sec  2.66 GBytes  22.9 Gbits/sec    0   3.08 MBytes
[  5]   2.00-3.00   sec  3.17 GBytes  27.2 Gbits/sec    0   3.08 MBytes
[  5]   3.00-4.00   sec  2.54 GBytes  21.9 Gbits/sec    0   3.08 MBytes
[  5]   4.00-5.00   sec  3.13 GBytes  26.9 Gbits/sec    0   3.08 MBytes
[  5]   5.00-6.00   sec  3.07 GBytes  26.4 Gbits/sec    0   3.08 MBytes
[  5]   6.00-7.00   sec  2.33 GBytes  20.0 Gbits/sec    0   3.08 MBytes
[  5]   7.00-8.00   sec  2.32 GBytes  19.9 Gbits/sec    0   3.08 MBytes
[  5]   8.00-9.00   sec  2.48 GBytes  21.3 Gbits/sec    0   3.08 MBytes
[  5]   9.00-10.00  sec  2.85 GBytes  24.4 Gbits/sec    0   3.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec                  receiver

GRE

One of the simplest and in-theory fastest tunneling protocols, GRE can be set up fairly easily by specifying local and peer IPs (as root):

ip tunnel add gre1 mode gre remote 172.16.42.2 local 172.16.42.1 ttl 255
ip link set gre1 up
ip addr add 172.16.42.5/30 dev gre1

The results are still staggering at 4.5gbps, though quite a bit lower than before. The reason for this is not entirely clear. CPU usage spikes to over 10%, and RAM stays roughly the same, so where's the bottleneck? Still, there's more than enough bandwidth available. More worryingly, there are a number of TCP retransmissions, suggesting some amount of packet loss.

Connecting to host 172.16.42.6, port 5201
[  5] local 172.16.42.5 port 35500 connected to 172.16.42.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   515 MBytes  4.32 Gbits/sec    0   2.06 MBytes
[  5]   1.00-2.00   sec   538 MBytes  4.51 Gbits/sec    0   2.29 MBytes
[  5]   2.00-3.00   sec   550 MBytes  4.61 Gbits/sec    0   2.55 MBytes
[  5]   3.00-4.00   sec   556 MBytes  4.67 Gbits/sec    0   2.68 MBytes
[  5]   4.00-5.00   sec   538 MBytes  4.51 Gbits/sec    0   2.96 MBytes
[  5]   5.00-6.00   sec   524 MBytes  4.39 Gbits/sec    0   3.11 MBytes
[  5]   6.00-7.00   sec   572 MBytes  4.80 Gbits/sec  1178   2.21 MBytes
[  5]   7.00-8.00   sec   549 MBytes  4.60 Gbits/sec    0   2.41 MBytes
[  5]   8.00-9.00   sec   512 MBytes  4.30 Gbits/sec    0   2.41 MBytes
[  5]   9.00-10.00  sec   544 MBytes  4.56 Gbits/sec  244   1.22 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.27 GBytes  4.53 Gbits/sec  1422             sender
[  5]   0.00-10.00  sec  5.27 GBytes  4.52 Gbits/sec                  receiver