VPNs

There are a few VPN options to consider when connecting to peers. Part of the choice relies on whether or not a peer supports a particular protocol, but the other issue is which protocols I want to support. For a long time, OpenVPN was the standard for DN42, since it's relatively simple to configure (unlike IPsec, which is faster), but with the rise of Wireguard, that has become a de-facto standard. Tunneling protocols like GRE are in theory a simpler method, but they do not include any authentication features, and for that reason minimal use of these protocol is seen on the network.

A major part of testing was determining the best-performing VPN protocols, though another consideration is how easy-to-use those protocols are.

Results

No VPN

With no VPN, only a direct (para-virtualized) connection between two boxes, the maximum throughput is 25gbps.

Connecting to host 172.16.42.2, port 5201
[  5] local 172.16.42.1 port 57730 connected to 172.16.42.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.97 GBytes  25.5 Gbits/sec    0   3.08 MBytes
[  5]   1.00-2.00   sec  2.66 GBytes  22.9 Gbits/sec    0   3.08 MBytes
[  5]   2.00-3.00   sec  3.17 GBytes  27.2 Gbits/sec    0   3.08 MBytes
[  5]   3.00-4.00   sec  2.54 GBytes  21.9 Gbits/sec    0   3.08 MBytes
[  5]   4.00-5.00   sec  3.13 GBytes  26.9 Gbits/sec    0   3.08 MBytes
[  5]   5.00-6.00   sec  3.07 GBytes  26.4 Gbits/sec    0   3.08 MBytes
[  5]   6.00-7.00   sec  2.33 GBytes  20.0 Gbits/sec    0   3.08 MBytes
[  5]   7.00-8.00   sec  2.32 GBytes  19.9 Gbits/sec    0   3.08 MBytes
[  5]   8.00-9.00   sec  2.48 GBytes  21.3 Gbits/sec    0   3.08 MBytes
[  5]   9.00-10.00  sec  2.85 GBytes  24.4 Gbits/sec    0   3.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec                  receiver

GRE

One of the simplest and in-theory fastest tunneling protocols, GRE can be set up fairly easily by specifying local and peer IPs (as root):

ip tunnel add gre1 mode gre remote 172.16.42.2 local 172.16.42.1 ttl 255
ip link set gre1 up
ip addr add 172.16.42.5/30 dev gre1

The results are still staggering at 4.5gbps, though quite a bit lower than before. The reason for this is not entirely clear. CPU usage spikes to over 10%, and RAM stays roughly the same, so where's the bottleneck? Still, there's more than enough bandwidth available. More worryingly, there are a number of TCP retransmissions, suggesting some amount of packet loss.

Connecting to host 172.16.42.6, port 5201
[  5] local 172.16.42.5 port 35500 connected to 172.16.42.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   515 MBytes  4.32 Gbits/sec    0   2.06 MBytes
[  5]   1.00-2.00   sec   538 MBytes  4.51 Gbits/sec    0   2.29 MBytes
[  5]   2.00-3.00   sec   550 MBytes  4.61 Gbits/sec    0   2.55 MBytes
[  5]   3.00-4.00   sec   556 MBytes  4.67 Gbits/sec    0   2.68 MBytes
[  5]   4.00-5.00   sec   538 MBytes  4.51 Gbits/sec    0   2.96 MBytes
[  5]   5.00-6.00   sec   524 MBytes  4.39 Gbits/sec    0   3.11 MBytes
[  5]   6.00-7.00   sec   572 MBytes  4.80 Gbits/sec  1178   2.21 MBytes
[  5]   7.00-8.00   sec   549 MBytes  4.60 Gbits/sec    0   2.41 MBytes
[  5]   8.00-9.00   sec   512 MBytes  4.30 Gbits/sec    0   2.41 MBytes
[  5]   9.00-10.00  sec   544 MBytes  4.56 Gbits/sec  244   1.22 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.27 GBytes  4.53 Gbits/sec  1422             sender
[  5]   0.00-10.00  sec  5.27 GBytes  4.52 Gbits/sec                  receiver

Wireguard

Wireguard is the first protocol to add on some encryption and authentication overhead, but compared to the rest it should still be blazingly fast. It's also fairly easy to configure, and uses simple public and private keys to connect to peers.

wg genkey > privkey
cat privkey | wg pubkey

Then, in a configuration file called wg0.conf:

[Interface]
PrivateKey = <privkey>
Address = 172.16.42.9/30
ListenPort = 51820

[Peer]
PublicKey = <peer pubkey>
AllowedIPs = 172.16.42.10/30
Endpoint = 172.16.42.2:51820

Even though Wireguard uses encryption, it pushes a whopping 3gbps, not that much less than just tunneling with GRE. Unfortunately, there's still some apparent packet loss, at least with the synthetic workload.

Connecting to host 172.16.42.10, port 5201
[  5] local 172.16.42.9 port 56980 connected to 172.16.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   350 MBytes  2.94 Gbits/sec  311   1.25 MBytes
[  5]   1.00-2.00   sec   362 MBytes  3.04 Gbits/sec    0   1.33 MBytes
[  5]   2.00-3.00   sec   351 MBytes  2.94 Gbits/sec    0   1.39 MBytes
[  5]   3.00-4.00   sec   365 MBytes  3.06 Gbits/sec    0   1.52 MBytes
[  5]   4.00-5.00   sec   352 MBytes  2.96 Gbits/sec    0   1.63 MBytes
[  5]   5.00-6.00   sec   358 MBytes  3.00 Gbits/sec  217   1.18 MBytes
[  5]   6.00-7.00   sec   361 MBytes  3.03 Gbits/sec    0   1.32 MBytes
[  5]   7.00-8.00   sec   330 MBytes  2.77 Gbits/sec    0   1.42 MBytes
[  5]   8.00-9.00   sec   345 MBytes  2.89 Gbits/sec    0   1.50 MBytes
[  5]   9.00-10.00  sec   342 MBytes  2.87 Gbits/sec    0   1.60 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.44 GBytes  2.95 Gbits/sec  528             sender
[  5]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec                  receiver