VPNs

There are a few VPN options to consider when connecting to peers. Part of the choice relies on whether or not a peer supports a particular protocol, but the other issue is which protocols I want to support. For a long time, OpenVPN was the standard for DN42, since it's relatively simple to configure (unlike IPsec, which is faster), but with the rise of Wireguard, that has become a de-facto standard. Tunneling protocols like GRE are in theory a simpler method, but they do not include any authentication features, and for that reason minimal use of these protocol is seen on the network.

A major part of testing was determining the best-performing VPN protocols, though another consideration is how easy-to-use those protocols are.

Usage

Part of the consideration for which VPN to use was whether the VPN is used on the DN42 network. With this in mind, I used the DN42 PingFinder to collect information about the protocols used by peers. I then took the JSON results from the PingFinder and used CyberChef to parse the VPN types out using the following recipe:

JPath_expression('$..peer_type','\\n')
Regular_expression('User defined','"(.*)"',true,true,false,false,false,false,'List capture groups')
Split(',','\\n')
Sort('Line feed',false,'Alphabetical (case sensitive)')
Count_occurrences({'option':'Simple string','string':'gre/plain'})

On the last line, Count_occurrences, I simply changed the string to whatever I was trying to count. The results are below (some peers support more than one of these):

Protocol	Occurrences
wireguard	214
openvpn	71
gre/plain	38
gre/ipsec	32
tinc	8
zerotier	7
fastd	6
pptp	2
other	25

Clearly, Wireguard has the most widespread support, due to its speed and simplicity, plus consistent support across platforms. OpenVPN also has good support, but lags far behind, and simple tunneling protocols like GRE show a fair amount of support. Other simply means that the peer is willing to try other VPN protocols by request.

Testing

Testing was performed between two VMs, each with 2 Ryzen cores and 2GB of RAM, connected using a VirtIO paravirtualized networking adapter (which has no hard bandwidth limit). In theory, this should give reliable results assuming no network bottleneck. Testing was performed on a subset of the above networks, based on their usage in DN42, their support across network devices (for example, IPsec is often incompatible between vendors), and how easy they are to configure (IPsec being incredibly difficult to set up and maintain securely).

No VPN

With no VPN, only a direct connection between two boxes, the maximum throughput is 25gbps.

Connecting to host 172.16.42.2, port 5201
[  5] local 172.16.42.1 port 57730 connected to 172.16.42.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.97 GBytes  25.5 Gbits/sec    0   3.08 MBytes
[  5]   1.00-2.00   sec  2.66 GBytes  22.9 Gbits/sec    0   3.08 MBytes
[  5]   2.00-3.00   sec  3.17 GBytes  27.2 Gbits/sec    0   3.08 MBytes
[  5]   3.00-4.00   sec  2.54 GBytes  21.9 Gbits/sec    0   3.08 MBytes
[  5]   4.00-5.00   sec  3.13 GBytes  26.9 Gbits/sec    0   3.08 MBytes
[  5]   5.00-6.00   sec  3.07 GBytes  26.4 Gbits/sec    0   3.08 MBytes
[  5]   6.00-7.00   sec  2.33 GBytes  20.0 Gbits/sec    0   3.08 MBytes
[  5]   7.00-8.00   sec  2.32 GBytes  19.9 Gbits/sec    0   3.08 MBytes
[  5]   8.00-9.00   sec  2.48 GBytes  21.3 Gbits/sec    0   3.08 MBytes
[  5]   9.00-10.00  sec  2.85 GBytes  24.4 Gbits/sec    0   3.08 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  27.5 GBytes  23.6 Gbits/sec                  receiver

GRE

One of the simplest and in-theory fastest tunneling protocols, GRE can be set up fairly easily by specifying local and peer IPs (as root):

ip tunnel add gre1 mode gre remote 172.16.42.2 local 172.16.42.1 ttl 255
ip link set gre1 up
ip addr add 172.16.42.5/30 dev gre1

The results are still staggering at 4.5gbps, though quite a bit lower than before. The reason for this is not entirely clear. CPU usage spikes to over 10%, and RAM stays roughly the same, so where's the bottleneck? Still, there's more than enough bandwidth available. More worryingly, there are a number of TCP retransmissions, suggesting some amount of packet loss.

Connecting to host 172.16.42.6, port 5201
[  5] local 172.16.42.5 port 35500 connected to 172.16.42.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   515 MBytes  4.32 Gbits/sec    0   2.06 MBytes
[  5]   1.00-2.00   sec   538 MBytes  4.51 Gbits/sec    0   2.29 MBytes
[  5]   2.00-3.00   sec   550 MBytes  4.61 Gbits/sec    0   2.55 MBytes
[  5]   3.00-4.00   sec   556 MBytes  4.67 Gbits/sec    0   2.68 MBytes
[  5]   4.00-5.00   sec   538 MBytes  4.51 Gbits/sec    0   2.96 MBytes
[  5]   5.00-6.00   sec   524 MBytes  4.39 Gbits/sec    0   3.11 MBytes
[  5]   6.00-7.00   sec   572 MBytes  4.80 Gbits/sec  1178   2.21 MBytes
[  5]   7.00-8.00   sec   549 MBytes  4.60 Gbits/sec    0   2.41 MBytes
[  5]   8.00-9.00   sec   512 MBytes  4.30 Gbits/sec    0   2.41 MBytes
[  5]   9.00-10.00  sec   544 MBytes  4.56 Gbits/sec  244   1.22 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.27 GBytes  4.53 Gbits/sec  1422             sender
[  5]   0.00-10.00  sec  5.27 GBytes  4.52 Gbits/sec                  receiver

Wireguard

Wireguard is the first protocol to add on some encryption and authentication overhead, but compared to the rest it should still be blazingly fast. It's also fairly easy to configure, and uses simple public and private keys to connect to peers.

wg genkey > privkey
cat privkey | wg pubkey

Then, in a configuration file called wg0.conf:

[Interface]
PrivateKey = <privkey>
Address = 172.16.42.9/30
ListenPort = 51820

[Peer]
PublicKey = <peer pubkey>
AllowedIPs = 172.16.42.10/30
Endpoint = 172.16.42.2:51820

Even though Wireguard uses encryption, it pushes a whopping 3gbps, not that much less than just tunneling with GRE. Unfortunately, there's still some apparent packet loss, at least with the synthetic workload.

Connecting to host 172.16.42.10, port 5201
[  5] local 172.16.42.9 port 56980 connected to 172.16.42.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   350 MBytes  2.94 Gbits/sec  311   1.25 MBytes
[  5]   1.00-2.00   sec   362 MBytes  3.04 Gbits/sec    0   1.33 MBytes
[  5]   2.00-3.00   sec   351 MBytes  2.94 Gbits/sec    0   1.39 MBytes
[  5]   3.00-4.00   sec   365 MBytes  3.06 Gbits/sec    0   1.52 MBytes
[  5]   4.00-5.00   sec   352 MBytes  2.96 Gbits/sec    0   1.63 MBytes
[  5]   5.00-6.00   sec   358 MBytes  3.00 Gbits/sec  217   1.18 MBytes
[  5]   6.00-7.00   sec   361 MBytes  3.03 Gbits/sec    0   1.32 MBytes
[  5]   7.00-8.00   sec   330 MBytes  2.77 Gbits/sec    0   1.42 MBytes
[  5]   8.00-9.00   sec   345 MBytes  2.89 Gbits/sec    0   1.50 MBytes
[  5]   9.00-10.00  sec   342 MBytes  2.87 Gbits/sec    0   1.60 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.44 GBytes  2.95 Gbits/sec  528             sender
[  5]   0.00-10.00  sec  3.43 GBytes  2.95 Gbits/sec                  receiver

OpenVPN

OpenVPN is not known for being fast, but it's not too bad and is much easier to configure than the dreaded IPsec protocol. Still, there are several steps to get it set up.

Server

Both the openvpn and easy-rsa packages are required on the server.

mkdir CA
ln -s /usr/share/easy-rsa/* CA/
cd CA/
nano vars

In the variables file, we instruct easy-rsa to use elliptic curves, which are faster and stronger than RSA, and set the common certificate subject.

set_var EASYRSA_REQ_COUNTRY "US"
set_var EASYRSA_REQ_PROVINCE "Washington"
set_var EASYRSA_REQ_CITY "Redmond"
set_var EASYRSA_REQ_ORG "Daniel Hellstern"
set_var EASYRSA_REQ_EMAIL "dn42@daniel.hellstern.org"
set_var EASYRSA_REQ_OU "Testing"
set_var EASYRSA_ALGO "ec"
set_var EASYRSA_DIGEST "sha256"

Then, simply run ./easyrsa init-pki and ./eayrsa build-ca (remember the password!) to create the certificate, then create and sign a server certificate:

./easyrsa gen-req server nopass
./easyrsa sign-req server server
sudo cp pki/{private/server.key,issued/server.crt,ca.crt} /etc/openvpn/server/

Then, create an OpenVPN server config from the sample:

sudo cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/server/
sudo gunzip /etc/openvpn/server/server.conf.gz
sudo nano /etc/openvpn/server/server.conf

Before configuring the server, generate a TA key, which will provide additional security to the connection:

openvpn --genkey --secret ta.key
sudo cp ta.key /etc/openvpn/server/

In the config file, change the following options:

;local a.b.c.d
local 172.16.42.2

;dev tun
dev tap

;server 10.8.0.0 255.255.255.255.0
server 172.16.42.12 255.255.255.252

;tls-auth ta.key 0
tls-crypt ta.key

;cipher AES-256-CBC
cipher AES-256-GCM
auth SHA256

;dh dh2048.pem
dh none

;user nobody
;group nobody
user nobody
group nobody

The server can then be started using sudo systemctl enable --now openvpn-server@server.

Client

For the client certificate, the process is similar:

mkdir ~/client-configs/keys/
./easyrsa gen-req client nopass
./easyrsa sign-req client client
sudo cp pki/{private/client.key,issued/client.crt} ~/client-configs/keys/

Then, copy the client.conf sample file and edit it:

;dev tun
dev tap

;remote my-server-2 1194
remote 172.16.42.2 1194

;tls-auth ta.key
;tls-auth ta.key

key-direction 1

Then, at the bottom, add the section tags for <ca></ca>, <cert></cert>, <key></key>, and <tls-auth></tls-auth>, and insert the relevant keys and certificates. Then, the config can be installed on the client:

sudo cp client.conf /etc/openvpn/client/client.conf
sudo systemctl enable --now openvpn-client@client

The results for OpenVPN are quite disappointing. Even on a multi-gigabit connection, the most OpenVPN can push is just over 300mbps. This can be partially explained by the fact that OpenVPN is using just a single core, and 70% of it.

Connecting to host 172.16.42.13, port 5201
[  5] local 172.16.42.14 port 48528 connected to 172.16.42.13 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  42.1 MBytes   353 Mbits/sec   40   84.7 KBytes
[  5]   1.00-2.00   sec  43.3 MBytes   363 Mbits/sec   51    104 KBytes
[  5]   2.00-3.00   sec  38.4 MBytes   322 Mbits/sec   39   78.3 KBytes
[  5]   3.00-4.00   sec  38.7 MBytes   325 Mbits/sec   28    110 KBytes
[  5]   4.00-5.00   sec  37.6 MBytes   315 Mbits/sec   27    100 KBytes
[  5]   5.00-6.00   sec  38.6 MBytes   323 Mbits/sec   17    105 KBytes
[  5]   6.00-7.00   sec  38.8 MBytes   326 Mbits/sec   33    117 KBytes
[  5]   7.00-8.00   sec  39.9 MBytes   334 Mbits/sec   23    105 KBytes
[  5]   8.00-9.00   sec  38.9 MBytes   326 Mbits/sec   43    104 KBytes
[  5]   9.00-10.00  sec  42.6 MBytes   357 Mbits/sec   42   79.6 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   399 MBytes   334 Mbits/sec  343             sender
[  5]   0.00-10.00  sec   398 MBytes   334 Mbits/sec                  receiver

Conclusion

The protocols I will use must be at the intersection of performance and compatibility. For this reason, it is a no-brainer to support Wireguard as a peering option. It easily outpaces any other VPN, and performs close to plain tunneling protocols. It is also easy enough to support simple GRE/Plain tunneling, which is similarly performant and easy to use. I will not use OpenVPN, despite its widespread usage, because it is difficult to configure and lacks in speed. Other protocols, like Tinc and ZeroTier, are used so rarely that it would not make sense to invest the time into supporting them.