feat(connlib): request larger buffers for UDP sockets

[thomaseizinger]

Sufficiently large receive buffers are important to sustain high-throughput as latency increases. If the receive buffer in the kernel is too small, packets need to be dropped on arrival.

Firefox uses 1MB in its QUIC stack 0. quic-go recommends to set send and receive buffers to 7.5 MB 1. Power users of Firezone are likely receiving a lot more traffic than the average Firefox user (especially with Internet Resource activated) so setting it to 10 MB seems reasonable. Sending packets is likely not as critical because we have back-pressure through our system such that we will stop reading IP packets when we cannot write to our UDP socket. The UDP socket is sitting in a separate thread and those threads are connected with dedicated queues which act as another buffer. However, as the data below shows, some systems have really small send buffers which are currently likely a speed bottleneck because we need to suspend writing so frequently.

Assuming a 50ms latency, the bandwidth-delay product tells us that we can (in theory) saturate a 1.6 Gbps link with a 10MB receive buffer (assuming the OS also has large enough buffer sizes in its TCP or QUIC stack):

80 Mb / 0.05s = 1600Mbps

Experiments and research 2 show the following:

OS	Receive buffer (default)	Receive buffer (this PR)	Send buffer (default)	Send buffer (this PR)
Windows	65KB	10MB	65KB	1MB
MacOS	786KB	8MB	9KB	1MB
Linux	212KB	212KB	212KB	212KB

With the exception of Linux, the OSes appear to be quite generous with how big they allow receive buffers to be. On Linux, these limit can be changed by setting the core.net.rmem_max and core.net.wmem_max parameters using sysctl.

Most of our users are on Windows and MacOS, meaning they immediately benefit from this without having to change any system settings. Larger client-side UDP receive buffers are critical for any "download" scenario which is likely the majority of usecases that Firezone is used for.

On Windows, increasing this receive buffer almost doubles the throughput in an iperf3 download test.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
firezone	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 22, 2025 4:23am

[thomaseizinger]

Interesing, with this patch, Windows directly hands me a UDP socket with a 10MB buffer size without changing any system settings. This almost doubles the throughput I am getting during a download:

Latest release:

PS C:\Users\thomas\Downloads\iperf-3.18-win64> .\iperf3.exe -c  10.6.96.4 -t 0
Connecting to host 10.6.96.4, port 5201
[  5] local 100.92.78.37 port 59737 connected to 10.6.96.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  46.5 MBytes   387 Mbits/sec
[  5]   1.01-2.02   sec  26.0 MBytes   216 Mbits/sec
[  5]   2.02-3.01   sec  14.2 MBytes   120 Mbits/sec
[  5]   3.01-4.01   sec  15.1 MBytes   126 Mbits/sec
[  5]   4.01-5.00   sec  18.1 MBytes   154 Mbits/sec
[  5]   5.00-6.00   sec  21.9 MBytes   184 Mbits/sec
[  5]   6.00-7.01   sec  27.6 MBytes   231 Mbits/sec
[  5]   7.01-8.02   sec  32.4 MBytes   269 Mbits/sec
[  5]   8.02-9.00   sec  36.9 MBytes   313 Mbits/sec
[  5]   9.00-10.01  sec  41.9 MBytes   350 Mbits/sec
[  5]  10.01-11.01  sec  46.2 MBytes   386 Mbits/sec
[  5]  11.01-12.00  sec  37.8 MBytes   320 Mbits/sec
[  5]  12.00-13.01  sec  42.9 MBytes   358 Mbits/sec
[  5]  13.01-14.01  sec  34.4 MBytes   286 Mbits/sec
[  5]  14.01-15.00  sec  33.6 MBytes   285 Mbits/sec
[  5]  15.00-16.00  sec  24.9 MBytes   208 Mbits/sec
[  5]  16.00-17.01  sec  27.1 MBytes   226 Mbits/sec
[  5]  17.01-18.00  sec  29.8 MBytes   252 Mbits/sec
[  5]  18.00-19.00  sec  33.6 MBytes   281 Mbits/sec
[  5]  19.00-20.01  sec  35.2 MBytes   295 Mbits/sec
[  5]  20.01-21.01  sec  41.9 MBytes   350 Mbits/sec
[  5]  21.01-22.00  sec  41.4 MBytes   350 Mbits/sec
[  5]  22.00-23.01  sec  35.6 MBytes   298 Mbits/sec
[  5]  23.01-24.01  sec  39.1 MBytes   326 Mbits/sec
[  5]  24.01-25.01  sec  39.9 MBytes   334 Mbits/sec
[  5]  25.01-26.01  sec  37.2 MBytes   312 Mbits/sec
[  5]  26.01-27.00  sec  33.0 MBytes   280 Mbits/sec
[  5]  27.00-28.01  sec  36.0 MBytes   301 Mbits/sec
[  5]  28.01-29.01  sec  17.6 MBytes   148 Mbits/sec
[  5]  29.01-30.01  sec  26.8 MBytes   224 Mbits/sec
[  5]  30.01-31.00  sec  30.0 MBytes   254 Mbits/sec
[  5]  31.00-32.01  sec  32.8 MBytes   274 Mbits/sec
[  5]  32.01-33.01  sec  36.1 MBytes   302 Mbits/sec
[  5]  33.01-33.91  sec  33.6 MBytes   314 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-33.91  sec  1.08 GBytes   274 Mbits/sec                  sender
[  5]   0.00-33.91  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

This PR:

PS C:\Users\thomas\Downloads\iperf-3.18-win64> .\iperf3.exe -c  10.6.96.4 -t 0
Connecting to host 10.6.96.4, port 5201
[  5] local 100.92.78.37 port 59569 connected to 10.6.96.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  42.1 MBytes   351 Mbits/sec
[  5]   1.01-2.01   sec  48.5 MBytes   407 Mbits/sec
[  5]   2.01-3.01   sec  51.2 MBytes   431 Mbits/sec
[  5]   3.01-4.01   sec  39.9 MBytes   332 Mbits/sec
[  5]   4.01-5.01   sec  58.9 MBytes   496 Mbits/sec
[  5]   5.01-6.01   sec  73.0 MBytes   612 Mbits/sec
[  5]   6.01-7.01   sec  73.1 MBytes   614 Mbits/sec
[  5]   7.01-8.00   sec  62.1 MBytes   526 Mbits/sec
[  5]   8.00-9.01   sec  68.0 MBytes   564 Mbits/sec
[  5]   9.01-10.00  sec  59.0 MBytes   499 Mbits/sec
[  5]  10.00-11.01  sec  70.2 MBytes   586 Mbits/sec
[  5]  11.01-12.01  sec  69.6 MBytes   584 Mbits/sec
[  5]  12.01-13.01  sec  66.5 MBytes   560 Mbits/sec
[  5]  13.01-14.01  sec  63.9 MBytes   533 Mbits/sec
[  5]  14.01-15.01  sec  67.0 MBytes   561 Mbits/sec
[  5]  15.01-16.01  sec  67.8 MBytes   571 Mbits/sec
[  5]  16.01-17.00  sec  68.1 MBytes   576 Mbits/sec
[  5]  17.00-18.01  sec  70.2 MBytes   583 Mbits/sec
[  5]  18.01-19.01  sec  73.1 MBytes   616 Mbits/sec
[  5]  19.01-20.01  sec  68.8 MBytes   574 Mbits/sec
[  5]  20.01-21.00  sec  56.8 MBytes   481 Mbits/sec
[  5]  21.00-22.01  sec  63.4 MBytes   526 Mbits/sec
[  5]  22.01-23.00  sec  68.6 MBytes   581 Mbits/sec
[  5]  23.00-24.01  sec  67.0 MBytes   562 Mbits/sec
[  5]  24.01-25.01  sec  56.1 MBytes   470 Mbits/sec
[  5]  25.01-26.01  sec  70.9 MBytes   595 Mbits/sec
[  5]  26.01-27.01  sec  68.9 MBytes   577 Mbits/sec
[  5]  27.01-28.01  sec  67.5 MBytes   564 Mbits/sec
[  5]  28.01-29.02  sec  72.8 MBytes   608 Mbits/sec
[  5]  29.02-30.01  sec  64.0 MBytes   541 Mbits/sec
[  5]  30.01-31.01  sec  35.4 MBytes   297 Mbits/sec
[  5]  31.01-32.01  sec  64.5 MBytes   541 Mbits/sec
[  5]  32.01-33.01  sec  60.2 MBytes   506 Mbits/sec
[  5]  33.01-34.01  sec  61.4 MBytes   514 Mbits/sec
[  5]  34.01-35.01  sec  47.9 MBytes   401 Mbits/sec
[  5]  35.01-36.01  sec  61.4 MBytes   514 Mbits/sec
[  5]  36.01-37.00  sec  68.2 MBytes   579 Mbits/sec
[  5]  37.00-38.00  sec  59.2 MBytes   496 Mbits/sec
[  5]  38.00-39.00  sec  69.1 MBytes   580 Mbits/sec
[  5]  39.00-40.00  sec  66.1 MBytes   555 Mbits/sec
[  5]  40.00-41.00  sec  65.8 MBytes   552 Mbits/sec
[  5]  41.00-42.00  sec  66.2 MBytes   554 Mbits/sec
[  5]  42.00-43.01  sec  70.2 MBytes   588 Mbits/sec
[  5]  43.01-44.01  sec  72.6 MBytes   608 Mbits/sec
[  5]  44.01-45.01  sec  51.4 MBytes   431 Mbits/sec
[  5]  45.01-46.01  sec  62.1 MBytes   520 Mbits/sec
[  5]  46.01-46.77  sec  51.6 MBytes   573 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-46.77  sec  2.88 GBytes   529 Mbits/sec                  sender
[  5]   0.00-46.77  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

[thomaseizinger]

@jamilbk Did you see any throughput improvements with this PR? Because if not, then we don't need to bother increasing the buffer sizes on MacOS and can save ourselves the memory testing (although I think it would be good to confirm that it does in fact no affect user-space RAM usage).

Edit: Built a new client here: https://github.com/firezone/firezone/actions/runs/14584121808/job/40906492534

[jamilbk]

I wasn't able to measure any immediately obvious throughput improvements on macOS. There may be a slight increase in the downstream path. I'm trying to test on iOS now. I need to build the swift workflow for this branch to get a release TestFlight client.

At least on macOS I may have seen an additional 2-3 MB of memory usage (up to ~38 MB) but it's hard to say if that's just due to more DNS requests that happened to be made.

38 MB is (IMO) getting dangerously close to the iOS 50 MB hard limit, so I'm doing more testing now.

Regardless of this PR or not, something in the upstream path in Apple is still slowing things down significantly with added latency in the upstream path.

Let me print the actual socket size the OS gives us because I think it's either clamping here, or we're hitting another small buffer in the upstream path for Apple (utun?).

macOS before

macOS with this PR

[jamilbk]

This might explain the download / upload discrepancy:

2025-04-21 18:13:11.031003 -0700	dev.firezone.firezone.network-extension	firezone_tunnel::sockets: Bound new IPv4 UDP p2p socket send_buf_size=1048576 recv_buf_size=8388608 port=49235
2025-04-21 18:13:11.031052 -0700	dev.firezone.firezone.network-extension	firezone_tunnel::sockets: Bound new IPv6 UDP p2p socket send_buf_size=1048576 recv_buf_size=8388608 port=63765

[jamilbk]

1 MB for 100 ms latency should still allow around 80 Mbps. I definitely don't see a +9 MB increase in userspace memory usage in the network extension process, so it's probably safe to say the buffer size only affects kernel memory (but on iOS, I'm not sure if that factors into our allowance even though it's kernel space).

[jamilbk]

@thomaseizinger Thoughts on trying 8 MB for send and recv on Apple?

[jamilbk]

What might be smart to do here is use much smaller buffers for the DNS queries since those don't need much bandwidth. The WireGuard socket will be the one where we need the 10 MB.

[thomaseizinger]

@thomaseizinger Thoughts on trying 8 MB for send and recv on Apple?

You are welcome to experiment with it (it is just a small code change). As argued in the PR description, I think there are diminishing returns in increasing the send-buffer size.

[jamilbk]

@thomaseizinger Thoughts on trying 8 MB for send and recv on Apple?

You are welcome to experiment with it (it is just a small code change). As argued in the PR description, I think there are diminishing returns in increasing the send-buffer size.

I'll push up another branch with a test.

I disagree about the assumption that upstream is not as critical. For consumer use cases yes (browsing, streaming video) but I think in corporate settings the expectation is the speeds will roughly track line rate minus some overhead to be roughly in line with the vpn they were using before.

Also the default iperf3 test many customers seem to be running defaults to testing upload :-/.

[thomaseizinger]

I disagree about the assumption that upstream is not as critical. For consumer use cases yes (browsing, streaming video) but I think in corporate settings the expectation is the speeds will roughly track line rate minus some overhead to be roughly in line with the vpn they were using before.

Also the default iperf3 test many customers seem to be running defaults to testing upload :-/.

We are disagreeing here on what we are debating. I am not saying upload is not critical. I am saying send buffer sizes are not as much of a critical component when it comes to achieving high throughput. connlib will automatically suspend writing when the send buffer is full. This will cause datagrams within our internal queues to build up to a point where those are full at which point we will stop reading from the TUN device, causing either packets to be dropped there or the sending application to be back-pressured with EWOULDBLOCK due to full buffers.

The kernel will already flush out UDP packets as fast as it can from the buffer to the wire. Increasing this buffer size is not going to make that any faster. My understanding of how this works together is that we have back-pressure with extremely low latency to the producer and therefore don't need large buffers.

The same is not true over the network. Receive buffers need to larger with increased latency because the remote needs to ackknowledge that it received a certain packet in order for backpressure to work.

[thomaseizinger]

I wasn't able to measure any immediately obvious throughput improvements on macOS. There may be a slight increase in the downstream path. I'm trying to test on iOS now. I need to build the swift workflow for this branch to get a release TestFlight client.

At least on macOS I may have seen an additional 2-3 MB of memory usage (up to ~38 MB) but it's hard to say if that's just due to more DNS requests that happened to be made.

38 MB is (IMO) getting dangerously close to the iOS 50 MB hard limit, so I'm doing more testing now.

Regardless of this PR or not, something in the upstream path in Apple is still slowing things down significantly with added latency in the upstream path.

Let me print the actual socket size the OS gives us because I think it's either clamping here, or we're hitting another small buffer in the upstream path for Apple (utun?).

macOS before

# macOS with this PR

These are awesome results! That is a 30% increase in download speed!

My guess would be the increased memory usage comes from the additional threads that we have been adding.

[thomaseizinger]

What might be smart to do here is use much smaller buffers for the DNS queries since those don't need much bandwidth. The WireGuard socket will be the one where we need the 10 MB.

Moved the setting of increased buffers to a setter that we only call for the P2P sockets.

[jamilbk]

I disagree about the assumption that upstream is not as critical. For consumer use cases yes (browsing, streaming video) but I think in corporate settings the expectation is the speeds will roughly track line rate minus some overhead to be roughly in line with the vpn they were using before.
Also the default iperf3 test many customers seem to be running defaults to testing upload :-/.

We are disagreeing here on what we are debating. I am not saying upload is not critical. I am saying send buffer sizes are not as much of a critical component when it comes to achieving high throughput. connlib will automatically suspend writing when the send buffer is full. This will cause datagrams within our internal queues to build up to a point where those are full at which point we will stop reading from the TUN device, causing either packets to be dropped there or the sending application to be back-pressured with EWOULDBLOCK due to full buffers.

The kernel will already flush out UDP packets as fast as it can from the buffer to the wire. Increasing this buffer size is not going to make that any faster. My understanding of how this works together is that we have back-pressure with extremely low latency to the producer and therefore don't need large buffers.

The same is not true over the network. Receive buffers need to larger with increased latency because the remote needs to ackknowledge that it received a certain packet in order for backpressure to work.

I knew I was probably missing something, makes sense.

[jamilbk]

These are awesome results! That is a 30% increase in download speed!

Hm maybe, bear in mind the Gateway is on a burstable CPU so YMMV. I think it's probably safe to merge and at least rules one of the possible buffer size constraints out.