Commits on Mar 3, 2023

1. conn: set SO_{SND,RCV}BUF to 7MB on the Bind UDP socket …

   The conn.Bind UDP sockets' send and receive buffers are now being sized
   to 7MB, whereas they were previously inheriting the system defaults.
   The system defaults are considerably small and can result in dropped
   packets on high speed links. By increasing the size of these buffers we
   are able to achieve higher throughput in the aforementioned case.
   
   The iperf3 results below demonstrate the effect of this commit between
   two Linux computers with 32-core Xeon Platinum CPUs @ 2.9Ghz. There is
   roughly ~125us of round trip latency between them.
   
   The first result is from commit 792b49c which uses the system defaults,
   e.g. net.core.{r,w}mem_max = 212992. The TCP retransmits are correlated
   with buffer full drops on both sides.
   
   Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
   [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
   [  5]   0.00-10.00  sec  4.74 GBytes  4.08 Gbits/sec  2742   285 KBytes
   - - - - - - - - - - - - - - - - - - - - - - - - -
   Test Complete. Summary Results:
   [ ID] Interval           Transfer     Bitrate         Retr
   [  5]   0.00-10.00  sec  4.74 GBytes  4.08 Gbits/sec  2742   sender
   [  5]   0.00-10.04  sec  4.74 GBytes  4.06 Gbits/sec         receiver
   
   The second result is after increasing SO_{SND,RCV}BUF to 7MB, i.e.
   applying this commit.
   
   Starting Test: protocol: TCP, 1 streams, 131072 byte blocks
   [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
   [  5]   0.00-10.00  sec  6.14 GBytes  5.27 Gbits/sec    0   3.15 MBytes
   - - - - - - - - - - - - - - - - - - - - - - - - -
   Test Complete. Summary Results:
   [ ID] Interval           Transfer     Bitrate         Retr
   [  5]   0.00-10.00  sec  6.14 GBytes  5.27 Gbits/sec    0    sender
   [  5]   0.00-10.04  sec  6.14 GBytes  5.25 Gbits/sec         receiver
   
   The specific value of 7MB is chosen as it is the max supported by a
   default configuration of macOS. A value greater than 7MB may further
   benefit throughput for environments with higher network latency and
   lower CPU clocks, but will also increase latency under load
   (bufferbloat). Some platforms will silently clamp the value to other
   maximums. On Linux, we use SO_{SND,RCV}BUFFORCE in case 7MB is beyond
   net.core.{r,w}mem_max.
   
   Co-authored-by: James Tucker <james@tailscale.com>
   Signed-off-by: James Tucker <james@tailscale.com>
   Signed-off-by: Jordan Whited <jordan@tailscale.com>

   

   jwhited and raggi committed Mar 3, 2023
   Copy the full SHA

   a6db822 View commit details

   Browse the repository at this point in the history