Add support for TCP HyStart #10287
Merged
Add support for TCP HyStart #10287
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Spike Curtis <spike@coder.com>
|
This is great btw. I didn't realize anyone used the CUBIC in gvisor. Gvisor itself still uses RENO by default because most folks don't care about CUBIC in data center environments. Thanks for implementing this! Not on the team anymore but it's great to see gvisor/net stack usage and improvements. |
|
@hbhasker is there a reason not to make CUBIC default? We'll need to benchmark to confirm, but it seems like kind of a layup given it's implemented already. |
|
The benchmarks never showed cubic doing any better at <5ms latency which is majority of gvisor usage. |
kevinGC
reviewed
Apr 16, 2024
kevinGC
reviewed
Apr 17, 2024
|
There was another reason. I implemented cubic with floating point operations. It might make sense to change it to fixed decimal similar to how linux does to save CPU costs and avoid floating point errors. |
Signed-off-by: Spike Curtis <spike@coder.com>
|
I ran the Iperf benchmarks |
You can use the benchstat tool to compare these. It currently shows that it needs more samples to get to 95% confidence. |
|
here's what I got for 10 samples: |
|
@kevinGC ok for this to go in? |
|
Yep, going through the merge process now. |
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 22, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 24, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 24, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 25, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 26, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 26, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 27, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 29, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 30, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
|
Just want to update: we're having issues with test flakes internally and want to fix those rather than forcibly commit. But still trying to get it in. |
copybara-service bot
pushed a commit
that referenced
this pull request
Apr 30, 2024
Adds support for the HyStart algorithm to TCP with CUBIC congestion control. HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets. This implementation of HyStart is based on the Linux Kernel, with some additional commentary on [constant values taken from the HyStart++ RFC](https://www.rfc-editor.org/rfc/rfc9406.html#section-4.3) (note we do not implement the RFC algorithm, preferring to follow Linux). I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT). Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.  With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.  Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection): without HyStart ``` INTERVAL THROUGHPUT 0.00-1.14 sec 44.1423 Mbits/sec 1.14-2.51 sec 48.9794 Mbits/sec 2.51-3.27 sec 21.9600 Mbits/sec ---------------------------------- 0.00-3.27 sec 40.9907 Mbits/sec ``` with HyStart ``` INTERVAL THROUGHPUT 0.00-1.13 sec 44.7275 Mbits/sec 1.13-2.31 sec 84.6852 Mbits/sec 2.31-3.10 sec 85.4545 Mbits/sec ---------------------------------- 0.00-3.10 sec 70.3722 Mbits/sec ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10287 from coder:hystart 5c2220c PiperOrigin-RevId: 627186105
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds support for the HyStart algorithm to TCP with CUBIC congestion control.
HyStart addresses a common problem during slow-start on paths with high bandwidth delay product (BDP), where using only dropped packets as an indication of congestion detects the congestion too late, and the congestion window (cwnd) has vastly overshot the usable bandwidth of the path, causing many many lost packets.
This implementation of HyStart is based on the Linux Kernel, with some additional commentary on constant values taken from the HyStart++ RFC (note we do not implement the RFC algorithm, preferring to follow Linux).
I have tested this implementation via an embedded Tailscale data plane on a path from Abu Dhabi to Helsinki (140 ms RTT).
Throughput vs time without HyStart shows a big dip around 2 seconds, due to packet loss from the mechanism described above.
With HyStart, slow-start exits just before we start exceeding the path bandwidth, and we stabilize throughput without packet loss.
Over a long-ish upload, this doesn't translate to a huge difference, because even without hystart, we stabilize to the right Tx rate within a few seconds. But, it makes a big difference for medium size transfers. If I limit the transfer to 3s (~30MB for my connection):
without HyStart
with HyStart