Add BBR to QUIC Congestion Control Algorithm

[KazuyaKanemura]

Your Feature Request

The congestion control algorithm utilized when using QUIC with HAProxy is currently Cubic.
Cubic tends to overly react to physical packet loss such as CRC errors, leading to a decrease in throughput. Therefore, it would be helpful to have the option to choose the BBR.

References:
BBR
When to use and when not to use BBR: An empirical analysis and evaluation study

What are you trying to do?

HAProxy can maintain stable throughput even in lossy communication environments with QUIC BBR.

quic-cc-algo { cubic | newreno | bbr }(max_window)

Example:

# BBR congestion control algorithm
quic-cc-algo bbr

Output of haproxy -vv

HAProxy version 2.8.5-aaba8d0 2023/12/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.5.html
Running on: Linux 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Thu Jan 12 16:32:13 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PCRE=1 USE_PCRE_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT +PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4).
Built with OpenSSL version : OpenSSL 3.1.5+quic 30 Jan 2024
Running on OpenSSL version : OpenSSL 3.1.5+quic 30 Jan 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.5.0 20210514 (Red Hat 8.5.0-16)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

[haproxyFred]

This is an interesting feature and for sure a must-have for our haproxy QUIC implementation. Thank you for this reminder.

I have begun to read the RFC of BBRv2 here https://www.ietf.org/archive/id/draft-cardwell-iccrg-bbr-congestion-control-02.html and I stopped reading it as soon as it mentions the packet pacing, so in relation with this part https://datatracker.ietf.org/doc/html/rfc9002#name-pacing. This is something we have not already implemented. If we look at the BBR RFC contents, a pacing rate state must be maintain. Here is a short extract about pacing and BBR:

4.6.2. Pacing Rate: BBR.pacing_rate

To help match the packet-arrival rate to the bottleneck bandwidth available to the flow, BBR paces data
packets. Pacing enforces a maximum rate at which BBR schedules quanta of packets for transmission.

But this does not mean we will never implement BBR, this only means there are some changes to be made on the TX part/path to stop sending bursts of packets if we want to support BBR .

[hloeung]

This is interesting, can it use the system configured congestion control algorithm?

net.ipv4.tcp_allowed_congestion_control = reno cubic bbr
net.ipv4.tcp_available_congestion_control = reno cubic bbr
net.ipv4.tcp_congestion_control = bbr

Or at the very least, use the system one, if it's one of the supported ones, and if it's not specified? So can be overridden if quic-cc-algo is specified?

[haproxyFred]

I am not sure to understand what you meant. Linux TCP statck uses its own congestion control algorithm implementations. There is no congestion control algorithm in the linux UDP stack. The sysctls you mentioned are for TCP. Nothing to see with UDP or QUIC.

[hloeung]

Right, I mean, check the system's TCP congestion control algorithm and have HAProxy default to using the same for QUIC (of course, with adding BBR support as per the OP) with the option to override by specifying quic-cc-algo.

So on systems where net.ipv4.tcp_congestion_control is set to reno, HAProxy will use newreno by default for QUIC. On systems where net.ipv4.tcp_congestion_control is cubic, HAProxy will use cubic for QUIC... and bbr if/when support is added.

[osevan]

Im waiting too