HAProxy ignores backend server H2 max concurrent stream setting

[AndrejG-git]

Detailed Description of the Problem

We have an HAProxy deployment where haproxy daemonset instances are used to load balance HTTP2 requests onto our backend servers.
Our backend servers limit the maximum concurrent stream count to 3.
The HAProxy config we use has the maximum concurrent streams count set to 1000 (tune.h2.max-concurrent-streams 1000).
Despite haproxy and the backend server both exchanging their SETTINGS and acknowledging them, in some rare instances, haproxy ignores the backend limit and sends more than 3 HTTP2 streams. This results in the backend server terminating the connection shortly afterwards.

Not allowed to include .pcap attachments, attaching screenshots of the behavior instead.
3480 is the backend port, 52324 is the haproxy client port for this connection

Overview of the entire TCP stream:

HAProxy SETTINGS frame (max streams 1000)

Backend server SETTINGS frame (max streams 3)

Backend SETTINGS ACK

HAProxy SETTINGS ACK

4 distinct HTTP streams sent by HAProxy

Expected Behavior

We would expect HAProxy to follow the backend server limitation on the max number of concurrent HTTP2 streams.

Steps to Reproduce the Behavior

We can only reproduce this at a customer lab, under heavy load, and even in that environment it only occurs sporadically.

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

Setting the tune.h2.be.max-concurrent-streams explicitly to 3 in the HAProxy config seems to resolve the issue.

What is your configuration?

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# #   HAProxy Ingress Controller
# #   --------------------------
# #   This file is automatically updated, do not edit
# #
#
global
    master-worker
    unix-bind mode 0600
    stats socket /var/run/haproxy/admin.sock level admin expose-fd listeners mode 600
    maxconn 100000
    hard-stop-after 10m
    log 127.0.0.1:514 len 1024 format rfc5424 local0
    log-tag ingress
    lua-load /etc/haproxy/lua/services.lua
    lua-load /etc/haproxy/lua/responses.lua
    tune.ssl.default-dh-param 2048
    lua-prepend-path /etc/REDACTED/?.lua
    lua-load /etc/REDACTED/XXX.lua
    tune.h2.max-concurrent-streams 1000
    tune.h2.initial-window-size 1048576
    tune.bufsize 131072
    nbthread 1

defaults
    log global
    maxconn 100000
    option redispatch
    option dontlognull
    option http-server-close
    option http-keep-alive
    timeout client          50s
    timeout client-fin      50s
    timeout connect         5s
    timeout http-keep-alive 1m
    timeout http-request    5s
    timeout queue           5s
    timeout server          50s
    timeout server-fin      50s
    timeout tunnel          1h
    option http-no-delay
    timeout connect 200
    timeout client 300s
    timeout server 5000
    option dontlog-normal
    no option prefer-last-server
    timeout client-fin 1s
    timeout server-fin 1s
    no option http-server-close

  # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# #
# #   BACKENDS
# #
#
backend XXX_west-site1-ingress-local_3480
    mode http
    balance leastconn
    acl https-request ssl_fc
    http-request set-header X-Original-Forwarded-For %[hdr(x-forwarded-for)] if { hdr(x-forwarded-for) -m found }
    http-request del-header x-forwarded-for
    option forwardfor
    http-reuse always
    option nolinger
    option httpchk
    http-check connect proto h2
    http-check send meth HEAD uri http://%s/oam/v1 ver HTTP/2 hdr XXX-health-check true
    http-check expect rstatus [0-9]+
    default-server max-reuse 500 fall 1 rise 5
    retries 3
    retry-on conn-failure empty-response
    option redispatch

Output of haproxy -vv

# haproxy -vv
HAProxy version 2.8.5-aaba8d0 2023/12/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.5.html
Running on: Linux 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=12).
Built with OpenSSL version : OpenSSL 1.1.1w  11 Sep 2023
Running on OpenSSL version : OpenSSL 1.1.1w  11 Sep 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

Additional Information

No response

[wtarreau]

Hi!

Actually it does respect the advertised limit once it's known, it's just that, like any H2 client, it has to guess a reasonable limit until it's known. As you've seen, it starts with tune.h2.be.max-concurrent-streams. That's in fact one well-known design limitation of the H2 protocol, the streams limit is negotiated after the connection is complete instead of during the TLS handshake, which forces the client to kill the extraneous streams and possibly retry them later.

What is surprising is that the server kills the connection as soon as the initial limit is passed. Normal servers respond with RST_STREAM(refused_stream) to force let the client know that the extra streams are rejected. But what the server does from a protocol perspective is allowed since any stream error may be promoted to a connection error. In this case, what you've done by setting the outgoing streams limit is pretty much valid and the right solution.

Also, please note that in any case, mixing multiple clients on a single H2 backend connection can have some well-known head-of-line blocking effects where a slow client can slow down all other ones. We've almost entirely mitigated that recently in 3.0 (or maybe 3.1?) but this remains a well-known pattern of multiplexing over TCP which is not well suited for all cases. When I'm seeing your value of 1000, it really scares me. And the fact that your backend servers are limited to 3 makes much more sense when used behind an aggregating load-balancer.

Another point to keep in mind is that TCP congestion control algorithms are mostly loss-based, which means that you frequently encounter losses over TCP that you don't necessarily notice, these are needed to evaluated the optimal window hence bandwidth. When combining many streams over a connection, a tail loss (the last packet in a series) takes time to be detected and will freeze everything that follows for the time it takes to detect that loss and retransmit the packet. When you have multiple connections in parallel, the loss of a connection only freezes the few streams of that connection and leaves more bandwidth for other connections to continue progressing. When you have all your streams on a single connection, all streams will experience pauses. This is another reason for not accumulating too many streams per connection. Better limit yourself to a few to a few tens than 1000.

[AndrejG-git]

Thank you for the detailed response. I am still a bit unclear on your explanation of the H2 client behavior; in the packet capture I've attached HAProxy acknowledges the backend server SETTINGS frame before issuing more than the maximum number of concurrent streams. Wouldn't we only expect this to happen before the acknowledgement? Is this a form of internal race condition?

I agree with your point that 1000 seems like an excessively large number for the frontend streams max, however, one of our clients has a very small connection pool and desires such a large number of concurrent streams, despite the apparent downsides.

Also if you could elaborate on your point about "mixing multiple clients on a single H2 backend connection", since our understanding is that by setting such a low backend streams value, we do avoid that issue, by essentially spreading out the frontend connections to many backend ones.

[wtarreau]

HAProxy acknowledges the backend server SETTINGS frame before issuing more than the maximum number of concurrent streams

Good point indeed, though at this point it's very likely that the streams were already created and the data starting to go out. I'll have a look to see if we can find a way to kill them before reaching the connection's buffer and get rid of already serialized data at the same time, but I can't promise anything.

if you could elaborate on your point about "mixing multiple clients on a single H2 backend connection" ...

What I mean is that indeed, while in your case with a limit of 3 you're clearly avoiding that, I was particularly shocked by the 1000 on the front and hope such large numbers are never used on your backend (and your client doesn't itself aggregate that many connections over a single one). That's more of a general comment to be careful about this, as we've seen quite often users getting trapped by aggregating too many streams over too few connections. But at 3, yes I agree you're quite fine. I remember that in the early days we started at 5 and suggested not going beyond 10 on the backend.

[AndrejG-git]

That makes sense, thank you