411's On 2.9-dev7 and above #2386

picoseconddillo · 2023-12-11T22:00:09Z

Detailed Description of the Problem

Hi,

Fairly recent user of HAProxy here, loving it so far, thanks!

We are running into an issue when trying to upgrade from 2.8.x to 2.9.

When we try to upgrade to 2.9 we start to get 411’s from some of our backends on some calls where we were getting 200's. We tested on 2.8.3, 2.8.4, and 2.8.5 and all exhibited the same issue.

Then we tested the various dev tags of 2.9 and got 200's as expected up until the 2.9-dev7.

The 411's are only showing up on our backends that are Windows servers behind AppGW's.

Expected Behavior

Routes that 200 in 2.8.x should return 200 in 2.9.x rather than 411.

Steps to Reproduce the Behavior

Curl endpoint behind HAProxy 2.8.x, get a 200
Deploy HAProxy 2.9.0, get 411

Do you have any idea what may have caused this?

Not really, I have a suspicion it's something with the series of Content Length commits between 2.9-dev6 and 2.9-dev7 though.

Do you have an idea how to solve the issue?

Not yet. We're new to HAProxy. We've tried various adjustments on the Content-Length headers in configuration but no dice so far.

What is your configuration?

This is just one of our test configs

  
global
    log stdout format raw local0 info
    maxconn 200
    daemon
    nbthread 3

  defaults
    log-format '{"level":"info","client":"%ci:%cp","time":"%tr","frontend_name":"%ft","frontend_backend_server":"%b/%s","TR":"%TR","Tw":"%Tw","Tc":"%Tc","Tr":"%Tr","Ta":"%Ta","status_code":"%ST","bytes_read":"%B","captured_request_cookie":"%CC","captured_response_cookie":"%CS","termination_state_with_cookie":"%tsc","connection_count":"%ac/%fc/%bc/%sc/%rc","queue_length":"%sq/%bq","host_header":"%hrl","response_headers":"%hs","request":"%r"}'
    error-log-format '{"level":"error","client":"%ci:%cp","time":"[%tr]","frontend_name":"%ft","connection_count":"%ac/%fc","message":"%[fc_err_str]/%[bc_err_str]","error_codes":"%[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed]","certificate_details":"%[ssl_fc_sni]/%sslv/%sslc","host_header":"%hr"}'
    log global
    option splice-auto
    option redispatch
    timeout client 60s
    timeout connect 60s
    timeout server 60s

  frontend tenants
    http-request capture req.hdr(X-Forwarded-For) len 64
    http-request capture req.hdr(host) len 30
    mode http

    bind *:443 ssl crt /usr/local/etc/ssl/tls.pem no-sslv3 alpn h2,http/1.1
    bind *:80

    option http-server-close
    option forwardfor
    http-request add-header X-Forwarded-Proto https
    http-request add-header X-Forwarded-Port 443
  
    http-response add-header Strict-Transport-Security max-age=15768000


    acl reviewservice_acl path_beg -i 
    use_backend reviewservice if reviewservice_acl
    acl aqua-hosting_acl path_beg -i /api/review/viewer/
    use_backend aqua-hosting if aqua-hosting_acl
    acl test-api_acl path_beg -i 
    use_backend test-api if test-api_acl
    acl identity_acl path_beg -i 
    use_backend identity if identity_acl
    acl smoketest_acl path_beg -i /istio-smoke-test/
    use_backend smoketest if smoketest_acl
    acl healthcheck_acl path_beg -i /healthcheck/readyz
    use_backend healthcheck if healthcheck_acl
    acl loadtest_acl path_beg -i /loadtest
    use_backend loadtest if loadtest_acl

    backend reviewservice
      mode http
      server reviewservice  ssl verify none sni req.hdr(host)
    backend test-api
      mode http
      server test-api  ssl verify none sni req.hdr(host)
    backend identity
      mode http
      server identity  ssl verify none sni req.hdr(host)
    backend smoketest
      mode http
      server smoketest ssl verify none sni req.hdr(host)
    backend healthcheck
      mode http
      http-request return status 299
    backend loadtest
      mode http
      http-request return status 299

Output of `haproxy -vv`

HAProxy version 2.9.0-fddb8c1 2023/12/05 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.0.html
Running on: Linux 5.15.0-1041-azure #48-Ubuntu SMP Tue Jun 20 20:34:08 UTC 2023 x86_64
Build options :
  TARGET  = linux-musl
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=32).
Built with OpenSSL version : OpenSSL 3.1.4 24 Oct 2023
Running on OpenSSL version : OpenSSL 3.1.4 24 Oct 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.1 20220924

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

No response

The text was updated successfully, but these errors were encountered:

lukastribus · 2023-12-11T22:52:04Z

Originally reported on discourse:
https://discourse.haproxy.org/t/2-8-x-to-2-9-sudden-411-issues/9305/

When we try to upgrade to 2.9 we start to get 411’s from some of our backends on some calls where we were getting 200's. We tested on 2.8.3, 2.8.4, and 2.8.5 and all exhibited the same issue.

Then we tested the various dev tags of 2.9 and got 200's as expected up until the 2.9-dev7.

Just reiterating to make sure it's not misinterpreted:

No issue in 2.8.3, 2.8.4, 2.8.5 and 2.9dev6.
2.9dev7 and 2.9.0 is affected.

Also:

After updating our logging it looks like only HTTP Posts with 0 or Undefined Content Lengths are being blocked in 2.9.

capflam · 2023-12-12T09:05:09Z

Indeed, I confirm the issue. When a content-length: 0 is set on the request, it is skipped when it is sent to the server. I will fix it. Thanks !

…riginally Commit f89ba27 ("BUG/MEDIUM: mux-h1; Ignore headers modifications about payload representation") introduced a regression. The Content-Length is no longer sent to the server for requests without payload but with a 'Content-Lnegth' header explicitly set to 0, like POST request with no payload. It is of course unexpected. In some cases, depending on the server, such requests are considered as invalid and a 411-Length-Required is returned. The above commit is not directly responsible for the bug, it only reveals a too lax condition to skip the 'Content-Length' header of bodyless requests. We must only skip this header if none was originally found, during the parsing. This patch should fix the issue #2386. It must be backported to 2.9.

capflam · 2023-12-13T15:48:50Z

I pushed a fix to 3.0-DEV. Thanks !

…riginally Commit f89ba27 ("BUG/MEDIUM: mux-h1; Ignore headers modifications about payload representation") introduced a regression. The Content-Length is no longer sent to the server for requests without payload but with a 'Content-Lnegth' header explicitly set to 0, like POST request with no payload. It is of course unexpected. In some cases, depending on the server, such requests are considered as invalid and a 411-Length-Required is returned. The above commit is not directly responsible for the bug, it only reveals a too lax condition to skip the 'Content-Length' header of bodyless requests. We must only skip this header if none was originally found, during the parsing. This patch should fix the issue haproxy#2386. It must be backported to 2.9. (cherry picked from commit 966a18e) Signed-off-by: Willy Tarreau <w@1wt.eu>

picoseconddillo · 2023-12-18T15:11:44Z

Awesome, thank you very much! And can confirm that the fix in 2.9.1 works very well.

picoseconddillo added status: needs-triage This issue needs to be triaged. type: bug This issue describes a bug. labels Dec 11, 2023

capflam added dev This issue affects the HAProxy development branch. status: reviewed This issue was reviewed. A fix is required. 2.9 This issue affects the HAProxy 2.9 stable branch. and removed status: needs-triage This issue needs to be triaged. labels Dec 12, 2023

capflam added status: fixed This issue is a now-fixed bug. and removed dev This issue affects the HAProxy development branch. status: reviewed This issue was reviewed. A fix is required. labels Dec 13, 2023

capflam closed this as completed Dec 15, 2023

capflam removed the 2.9 This issue affects the HAProxy 2.9 stable branch. label Dec 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

411's On 2.9-dev7 and above #2386

411's On 2.9-dev7 and above #2386

picoseconddillo commented Dec 11, 2023 •

edited

lukastribus commented Dec 11, 2023

capflam commented Dec 12, 2023

capflam commented Dec 13, 2023

picoseconddillo commented Dec 18, 2023

411's On 2.9-dev7 and above #2386

411's On 2.9-dev7 and above #2386

Comments

picoseconddillo commented Dec 11, 2023 • edited

Detailed Description of the Problem

Expected Behavior

Steps to Reproduce the Behavior

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

What is your configuration?

Output of haproxy -vv

Last Outputs and Backtraces

Additional Information

lukastribus commented Dec 11, 2023

capflam commented Dec 12, 2023

capflam commented Dec 13, 2023

picoseconddillo commented Dec 18, 2023

picoseconddillo commented Dec 11, 2023 •

edited

Output of `haproxy -vv`