Detailed Description of the Problem
I noticed that for haproxy errors CD, cD and SD, the HTTP response code sent to the client is sometime "200".
This is seen in production on a busy website (several thousands of simultaneous users).
It does trigger in very specific case:
- the client is using a very slow internet connection
- the connection errors in CD, cD or SD due to a timeout or connection breakdown
- the severance of the connection makes a JS file of ~9 MB to be only partially transferred to the client
If all of those conditions are met, we see those kind of logs:
Nov 2 08:46:02 aphrodite haproxy[111996]: [REDACTED]:41937 [02/Nov/2021:08:43:56.954] https~ app-1-tomcat0/app-1-tomcat0 1/0/0/5/125115 200 5606487 - - SD-- 750/750/18/17/0 0/0 "GET https://example.org/REDACTED.js HTTP/2.0"
Nov 2 08:46:02 aphrodite haproxy[111996]: [REDACTED]:41168 [02/Nov/2021:08:45:35.191] https~ app-1-tomcat0/app-1-tomcat0 1/0/0/5/27161 200 1672581 - - CD-- 750/750/17/16/0 0/0 "GET https://example.org/REDACTED.js HTTP/2.0"
Nov 2 15:16:48 aphrodite haproxy[111996]: [REDACTED]:50483 [02/Nov/2021:15:06:37.869] https~ app-8-tomcat0/app-8-tomcat0 1/0/1/36/610840 200 7709764 - - cD-- 6944/6944/106/106/0 0/0 "GET https://example.org/REDACTED.js HTTP/2.0"
As you can see, we have errors and the size of the transferred file is wrong, but the status code is still 200.
It results in incomplete files being considered as complete by the client's browser, which breaks website. Also the file is cached by the browser and have to be cleared to download the correct file.
Expected Behavior
The return code should not be 200 on errors.
Steps to Reproduce the Behavior
That's the hard part, I was not able to reproduce it myself by emulating a very slow connection... I'm still looking at how to reproduce it.
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
Maybe set the http status code to "408 Request Timeout" in those cases?
What is your configuration?
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats socket /run/haproxy/monitoring.sock mode 666 level user
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 20000
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.0 no-tls-tickets
ssl-default-server-options ssl-min-ver TLSv1.0 no-tls-tickets
ssl-dh-param-file /usr/local/etc/tls/dh2048.pem
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 15m
timeout server 15m
timeout http-request 5s
timeout client-fin 30s
timeout tunnel 1h
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend https
filter compression
compression algo gzip
compression type text/html text/plain text/xml text/css text/csv text/rtf text/richtext text/javascript application/x-javascript application/javascript application/ecmascript application/rss+xml application/xml application/json application/wasm
mode http
bind :443,:::443 v6only ssl crt /usr/local/etc/tls/haproxy alpn h2,http/1.1
bind :80,:::80 v6only
http-request set-header X-Forwarded-Proto https if { ssl_fc }
redirect scheme https code 301 if !{ ssl_fc }
option forwardfor
default_backend error404
http-response set-header x-proxy-id aphrodite
http-response set-header Strict-Transport-Security "max-age=31536000; preload"
acl domain_identity hdr_dom(host) -i identity.example.com
use_backend identity if domain_identity
use_backend %[base,map_beg(/usr/local/etc/example/haproxy.map)]
backend error404
mode http
errorfile 503 /etc/haproxy/errors/404.http
backend fake_200
mode http
errorfile 503 /etc/haproxy/errors/200.http
backend identity
filter compression
compression algo gzip
compression type text/html text/plain text/xml text/css text/csv text/rtf text/richtext text/javascript application/x-javascript application/javascript application/ecmascript application/rss+xml application/xml application/json application/wasm
balance roundrobin
server id-1 identity-node-1.example.com:8080 check
server id-2 identity-node-2.example.com:8080 check
backend redirect_to_site1
http-request redirect location https://site1.example.org%[capture.req.uri]
backend redirect_to_site2
http-request redirect location https://site2.example.org%[capture.req.uri]
backend app-test-1-tomcat0
filter compression
compression algo gzip
compression type text/html text/plain text/xml text/css text/csv text/rtf text/richtext text/javascript application/x-javascript application/javascript application/ecmascript application/rss+xml application/xml application/json application/wasm
balance roundrobin
server app-test-1-tomcat0 app-test-1.int.example.org:8080 check
...
hundreds of backends
...Output of haproxy -vv
HAProxy version 2.4.7-1~bpo11+1 2021/10/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.7.html
Running on: Linux 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -O2 -ffile-prefix-map=/build/haproxy-2.4.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
DEBUG =
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.1.1k 25 Mar 2021
Running on OpenSSL version : OpenSSL 1.1.1k 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace
Last Outputs and Backtraces
No response
Additional Information
No response
Detailed Description of the Problem
I noticed that for haproxy errors CD, cD and SD, the HTTP response code sent to the client is sometime "200".
This is seen in production on a busy website (several thousands of simultaneous users).
It does trigger in very specific case:
If all of those conditions are met, we see those kind of logs:
As you can see, we have errors and the size of the transferred file is wrong, but the status code is still 200.
It results in incomplete files being considered as complete by the client's browser, which breaks website. Also the file is cached by the browser and have to be cleared to download the correct file.
Expected Behavior
The return code should not be 200 on errors.
Steps to Reproduce the Behavior
That's the hard part, I was not able to reproduce it myself by emulating a very slow connection... I'm still looking at how to reproduce it.
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
Maybe set the http status code to "408 Request Timeout" in those cases?
What is your configuration?
Output of
haproxy -vvLast Outputs and Backtraces
No response
Additional Information
No response