Haproxy sending large response (Almost 90kb) using txn:reply and txn:done returns 500 internal error.

[anandcta123]

Detailed Description of the Problem

I need to send a response from the haproxy lua script from an action registered using register_action. The response size is almost 90kb (if compressed), else it is 240kb. If I try to send this reply using txn:done and txn:reply, the client receives an 500 internal server error. Only way, it works is by allocating enough buffer size using tune.bufsize.

I am aware of an old thread #1447. Is there any recent enhancements done to solve this bug.

If not, please let me know, is there any alternative way available to solve this error or is it possible to dynamically control the bufsize.

Expected Behavior

Expecting to send large response from haproxy.

Steps to Reproduce the Behavior

1. Register a action.
2. Send a large response using txn:reply and txn:done.

Do you have any idea what may have caused this?

As per the documentation, default tune.bufsize is 16kb and txn:reply uses only one buffer and this buffer size is not sufficient.

Do you have an idea how to solve the issue?

No.

What is your configuration?

global
	log /dev/log	local0 
	lua-prepend-path /etc/haproxy/utils/?.lua
	lua-load-per-thread /etc/haproxy/rdwr_httpclient_connector.lua
	chroot /var/lib/haproxy
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	httpclient.ssl.verify none

	set-var proc.rdwr_app_ep_addr str("f808f4f445944ee49bd487367fb7f336.oop.radwarecloud.net")
	set-var proc.x_rdwr_app_id str("f808f4f4-4594-4ee4-9bd4-87367fb7f336")
	set-var proc.x_rdwr_api_key str("d9be6942-2c5b-449a-a302-954d1d7b1143")
	set-var proc.rdwr_app_ep_port int(443) #optional
	set-var proc.rdwr_app_ep_timeout int(0) #optional
	set-var proc.rdwr_app_ep_ssl int(1) #optional
        set-var proc.rdwr_partial_body_size int(10420)#optional
        set-var proc.rdwr_true_client_ip_header str("x-remote-ip") #optional
 	set-var proc.rdwr_bot_manager_enabled str("true")
	set-var proc.rdwr_js_snippet str("")
	
        set-var proc.static_extensions.enabled str("false")
	# Space seperated methods and extensions.
        set-var proc.static_extensions.list_of_methods_not_to_inspect str("GET HEAD")
	set-var proc.static_extensions.list_of_bypassed_extensions str("png jpg css js jpeg gif ico ttf svg woff woff2 svc swf otf eot")
	set-var proc.static_extensions.inspect_if_query_string_exists str("true")

#	tune.bufsize 102400
	tune.maxrewrite 2000
        h1-case-adjust authorization Authorization

resolvers mydns
        nameserver dns1 10.221.1.47:53
        parse-resolv-conf
        timeout retry   1s
        hold valid 10s
        hold nx 30s
        hold other 30s
        hold obsolete 60s
        accepted_payload_size 8192


defaults
	log global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

frontend fe_main
    mode http
    bind *:80 v4v6
    bind *:443 ssl crt /etc/ssl/private/mydomain.pem

    option http-buffer-request
    
    #http-request do-resolve(txn.myip,mydns,ipv4) str("8ea3a2a7b0df42b48f7ba1ab5bee4143.oop.radwarecloud.net")
    #http-request capture var(txn.myip) len 40

    http-request wait-for-body time 1s at-least 10420 
    http-request lua.rdwrHttpClientConnector
 
    filter lua.botm-filter

    option h1-case-adjust-bogus-client

    http-request capture req.hdr(Authorization) len 253

    use_backend be_servers
	
backend be_servers

    mode http
    filter compression
    compression algo gzip
    compression offload

    option h1-case-adjust-bogus-server
 
    balance roundrobin
    server server1 10.26.30.194:8080 check

Output of haproxy -vv

HAProxy version 2.9.7-1ppa1~focal 2024/04/06 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.7.html
Running on: Linux 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-Fn1iYP/haproxy-2.9.7=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=16).
Built with OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

No response

[Darlelet]

The limitation still applies, it is mentioned in txn:done() documentation.
As mentioned in #1447, Lua filters (it is not an experimental feature anymore) would probably be the way to go to overcome the bufsize limitation (in order to stream the response)

[anandcta123]

My design involve to forward the request to an external service using LUA API's of httpclient inside an action. I need to forward the response received as is without forwarding to the backend server. I written a filter, where it operates on client request or backend server response. But how I can call a filter for the response received by LUA API's?. Please let me know, is there any documentation or inputs to write a filter for my task.

[capflam]

I guess it is easier in your case to use a lua service, called via a use-service action, instead of a lua action. Eventually, you can create a lua sample if the service must be conditionally called.

[anandcta123]

Thanks for the help. Really amazing solution. This helped me to solve the issue.