FPM app - incorrect content-length

[jakubvojacek]

Detailed Description of the Problem

When using haproxy directly with PHP FPM (instead of nginx in between), haproxy adds content-length header with an incorrect (always zero) value which cannot be removed.

Some http clients refused to communicate with the API due to content-lenght being always 0 while there are some data

Expected Behavior

Either respect content-length value returned from PHP or allow me to delete the header completely. I tried http-response del-header but the header was still present in the response. When using this directive when proxying to nginx, it removes the header correctly.

Steps to Reproduce the Behavior

1. Create php-fpm app with simple script
2. make it return some string, make sure to set up header content-length correctly
3. use haproxy to proxy traffic to fpm
4. check returned headers

curl -I https://mw.myapp.eu/api/xxx
content-type: application/json; charset=utf-8
content-length: 0
....

Do you have any idea what may have caused this?

unfortunately not

Do you have an idea how to solve the issue?

unfortunately not

What is your configuration?

fcgi-app php-myapp-fpm
    log-stderr global
    pass-header authorization
    docroot /home/domains/myapp/www
    set-param SCRIPT_NAME /index.php

frontend hafrontend
    ....
    use_backend myapp-fpm if { hdr(host) -i mw.myapp.eu } 

backend myapp-fpm
	use-fcgi-app php-myapp-fpm
	http-response del-header content-length
	server-template myapp 1 _myapp_fpm._tcp.myapp.lcl proto fcgi resolvers mydns check cookie myapp init-addr none

Output of haproxy -vv

HAProxy version 2.4.8-d1f8d41 2021/11/03 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.8.html
Running on: Linux 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64
Build options :
  TARGET  = linux-musl
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_NS=1 USE_PROMEX=1
  DEBUG   = 

Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=64).
Built with OpenSSL version : OpenSSL 1.1.1l  24 Aug 2021
Running on OpenSSL version : OpenSSL 1.1.1l  24 Aug 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.45 2021-06-15
Running on PCRE version : 8.45 2021-06-15
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.3.1 20211027

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[COMP] compression
	[TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

No response

[git001]

Please can you try a more recent version of haproxy as you can see on this page http://www.haproxy.org/bugs/bugs-2.4.8.html are a lot of fixes in the current 2.4 version.

[jakubvojacek]

Tested with Linux motv-cico-haproxy01 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64. Removing header still does not seem to work

[git001]

Ähm that the OS Version not the HAProxy version.

[jakubvojacek]

Sorry, copied wrong line HAProxy version 2.5.5-1~bpo11+1 2022/03/14 - https://haproxy.org/

[jakubvojacek]

I got a reproducible sample with ruby client

# frozen_string_literal: true

require 'httparty'

def httparty_request(options)
  response = HTTParty.post(
    "#{options[:mw_url]}/api/translator/exportTranslations",
    body: {
      'data' => {
        'format' => 'ios',
        'language' => 'en'
      }
    }.to_json
  )

  puts response
end

httparty_request(mw_url: 'https://mw.cico.tv')
puts "Translations downloaded"
puts "Translations downloaded"

requires ruby and gem httparty.

This script will not work, for me it ends with

Traceback (most recent call last):
	20: from cico.rb:19:in `<main>'
	19: from cico.rb:6:in `httparty_request'
	18: from /var/lib/gems/2.7.0/gems/httparty-0.20.0/lib/httparty.rb:650:in `post'
	17: from /var/lib/gems/2.7.0/gems/httparty-0.20.0/lib/httparty.rb:543:in `post'
	16: from /var/lib/gems/2.7.0/gems/httparty-0.20.0/lib/httparty.rb:613:in `perform_request'
	15: from /var/lib/gems/2.7.0/gems/httparty-0.20.0/lib/httparty/request.rb:152:in `perform'
	14: from /usr/lib/ruby/2.7.0/net/http.rb:1483:in `request'
	13: from /usr/lib/ruby/2.7.0/net/http.rb:933:in `start'
	12: from /usr/lib/ruby/2.7.0/net/http.rb:1485:in `block in request'
	11: from /usr/lib/ruby/2.7.0/net/http.rb:1492:in `request'
	10: from /usr/lib/ruby/2.7.0/net/http.rb:1536:in `transport_request'
	 9: from /usr/lib/ruby/2.7.0/net/http/response.rb:168:in `reading_body'
	 8: from /usr/lib/ruby/2.7.0/net/http/response.rb:235:in `body'
	 7: from /usr/lib/ruby/2.7.0/net/http/response.rb:210:in `read_body'
	 6: from /usr/lib/ruby/2.7.0/net/http/response.rb:290:in `read_body_0'
	 5: from /usr/lib/ruby/2.7.0/net/http/response.rb:285:in `inflater'
	 4: from /usr/lib/ruby/2.7.0/net/http/response.rb:292:in `block in read_body_0'
	 3: from /usr/lib/ruby/2.7.0/net/http/response.rb:322:in `read_chunked'
	 2: from /usr/lib/ruby/2.7.0/net/protocol.rb:201:in `readline'
	 1: from /usr/lib/ruby/2.7.0/net/protocol.rb:191:in `readuntil'
/usr/lib/ruby/2.7.0/net/protocol.rb:225:in `rbuf_fill': end of file reached (EOFError)

When you swich the url to https://mw.motv.eu it starts to work immediatelly. The only difference is that in the first case, its haproxy -> fmp, in second its haproxy -> nginx -> haproxy

The thing is that it works fine from curl from command line. I tried curl -vvv comparing the outputs and didnt see a difference except for the content-length

[capflam]

Are you sure your script adds a content-length header ? because the fcgi app will add it in HTTP/1.1 if there is nor content-length header neither transfer-encoding header AND if all data were received. Idea it to avoid, as far as possible to chunk the response. Most of time, the scripts don't set those headers.

Ah. There is indeed a bug with HEAD request, the content-length is set to 0 while it should be skipped. With this patch, it should work:

iff --git a/src/fcgi-app.c b/src/fcgi-app.c
index 8d180274a..63e0bd2dd 100644
--- a/src/fcgi-app.c
+++ b/src/fcgi-app.c
@@ -349,7 +349,7 @@ static int fcgi_flt_http_headers(struct stream *s, struct filter *filter, struct
 
                /* Add the header "Content-Length:" if possible */
                sl = http_get_stline(htx);
-               if (sl &&
+               if (s->txn->meth != HTTP_METH_HEAD && sl &&
                    (sl->flags & (HTX_SL_F_XFER_LEN|HTX_SL_F_CLEN|HTX_SL_F_CHNK)) == HTX_SL_F_XFER_LEN &&
                    (htx->flags & HTX_FL_EOM)) {
                        struct htx_blk * blk;

However, for a GET or a POST request, it works:

$ curl -k -i  --http1.1 https://mw.motv.eu/api/translator/exportTranslations
HTTP/1.1 200 OK
server: nginx
date: Tue, 05 Apr 2022 16:21:40 GMT
content-type: application/json; charset=utf-8
transfer-encoding: chunked
keep-alive: timeout=10
vary: Accept-Encoding
access-control-allow-origin: *
access-control-allow-headers: Authorization, User-authorization, Content-type
set-cookie: _nss=1; path=/; HttpOnly; SameSite=Strict
vary: X-Requested-With
x-motv-api-status: 6
strict-transport-security: max-age=16000000; includeSubDomains; preload;
set-cookie: SERVERID=middleware; path=/

{"status":6,"response":"format"}

And:

$ curl -k -i --http1.1 https://mw.cico.tv/api/translator/exportTranslations
HTTP/1.1 200 OK
access-control-allow-origin: *
access-control-allow-headers: Authorization, User-authorization, Content-type
set-cookie: _nss=1; path=/; secure; HttpOnly; SameSite=Strict
vary: X-Requested-With
content-type: application/json; charset=utf-8
x-motv-api-status: 6
strict-transport-security: max-age=16000000; includeSubDomains; preload;
date: Tue, 05 Apr 2022 16:21:57 GMT
content-length: 32

{"status":6,"response":"format"}

So, your script does not add a content-length header because nginx returns a chunked request while haproxy is able to set the content-length. I'll fix the HEAD case soon.

[jakubvojacek]

Hello

sorry, you're right. I wasnt sending the content-lenght header at the time of the testing. I removed it temporarily while I was playing around with curl -I .. and never put it back.

Ah. There is indeed a bug with HEAD request, the content-length is set to 0 while it should be skipped. With this patch, it should work:

Do you want to keep this issue open to track the above or can it be closed?

Thanks alot 👍

[capflam]

Yes, it will be closed once the bug will be fixed and backported.