Crash (SEGV) in http_wait_for_response in 2.2.19, 2.2.24, and 2.2.26 because sl (start line) variable is NULL.

[frobware]

Detailed Description of the Problem

haproxy faults / crashes in http_wait_for_response() because the variable sl is NULL, after calling sl = http_get_stline(htx);.

I have 3 core dumps from 2.2.19, 2 core dumps from 2.2.24 and 1 core dump from 2.2.26 -- each faulting in the same way.

Expected Behavior

No crash; the call to sl = http_get_stline(htx); should check for NULL and return early.

Steps to Reproduce the Behavior

I do not have a reproducer; this was reported by a customer and it is occuring a few times a day with any of 2.2.19, 2.2.24, and 2.2.26.

Do you have any idea what may have caused this?

It looks like sl is NULL because the underlying htx_blk is of type HTX_BLK_EOM.

/* Returns the next unporocessed start line in the HTX message. It returns NULL
 * if the start-line is undefined (first == -1). Otherwise, it returns the
 * pointer on the htx_sl structure.
 */
struct htx_sl *http_get_stline(struct htx *htx)
{
    struct htx_blk *blk;
 
    blk = htx_get_first_blk(htx);
    if (!blk || (htx_get_blk_type(blk) != HTX_BLK_REQ_SL && htx_get_blk_type(blk) != HTX_BLK_RES_SL))
        return NULL;
// ^^We return NULL because blk type is HTX_BLK_EOM
    return htx_get_blk_ptr(htx, blk);
}

(gdb) p *htx
$53 = {
  size = 32720, 
  data = 85, 
  tail = 4, 
  head = 0, 
  first = 4, 
  tail_addr = 85, 
  head_addr = 0, 
  end_addr = 0, 
  extra = 0, 
  flags = 16, 
  blocks = 0x7f19942bba70 "j"
}
 
(gdb) set $blk = (struct htx_blk *)(htx->blocks + (htx->size - (htx->first + 1) * sizeof(struct htx_blk)))
 
(gdb) p *$blk
$71 = {
  addr = 84, 
  info = 1879048193
}

And to decode info we would call htx_get_blk_type()

/* Returns the type of the block <blk> */
static inline enum htx_blk_type htx_get_blk_type(const struct htx_blk *blk)
{
    return (blk->info >> 28);
}

(gdb) p $blk->info >> 28
$73 = 7
 
(gdb) p/x $blk->info >> 28
$74 = 0x7
 
(gdb) p/d HTX_BLK_EOM 
$78 = 7
 
(gdb) p/x HTX_BLK_EOM 
$79 = 0x7

Do you have an idea how to solve the issue?

Handle the case where the call to sl = http_get_stline(htx); is NULL and return/retry.

What is your configuration?

I will post an update once I have a redacted version of the haproxy config.

Output of haproxy -vv

HA-Proxy version 2.2.19-7ea3822 2021/11/29 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.19.html
Running on: Linux 4.18.0-425.3.1.el8.x86_64 #1 SMP Fri Sep 30 11:45:06 EDT 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_ZLIB=1
  DEBUG   = 

Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=16).
Built with OpenSSL version : OpenSSL 1.1.1g FIPS  21 Apr 2020
Running on OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.4.1 20200928 (Red Hat 8.4.1-1)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
              h2 : mode=HTTP       side=FE|BE     mux=H2
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace
	[CACHE] cache
	[FCGI] fcgi-app

Last Outputs and Backtraces

[aim@hl-rhel84 2.2.19]$ gdb haproxy core.haproxy.1000560000.28adb01d69c44b598e4117023dfadef8.1189525.1671442689000000
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-19.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from haproxy...Reading symbols from /usr/lib/debug/usr/sbin/haproxy-2.2.19-3.el8.x86_64.debug...done.
done.

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 1037]
[New LWP 1033]
[New LWP 1032]
[New LWP 1035]
[New LWP 1036]
[New LWP 1038]
[New LWP 1039]
[New LWP 1034]

warning: .dynamic section for "/lib64/libdl.so.2" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/r'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00005566e59418eb in http_wait_for_response (s=0x7f1994398150, rep=0x7f19943981c0, an_bit=4194304) at src/http_ana.c:1704
1704		txn->status = sl->info.res.status;
[Current thread is 1 (Thread 0x7f19b104d700 (LWP 1037))]
(gdb) t a a bt full

Thread 8 (Thread 0x7f19b2878700 (LWP 1034)):
#0  0x00007f19b44130f7 in glob@@GLIBC_2.27 () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000100000000 in ?? ()
No symbol table info available.
#2  0x000000000000073a in ?? ()
No symbol table info available.
#3  0x0000000d00000001 in ?? ()
No symbol table info available.
#4  0x0000000400000000 in ?? ()
No symbol table info available.
#5  0x0000000000000774 in ?? ()
No symbol table info available.
#6  0x0000066a00000004 in ?? ()
No symbol table info available.
#7  0x0000000400000000 in ?? ()
No symbol table info available.
#8  0x0000000000000671 in ?? ()
No symbol table info available.
#9  0x0000067800000004 in ?? ()
No symbol table info available.
#10 0x0000000400000000 in ?? ()
No symbol table info available.
#11 0x0000000000000681 in ?? ()
No symbol table info available.
#12 0x000006a700000004 in ?? ()
No symbol table info available.
#13 0x0000000400000000 in ?? ()
No symbol table info available.
#14 0x00000000000006ab in ?? ()
No symbol table info available.
#15 0x000006b800000004 in ?? ()
No symbol table info available.
#16 0x0000000400000000 in ?? ()
No symbol table info available.
#17 0x00000000000006da in ?? ()
No symbol table info available.
#18 0x000006ea00000004 in ?? ()
No symbol table info available.
#19 0x0000000400000000 in ?? ()
No symbol table info available.
#20 0x00000000000006f6 in ?? ()
No symbol table info available.
#21 0x0000070400000004 in ?? ()
No symbol table info available.
#22 0x0000000400000000 in ?? ()
No symbol table info available.
#23 0x0000000000000709 in ?? ()
No symbol table info available.
#24 0x0000071d00000004 in ?? ()
No symbol table info available.
#25 0x0000000400000000 in ?? ()
No symbol table info available.
#26 0x0000000000000730 in ?? ()
No symbol table info available.
#27 0x0000074f00000004 in ?? ()
No symbol table info available.
#28 0x0000000400000000 in ?? ()
No symbol table info available.
#29 0x0000000000000d1d in ?? ()
No symbol table info available.
#30 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 7 (Thread 0x7f19abfff700 (LWP 1039)):
#0  0x00007f19b54f2cf6 in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 6 (Thread 0x7f19b084c700 (LWP 1038)):
#0  0x00007f19b44130f7 in glob@@GLIBC_2.27 () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000400000000 in ?? ()
No symbol table info available.
#2  0x00000000000002ee in ?? ()
No symbol table info available.
#3  0x0000084a00000004 in ?? ()
No symbol table info available.
#4  0x0000000400000000 in ?? ()
No symbol table info available.
#5  0x000000000000046d in ?? ()
No symbol table info available.
#6  0x0000091900000001 in ?? ()
No symbol table info available.
#7  0x0000000400000000 in ?? ()
No symbol table info available.
#8  0x0000000000000941 in ?? ()
No symbol table info available.
#9  0x000009a000000004 in ?? ()
No symbol table info available.
#10 0x0000000400000000 in ?? ()
No symbol table info available.
#11 0x00000000000009a2 in ?? ()
No symbol table info available.
#12 0x000009a300000004 in ?? ()
No symbol table info available.
#13 0x0000000400000000 in ?? ()
No symbol table info available.
#14 0x00000000000009a4 in ?? ()
No symbol table info available.
#15 0x000009a500000004 in ?? ()
No symbol table info available.
#16 0x0000000400000000 in ?? ()
No symbol table info available.
#17 0x00000000000009a6 in ?? ()
No symbol table info available.
#18 0x000009a800000004 in ?? ()
No symbol table info available.
#19 0x0000000400000000 in ?? ()
No symbol table info available.
#20 0x00000000000009ab in ?? ()
No symbol table info available.
#21 0x000009ad00000004 in ?? ()
No symbol table info available.
#22 0x0000000400000000 in ?? ()
No symbol table info available.
#23 0x00000000000009b0 in ?? ()
No symbol table info available.
#24 0x000009b200000004 in ?? ()
No symbol table info available.
#25 0x0000000400000000 in ?? ()
No symbol table info available.
#26 0x00000000000009b3 in ?? ()
No symbol table info available.
#27 0x000009b400000004 in ?? ()
No symbol table info available.
#28 0x0000000400000000 in ?? ()
No symbol table info available.
#29 0x00000000000009b5 in ?? ()
No symbol table info available.
#30 0x000009bd00000004 in ?? ()
No symbol table info available.
#31 0x0000000400000000 in ?? ()
No symbol table info available.
#32 0x00000000000009be in ?? ()
No symbol table info available.
#33 0x000009c000000004 in ?? ()
No symbol table info available.
#34 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 5 (Thread 0x7f19b184e700 (LWP 1036)):
#0  0x00007f19b54f2cf6 in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000000030 in ?? ()
No symbol table info available.
#2  0x00007f19b1843470 in ?? ()
No symbol table info available.
#3  0x0000000000000001 in ?? ()
No symbol table info available.
#4  0x00007f19b18434d0 in ?? ()
No symbol table info available.
#5  0x00005566e5a38987 in raw_sock_to_buf (conn=0x7f19b54f2cdc <recv+12>, xprt_ctx=0x7f199c17c9c8, buf=0x0, count=93900427721766, flags=32537) at src/raw_sock.c:331
        ret = 139748329141456
        try = 139748329141360
        done = 1
#6  0x00007ffd74f9cbbf in ?? ()
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (Thread 0x7f19b2077700 (LWP 1035)):
#0  0x00007f19b54f2cf6 in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (Thread 0x7f19b616ea80 (LWP 1032)):
#0  0x00007f19b44130f7 in glob@@GLIBC_2.27 () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000000100000000 in ?? ()
No symbol table info available.
#2  0x00000000000007b3 in ?? ()
No symbol table info available.
#3  0x0000065500000004 in ?? ()
No symbol table info available.
#4  0x0000000400000000 in ?? ()
No symbol table info available.
#5  0x00000000000000ba in ?? ()
No symbol table info available.
#6  0x000000cd00000004 in ?? ()
No symbol table info available.
#7  0x0000000400000000 in ?? ()
No symbol table info available.
#8  0x00000000000000ef in ?? ()
No symbol table info available.
#9  0x0000013c00000004 in ?? ()
No symbol table info available.
#10 0x0000000400000000 in ?? ()
No symbol table info available.
#11 0x000000000000020b in ?? ()
No symbol table info available.
#12 0x0000020f00000004 in ?? ()
No symbol table info available.
#13 0x0000000400000000 in ?? ()
No symbol table info available.
#14 0x000000000000024f in ?? ()
No symbol table info available.
#15 0x000002b400000004 in ?? ()
No symbol table info available.
#16 0x0000000400000000 in ?? ()
No symbol table info available.
#17 0x00000000000002c7 in ?? ()
No symbol table info available.
#18 0x000002e900000004 in ?? ()
No symbol table info available.
#19 0x0000000400000000 in ?? ()
No symbol table info available.
#20 0x0000000000000337 in ?? ()
No symbol table info available.
#21 0x0000037f00000004 in ?? ()
No symbol table info available.
#22 0x0000000400000000 in ?? ()
No symbol table info available.
#23 0x0000000000000380 in ?? ()
No symbol table info available.
#24 0x0000039b00000004 in ?? ()
No symbol table info available.
#25 0x0000000400000000 in ?? ()
No symbol table info available.
#26 0x00000000000002ab in ?? ()
No symbol table info available.
#27 0x000003ed00000004 in ?? ()
No symbol table info available.
#28 0x0000000400000000 in ?? ()
No symbol table info available.
#29 0x0000000000000743 in ?? ()
No symbol table info available.
#30 0x0000075500000004 in ?? ()
No symbol table info available.
#31 0x0000000400000000 in ?? ()
No symbol table info available.
#32 0x000000000000075c in ?? ()
No symbol table info available.
#33 0x0000076000000004 in ?? ()
No symbol table info available.
#34 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (Thread 0x7f19b4315700 (LWP 1033)):
#0  0x00007f19b54f2cf6 in recv () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (Thread 0x7f19b104d700 (LWP 1037)):
#0  0x00005566e59418eb in http_wait_for_response (s=0x7f1994398150, rep=0x7f19943981c0, an_bit=4194304) at src/http_ana.c:1704
        sess = 0x5566e7e06990
        txn = 0x7f1994103c20
        msg = 0x7f1994103c20
        htx = 0x7f19942bba40
        si_b = 0x7f1994398450
        srv_conn = 0x7f1994398160
        sl = 0x0
        n = 1
#1  0x00005566e5956787 in process_stream (t=0x7f199c0a3ad0, context=0x7f1994398150, state=1028) at src/stream.c:1857
        max_loops = 199
        ana_list = 20971520
        ana_back = 20971520
        flags = 3221225474
        srv = 0x5566e6e673b0
        s = 0x7f1994398150
        sess = 0x5566e7e06990
        rqf_last = 8651264
        rpf_last = 3221225472
        rq_prod_last = 8
        rq_cons_last = 8
        rp_cons_last = 8
        rp_prod_last = 8
        req_ana_back = 32768
        req = 0x7f1994398160
        res = 0x7f19943981c0
        si_f = 0x7f19943983f8
        si_b = 0x7f1994398450
        rate = 4
#2  0x00005566e5a74168 in run_tasks_from_lists (budgets=0x7f19b1042b0c) at src/task.c:483
        process = 0x5566e5954a4d <process_stream>
        tl_queues = 0x5566e5e9dc30 <task_per_thread+688>
        t = 0x7f199c0a3ad0
        budget_mask = 7 '\a'
        done = 0
        queue = 1
        state = 1028
        ctx = 0x7f1994398150
#3  0x00005566e5a74a01 in process_runnable_tasks () at src/task.c:679
        tt = 0x5566e5e9dc00 <task_per_thread+640>
        lrq = 0x0
        grq = 0x0
        t = 0x7f199c0a3ad0
        default_weights = {64, 48, 16}
        max = {0, 195, 0}
        max_total = 48
        tmp_list = 0x0
        queue = 3
        max_processed = 196
#4  0x00005566e5a08716 in run_poll_loop () at src/haproxy.c:2947
        next = 700411331
        wake = 0
#5  0x00005566e5a08c07 in run_thread_poll_loop (data=0x5) at src/haproxy.c:3112
        ptaf = 0x5566e5d6a640 <per_thread_alloc_list>
        ptif = 0x5566e5d6a650 <per_thread_init_list>
        ptdf = 0x0
        ptff = 0x0
        init_left = 0
        init_mutex = {
          __data = {
            __lock = 0, 
            __count = 0, 
            __owner = 0, 
            __nusers = 0, 
            __kind = 0, 
            __spins = 0, 
            __elision = 0, 
            __list = {
              __prev = 0x0, 
              __next = 0x0
            }
          }, 
          __size = '\000' <repeats 39 times>, 
          __align = 0
        }
        init_cond = {
          __data = {
            {
              __wseq = 25, 
              __wseq32 = {
                __low = 25, 
                __high = 0
              }
            }, 
            {
              __g1_start = 15, 
              __g1_start32 = {
                __low = 15, 
                __high = 0
              }
            }, 
            __g_refs = {0, 0}, 
            __g_size = {0, 0}, 
            __g1_orig_size = 20, 
            __wrefs = 0, 
            __g_signals = {0, 0}
          }, 
          __size = "\031\000\000\000\000\000\000\000\017", '\000' <repeats 23 times>, "\024", '\000' <repeats 14 times>, 
          __align = 25
        }
#6  0x00007f19b54e914a in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7  0x00007f19b4412dc3 in glob@@GLIBC_2.27 () from /lib64/libc.so.6
No symbol table info available.
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Additional Information

The build of haproxy has been augmented with -g -ggdb3 -O0 -fno-omit-frame-pointer.

From ed4f74cad83a824565848b0000c2ce4ee058b432 Mon Sep 17 00:00:00 2001
From: Andrew McDermott <amcdermo@redhat.com>
Date: Tue, 13 Dec 2022 06:57:43 +0000
Subject: [PATCH 1/2] Force debug build

---
 haproxy.spec | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/haproxy.spec b/haproxy.spec
index 001a7f3..3db627d 100644
--- a/haproxy.spec
+++ b/haproxy.spec
@@ -10,7 +10,7 @@
 
 Name:           haproxy
 Version:        2.2.19
-Release:        2%{?dist}
+Release:        3%{?dist}
 Summary:        Do not ship, install or use this, use %{real_name} subpackage instead
 
 License:        GPLv2+
@@ -59,7 +59,7 @@ regparm_opts=
 regparm_opts="USE_REGPARM=1"
 %endif
 
-%{__make} %{?_smp_mflags} CPU="generic" TARGET="linux-glibc" USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 ${regparm_opts} ADDINC="%{optflags}" ADDLIB="%{__global_ldflags}"
+%{__make} %{?_smp_mflags} V=1 CPU="generic" TARGET="linux-glibc" USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1 USE_CRYPT_H=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 ${regparm_opts} ADDINC="%{optflags} -g -ggdb3 -O0 -fno-omit-frame-pointer" ADDLIB="%{__global_ldflags}"
 
 %install
 %{__make} install-bin DESTDIR=%{buildroot} PREFIX=%{_prefix} TARGET="linux-glibc"
@@ -93,6 +93,9 @@ fi
 %{_sbindir}/%{name}
 
 %changelog
+* Tue Dec 13 2022 Andrew McDermott <amcdermo@redhat.com> - 2.2.19-3
+- Makefile: enable debug
+
 * Wed Feb 16 2022 Andrew McDermott <amcdermo@redhat.com> - 2.2.19-2
 - Add Set-Cookie2 fix from upstream
 
-- 
2.36.2

[capflam]

I'll wait for your config because at first glance, the only way to make it possible is to have a 1XX messages not properly handled.

[frobware]

I'll wait for your config because at first glance, the only way to make it possible is to have a 1XX messages not properly handled.

Are there other variables from the core file that I could expand/print/display/deref that would help at all?

[capflam]

Well, it could be useful to know the response origin. In your trace, in #0 (http_wait_for_response) frame, the variable n is equal to 1. It means a 1xx message was handled. In the HTX message, there are 5 blocks (.tail = 4 and .head = 0). The last block is the EOM one. Others are the 1xx start-line followed by 2 headers and the end-of-header block. Of course, here the EOM block is unexpected.

You can try to get the backend name (s->be->id), the target type (*s->target). This will help us to know the response origin. You can also try to dump the htx content:

(gdb) call b_reset(&trash)
(gdb) call htx_dump(&trash, htx, 1)
(gdb) printf "%s\n", trash.area

[frobware]

AFAIK, I cannot make calls in gdb because I am not attached to any process. I only have the core dump.

(gdb) p s->be
$1 = (struct proxy *) 0x5566e6e61610
(gdb) p s->be->id
$2 = 0x5566e6e62f40 "be_secure:<redacted>:keycloak"

(gdb) p *s->target
$3 = OBJ_TYPE_SERVER

[capflam]

Thanks. So the response was sent from a server. Do you know if it is a H1, H2 or FCGI server ?

[frobware]

Thanks. So the response was sent from a server. Do you know if it is a H1, H2 or FCGI server ?

I don't - no reproducer and no access logs, and currently no matching config file associated with the core file. If I walk up the stack and inspect other variables can I infer/deduce the server type?

[frobware]

Is there -DDEBUG_XXX flag that would be useful here if it were possible to deploy a binary with additional tracing/debugging enabled? (I am wary of the performance hit as the current fault is occurring in a production cluster.)

[capflam]

I don't - no reproducer and no access logs, and currently no matching config file associated with the core file. If I walk up the stack and inspect other variables can I infer/deduce the server type?

You can try that: p *((struct conn_stream *)s->si[1].end)->conn

[frobware]

You can try that: p *((struct conn_stream *)s->si[1].end)->conn

(gdb) p *((struct conn_stream *)s->si[1].end)->conn
$4 = {
  obj_type = OBJ_TYPE_CONN, 
  err_code = 0 '\000', 
  send_proxy_ofs = 0, 
  flags = 13056, 
  ctrl = 0x5566e5d6d3e0 <proto_tcpv4>, 
  xprt = 0x5566e5d5fa20 <ssl_sock>, 
  mux = 0x5566e5d57de0 <h2_ops>, 
  xprt_ctx = 0x7f198c13bc80, 
  ctx = 0x7f19ac0b9160, 
  owner = 0x5566e7e06990, 
  target = 0x5566e6e673b0, 
  subs = 0x7f198c13bca8, 
  list = {
    next = 0x7f19943aaa38, 
    prev = 0x7f19943aaa38
  }, 
  session_list = {
    n = 0x7f19943aaa48, 
    p = 0x7f19943aaa48
  }, 
  handle = {
    fd = 2509
  }, 
  proxy_netns = 0x0, 
  destroy_cb = 0x0, 
  src = 0x7f199423a550, 
  dst = 0x7f199824b820, 
  proxy_authority = 0x0, 
  idle_time = 0, 
  proxy_authority_len = 254 '\376', 
  proxy_unique_id = {
    ptr = 0x0, 
    len = 0
  }
}

[capflam]

ok, it is a H2 response. I must check the code. But it don't know if a 1XX HEADER frame with a end-stream flag is valid in H2. And how the H2 multiplexer handle this case if it happens. Thanks. I'm digging ...

[capflam]

So indeed, it is invalid and should be rejected.

For a response only, a server MAY send any number of interim responses before the HEADERS frame containing a final response. An interim response consists of a HEADERS frame (which might be followed by zero or more CONTINUATION frames) containing the control data and header section of an interim (1xx) HTTP response (see Section 15 of [HTTP]). A HEADERS frame with the END_STREAM flag set that carries an informational status code is malformed (Section 8.1.1).

But it seems the code does not test this case. I will check tomorrow with @wtarreau to be sure. If I'm right, it will be fixed. Thanks again for your help !

[capflam]

Ok, I pushed the fix. It will be backported as usual. Thanks. Note that this issue reveals a bug with your server that should be fixed.

[frobware]

Many thanks for the super quick fix!

[nickthetait]

This vulnerability is being tracked as CVE-2023-0056.