Segmentation fault when parsing configuration file on x86_64 linux-musl

[mhameed]

Detailed Description of the Problem

Originally reported here:
https://gitlab.alpinelinux.org/alpine/aports/-/issues/16184

bisecting commits between v2.8.0 and v3.0.0, it seems that commit:7a21c3a4ef1577985621953a79e2776ef97837ab
introduced this bug.

Expected Behavior

Expected haproxy to parse and print messages without segfaulting.

Steps to Reproduce the Behavior

On an alpine 3.20 vm, install dependancies:

    apk add \
      alpine-sdk bash curl gdb git openssl-dev \
      lua5.4-dev linux-headers zlib-dev pcre2-dev

place dhparams and config file into directory:

    mkdir -p /etc/haproxy/certs
    curl https://ssl-config.mozilla.org/ffdhe2048.txt > /etc/haproxy/dhparam.txt
    curl https://gitlab.alpinelinux.org/alpine/aports/uploads/95257038a5b05c1db169dbcf89f71fe0/haproxy.cfg > /etc/haproxy/haproxy.cfg

Clone the haproxy git repo, checkout tag v2.8.0, and compile:

    make clean
    make TARGET=linux-musl USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_NS=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1  LUA_INC=/usr/include/lua5.4/ LUA_LIB=/usr/lib/lua5.4
    ./haproxy -dr -V -c -f /etc/haproxy/haproxy.cfg # no segmentation fault.

Now checkout tag v3.0.0, and recompile and rerun the config file check, and you will observe a segmentation fault.

Doing a git bisect to locate the commit that introduced this issue seems to point at commit:7a21c3a4ef1577985621953a79e2776ef97837ab

Thanks

Do you have any idea what may have caused this?

7a21c3a

Do you have an idea how to solve the issue?

No response

What is your configuration?

global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    maxconn     1000
    user        haproxy
    group       haproxy
    daemon
    stats socket /var/lib/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 2m
    # intermediate configuration
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    # curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam.pem
    ssl-dh-param-file /etc/haproxy/dhparam.txt

resolvers google
    nameserver dns1 8.8.8.8:53
    accepted_payload_size 8192

defaults http_with_logging
    mode http
    log global
    option httplog
    option dontlognull
    option  log-health-checks

defaults http_with_forward from http_with_logging
    option forwardfor       except 127.0.0.0/8
    option redispatch
    retries 3
    timeout http-request 10s
    timeout queue 5s
    timeout connect 5s
    timeout client 50s
    timeout server 3s
    timeout http-keep-alive 3s
    timeout check 5s
    timeout tunnel  90s    # long enough for websocket pings every 55 seconds

defaults tcp_with_logging
    mode tcp
    log global
    option  log-health-checks
    timeout queue 5s
    timeout connect 10s
    timeout client 1m
    timeout server 1m

frontend fe_etcd from tcp_with_logging
    bind *:2379 
    log global
    default_backend bk_etcd

backend bk_etcd from tcp_with_logging
    log global
    option httpchk
    http-check connect ssl
    http-check send meth GET uri /health
    http-check expect status 200
    server etcd1 etcd1.in.mutable.host:2379 check verify none
    server etcd2 etcd2.in.mutable.host:2379 check verify none
    server etcd3 etcd3.in.mutable.host:2379 check verify none

frontend stats from http_with_forward
   bind *:9000
   http-request use-service prometheus-exporter if { path /metrics }
   stats enable
   stats uri /stats
   stats refresh 30s

frontend https_in from http_with_forward
    bind *:443 ssl alpn h2,http/1.1 strict-sni crt /etc/haproxy/certs/
    http-request del-header X-Forwarded-For
    http-request del-header X-Forwarded-Proto
    use_backend hugo_back if { hdr(host) -i hugo.mutable.com }

backend hugo_back from http_with_forward
    disabled
    option httpchk
    http-check send hdr User-Agent "haproxy" meth GET uri /
    http-check expect status 200
    no option http-server-close
    http-request add-header X-Forwarded-Proto https
    server ho ho.in.mutable.host:80 check-send-proxy send-proxy

Output of haproxy -vv

HAProxy version 3.0.0 2024/05/29 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.0.html
Running on: Linux 6.6.32-0-virt #1-Alpine SMP PREEMPT_DYNAMIC Fri, 24 May 2024 10:11:26 +0000 x86_64
Build options :
  TARGET  = linux-musl
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_NS=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = 

Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=1).
Built with OpenSSL version : OpenSSL 3.3.1 4 Jun 2024
Running on OpenSSL version : OpenSSL 3.3.1 4 Jun 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with zlib version : 1.3.1
Running on zlib version : 1.3.1
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.43 2024-02-16
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 13.2.1 20240309

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

(gdb) t a a bt full

Thread 1 (LWP 1359):
#0  0x0000562d8acb7e3c in lf_expr_deinit (expr=expr@entry=0x7f0de23d21c0) at src/log.c:3491
        __ret = 0x7f0de23d21c0
        __n = 0x7f0de2380f10
        __p = 0x7f0de2380f10
#1  0x0000562d8acce854 in free_tcpcheck_http_hdr (hdr=0x7f0de23d21b0) at src/tcpcheck.c:84
No locals.
#2  free_tcpcheck_http_hdr (hdr=0x7f0de23d21b0) at src/tcpcheck.c:79
No locals.
#3  free_tcpcheck_http_hdrs (hdrs=hdrs@entry=0x7f0de23cd728) at src/tcpcheck.c:98
        hdr = 0x7f0de23d21b0
        bhdr = 0x7f0de23cd6d8
#4  0x0000562d8accea2b in free_tcpcheck (rule=0x7f0de23cd680, in_pool=0) at src/tcpcheck.c:130
No locals.
#5  0x0000562d8accec22 in free_tcpcheck (in_pool=0, rule=<optimized out>) at src/tcpcheck.c:108
        __ptr = <optimized out>
#6  deinit_tcpchecks () at src/tcpcheck.c:3780
        rs = 0x7f0de23c6640
        r = <optimized out>
        rb = 0x7f0de23d2790
        node = 0x7f0de23c6650
        next = 0x7f0de222b300
#7  0x0000562d8ad6b0ed in deinit () at src/haproxy.c:2949
        p = <optimized out>
        p0 = <optimized out>
        wl = <optimized out>
        wlb = <optimized out>
        uap = <optimized out>
        ua = <optimized out>
        log = <optimized out>
        logb = <optimized out>
        bol = <optimized out>
        bolb = <optimized out>
        pdf = 0x7f0de2a718a0
        pdfb = <optimized out>
        pxdf = <optimized out>
        pxdfb = <optimized out>
        srvdf = <optimized out>
        srvdfb = <optimized out>
        tif = <optimized out>
        tifb = <optimized out>
        tdf = <optimized out>
        tdfb = <optimized out>
        taf = <optimized out>
        tafb = <optimized out>
        tff = <optimized out>
        tffb = <optimized out>
        pscf = <optimized out>
        pscfb = <optimized out>
        pcf = <optimized out>
        pcfb = <optimized out>
        ppcf = <optimized out>
        ppcfb = <optimized out>
        prcf = <optimized out>
        prcfb = <optimized out>
        pprs = <optimized out>
        pprsb = <optimized out>
        cur_fd = <optimized out>
#8  0x0000562d8ad6b7f2 in deinit_and_exit (status=status@entry=0) at src/haproxy.c:3052
No locals.
#9  0x0000562d8abffb50 in main (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:3996
        err = <optimized out>
        retry = <optimized out>
        limit = {rlim_cur = 4096, rlim_max = 4096}
        pidfd = <optimized out>
        intovf = <optimized out>
        msg = <optimized out>
(gdb)

Additional Information

No response

[Darlelet]

Not cool indeed

Thanks for filing an issue, I bet it has to do something with TCP check expr migration (through lf_expr_xfer()), I think we have all the necessary material to move forward, I'll have a look

[Darlelet]

I'm able to reproduce, the bug may only be triggered during soft-stop or config check (-c) if a logformat expression is used for tcpchecks/httpchecks from defaults section or from a proxy that is disabled. In this case, lf_expr may still belong to proxy lf_checks parent list while the proxy is already freed.

Here it happens with tcpchecks because tcpchecks are cleaned up after the proxy is freed. It wasn't an issue prior to 7a21c3a because there was not dependency to proxy from tcpcheck. (tcpcheck expr migration in itself is fine)

I'll provide a fix, thanks again for reporting this.