src/ssl_sock.c: Segmentation fault when using certificates in configuration on windows

[sgnn7]

Detailed Description of the Problem

When running HAProxy 2.4.17 on Windows built/run against OpenSSL 1.0.2 series and using certificates on the frontend, application crashes with a SIGSEGV. Removal of the ssl/cert in the bind command makes the HAProxy start up without issues.

Expected Behavior

HAProxy starts up without a segmentation fault

Steps to Reproduce the Behavior

Pre-requisites:

• Windows
• OpenSSL 1.0.2u dlls built with MSYS2/MinGW-x64
• HAProxy 2.4.17
  • Built using Cygwin and following flags:

    make PREFIX="${dist_dir}" \
       TARGET=cygwin \
       USE_LIBCRYPT= \
       USE_PCRE=  \
       USE_PCRE2=  \
       USE_STATIC_PCRE2=  \
       USE_PCRE2_JIT=  \
       USE_THREAD=0 \
       USE_ZLIB=1 \
       USE_OPENSSL=1 \
       DEBUG_FULL=1 \
       SSL_INC="${ssl_dir_prefix}/include" \
       SSL_LIB="${ssl_dir_prefix}/lib" \
       CFLAGS="-O0 -g -w -fwrapv" \
       LDFLAGS="-Wl,-rpath=${ssl_dir_prefix}/lib" \
       -j8 all
  

• Run ./haproxy.exe -f ./haproxy.cfg with following in the settings (full config provided as part of this issue):

...

frontend foo
  # bind *:3834 # working
  bind *:3834 ssl crt "./certs/openssl_rsa_server.pem" ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384
  mode http
  option tcplog
  option http-server-close
  option forwardfor
  default_backend foo_backend
  monitor-uri /haproxy_test
  ...

Do you have any idea what may have caused this?

Only an educated guess given that this codebase is new to me and that the source file in question is massive:

The context being freed using SSL_CTX_free(bind_conf->default_ctx) here (L2994 in ssl_sock.c) is assumed not to be the same pointer as ckch_inst->ctx but that's not the case:

...
(gdb) p ckch_inst->ctx
$17 = (SSL_CTX *) 0x40f2e0
(gdb) p bind_conf->default_ctx
$18 = (SSL_CTX *) 0x40f2e0

Assumption here from reading the code is that we want to dispose of the old bind_conf->default_ctx which is in actuality also the memory location of ckch_inst->ctx (an incomplete struct that we want to use moving forward) so when SSL_CTX_free(bind_conf->default_ctx) is trying to zeroize this struct, it hits invalid memory pointers. Excerpt from the backtrace:

...
0x000000006318ce8a in x509_verify_param_zero (param=0xfeeefeeefeeefeee) at C:/Users/foo/tmp/haproxy-cert-crash/openssl/openssl-1.0.2u/crypto/x509/x509_vpm.c:138
138         param->name = NULL;

Do you have an idea how to solve the issue?

My guess for the fix patch:

From ae274daea27899bdbebb500510dd2034fdeebf56 Mon Sep 17 00:00:00 2001
From: Srdjan Grubor <sgnn7@sgnn7.org>
Date: Wed, 29 Jun 2022 15:55:54 -0500
Subject: [PATCH] BUG/MEDIUM: ssl: Fix crash when loading certificates from
 file

When using certificates from file and running against OpenSSL, the
assumption that the current and the old context are different pointers
may not be valid. This change ensures that we do not zeroize an
incompletely-initialized SSL context.
---
 src/ssl_sock.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 7f74977e1..52d9bb953 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -2991,7 +2991,7 @@ void ssl_sock_load_cert_sni(struct ckch_inst *ckch_inst, struct bind_conf *bind_
 	}
 
 	/* replace the default_ctx if required with the instance's ctx. */
-	if (ckch_inst->is_default) {
+	if (ckch_inst->is_default && bind_conf->default_ctx != ckch_inst->ctx) {
 		SSL_CTX_free(bind_conf->default_ctx);
 		SSL_CTX_up_ref(ckch_inst->ctx);
 		bind_conf->default_ctx = ckch_inst->ctx;
-- 
2.36.1

What is your configuration?

# Basic configuration

global
    log 127.0.0.1 local0 debug
    ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384

# Some sane defaults
defaults
    log     global
    option  dontlognull
    retries 3
    # option  redispatch
    option  redispatch httplog
    timeout client 5s
    timeout server 5s
    timeout connect 5s

# This declares a view into HAProxy statistics, on port 3833
# You do not need credentials to view this page and you can
# turn it off once you are done with setup.
listen stats
    bind *:3833
    mode http
    stats enable
    stats uri /

# This section is to reload DNS Records
# Replace these addresses with your DNS Server IP addresses.
resolvers my-dns
    nameserver dns1 8.8.8.8:53
    nameserver dns2 1.1.1.1:53
    resolve_retries 3
    timeout resolve 2s
    timeout retry 1s
    accepted_payload_size 8192
    hold valid 10s
    hold obsolete 60s

frontend e2e_request_frontend
#    bind *:3834
    bind *:3834 ssl crt "./certs/openssl_rsa_server.pem" ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384
    mode http
    option tcplog
    option http-server-close
    option forwardfor
    default_backend e2e_request_backend
    monitor-uri /haproxy_test


backend e2e_request_backend
    # default server ssl ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384
    balance roundrobin
    mode http
    server-template mothership 5 10.17.1.10:443 check port 443 ssl verify none check
    # server server1 server:443 check port 443 ssl verify none check
    # server server1 server:443 ssl verify none check resolvers my-dns init-addr none resolve-prefer ipv4 maxconn 20

Output of haproxy -vv

$ ./haproxy -vv
HAProxy version 2.4.17-9f97155 2022/05/13 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.17.html
Running on: CYGWIN_NT-10.0-17763 3.3.5-341.x86_64 2022-05-13 12:27 UTC x86_64
Build options :
  TARGET  = cygwin
  CPU     = generic
  CC      = cc
  CFLAGS  = -O0 -g -w -fwrapv
  OPTIONS = USE_PCRE= USE_PCRE2= USE_PCRE2_JIT= USE_THREAD=0 USE_STATIC_PCRE2= USE_LIBCRYPT= USE_OPENSSL=1 USE_ZLIB=1
  DEBUG   =

Feature list : -EPOLL -KQUEUE -NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE -LIBCRYPT -CRYPT_H -GETADDRINFO +OPENSSL -LUA -FUTEX -ACCEPT4 -CLOSEFROM +ZLIB -SLZ -CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD +OBSOLETE_LINKER -PRCTL -PROCCTL -THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.0.2u 20 Dec 2019
Running on OpenSSL version : OpenSSL 1.0.2u  20 Dec 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using:
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): no
Built with gcc compiler version 11.3.0

Available polling systems :
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 2 (2 usable), will use poll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=

Available services : none

Available filters :
        [SPOE] spoe
        [CACHE] cache
        [FCGI] fcgi-app
        [COMP] compression
        [TRACE] trace

Last Outputs and Backtraces

(gdb) bt
#0  0x000000006318ce8a in x509_verify_param_zero (param=0xfeeefeeefeeefeee) at C:/Users/foo/tmp/haproxy-cert-crash/openssl/openssl-1.0.2u/crypto/x509/x509_vpm.c:138
#1  0x000000006318d0f7 in X509_VERIFY_PARAM_free (param=0xfeeefeeefeeefeee) at C:/Users/foo/tmp/haproxy-cert-crash/openssl/openssl-1.0.2u/crypto/x509/x509_vpm.c:202
#2  0x000000027d140e07 in SSL_CTX_free (a=0x40f410) at C:/Users/foo/tmp/haproxy-cert-crash/openssl/openssl-1.0.2u/ssl/ssl_lib.c:2137
#3  0x000000010040bb6e in ssl_sock_load_cert_sni (ckch_inst=ckch_inst@entry=0x8000cae90, bind_conf=bind_conf@entry=0x8000c6920) at src/ssl_sock.c:2995
#4  0x000000010041030e in ssl_sock_load_ckchs (path=0x80007ee44 "./certs/openssl_rsa_server.pem", ssl_conf=0x0, sni_filter=0x0, fcount=0, err=0xffffc380, ckch_inst=0xffffb198,
    bind_conf=0x8000c6920, ckchs=0x8000c6d80) at src/ssl_sock.c:3539
#5  ssl_sock_load_ckchs (err=0xffffc380, ckch_inst=0xffffb198, fcount=0, sni_filter=0x0, ssl_conf=0x0, bind_conf=0x8000c6920, ckchs=0x8000c6d80,
    path=0x80007ee44 "./certs/openssl_rsa_server.pem") at src/ssl_sock.c:3527
#6  ssl_sock_load_cert (path=0x80007ee44 "./certs/openssl_rsa_server.pem", bind_conf=0x8000c6920, err=0xffffc380) at src/ssl_sock.c:3705
#7  0x000000010051c2b7 in cfg_parse_listen (file=0x80006e230 "haproxy.cfg", linenum=44, args=0xffffc700, kwm=<optimized out>) at src/cfgparse-listen.c:438
#8  0x00000001004dfb0a in readcfgfile (file=0x80006e230 "haproxy.cfg") at src/cfgparse.c:2332
#9  0x00000001005a51cd in init (argv=<optimized out>, argc=<optimized out>) at src/haproxy.c:1834
#10 main (argc=<optimized out>, argv=0xffffcc00) at src/haproxy.c:2921

Additional Information

No response

[wlallemand]

If you wanted to disable threading, that would be USE_THREAD= not USE_THREAD=0 on your make line.

Your problem is quite strange, I couldn't reproduce it with linux/openssl 1.0.1u. the SSL_CTX_free() is supposed to work here, because it uses a refcount. In ckch_inst_new_load_store() the refcount is incremented (SSL_CTX_up_ref) to be used in bind_conf->default_ctx and ckch_inst->ctx. So at this step the refcount is supposed to be set to 2, and the free shouldn't be a problem.

How did you generate your certificate? could you share a openssl x509 -in ./certs/openssl_rsa_server.pem -text -noout ?

[sgnn7]

@wlallemand Thanks for your feedback!

If you wanted to disable threading, that would be USE_THREAD= not USE_THREAD=0 on your make line.

👍 Good point - I think I just made a typo copying things for the test environment

Your problem is quite strange, I couldn't reproduce it with linux/openssl 1.0.1u. the SSL_CTX_free() is supposed to work here, because it uses a refcount. In ckch_inst_new_load_store() the refcount is incremented (SSL_CTX_up_ref) to be used in bind_conf->default_ctx and ckch_inst->ctx. So at this step the refcount is supposed to be set to 2, and the free shouldn't be a problem.

Hmm interesting - the SSL_CTX_free for sure proceeds to zeroize when it shouldn't. Let me do some more debugging on this in my test environment while paying attention to ref counts. It could be that there is some difference in the OS or possibly around OpenSSL build flags. I'll ping back when I find out more on this.

How did you generate your certificate? could you share a openssl x509 -in ./certs/openssl_rsa_server.pem -text -noout ?

I will see if I can get the cert itself since I'm just using it for testing right now anyways.

[wlallemand]

It could be something related to mingw or your compilation flags, the pointers in your stack trace are quiet strange. (param=0xfeeefeeefeeefeee and a=0x40f410). Also you overwrote the CFLAGS, you could have provoked something by changing them.

We are building for Windows on the github CI, (but not running the reg-test so we could have some problems). You could check https://github.com/haproxy/haproxy/blob/master/.github/workflows/windows.yml
Maybe try not to rewrite the CFLAGS, and use ADDLIB="-Wl,-rpath=${ssl_dir_prefix}/lib" instead of changing the LDFLAGS.

[sgnn7]

@wlallemand I've confirmed that the issue only occurs on Windows and not on Linux (Ubuntu 20.04.4 LTS). I'll try the suggested changes to the flags (USE_THREAD and CFLAGS) next.

Here's the output of the test cert openssl x509 -in ./certs/openssl_rsa_server.pem -text -noout:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 18097278524571699240 (0xfb2672ef4266c828)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=NY, L=New York, O=Test, OU=Foo, CN=www.bar.com
        Validity
            Not Before: Jul  1 19:10:55 2022 GMT
            Not After : Jun 28 19:10:55 2032 GMT
        Subject: C=US, ST=NY, L=New York, O=Test, OU=Foo, CN=www.bar.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:c3:ca:1b:56:3b:d8:a0:80:fd:f8:5e:44:c1:0a:
                    49:8e:2a:56:eb:40:99:6b:d3:6d:34:56:b7:8f:3a:
                    c0:1a:17:db:9a:1a:d1:42:e9:19:e0:ef:51:ff:f5:
                    cd:d7:53:9a:c0:f1:7a:92:ae:73:17:98:65:e6:64:
                    29:87:7f:74:f0:cc:12:b9:60:3a:bd:72:21:f8:7c:
                    c3:f7:7b:4e:15:cc:91:2d:ba:1b:a3:45:dd:42:44:
                    08:38:39:9a:be:ab:66:d0:01:ca:d6:7d:b8:e5:ee:
                    33:25:25:5c:70:20:ee:39:51:ad:78:a2:2b:7a:a1:
                    89:2f:f5:e3:59:d4:ac:15:61:60:29:b4:01:f3:4b:
                    21:70:bf:1c:65:cb:99:ab:72:96:02:6f:1d:73:5d:
                    5e:21:ea:f7:78:27:28:c0:08:b7:15:e8:12:37:13:
                    87:af:ce:8f:01:95:83:35:fb:ad:02:3c:33:5e:15:
                    00:db:4e:fe:35:11:94:ff:f1:1f:e2:8c:67:db:9c:
                    57:30:17:0c:21:e2:20:03:45:5c:c1:09:53:01:63:
                    ba:38:65:e8:06:27:c0:8f:ea:0d:30:de:12:c4:68:
                    c7:51:25:b9:d2:87:77:41:ba:29:e9:46:cc:e4:63:
                    86:ea:7d:a8:2a:7e:56:04:78:a4:a4:67:95:eb:8a:
                    25:22:80:5c:0c:bc:54:d8:b1:7d:7a:81:fa:c0:1c:
                    c5:b1:59:6f:4c:60:61:af:9b:ec:2c:99:b7:c8:bd:
                    2b:09:4e:a8:af:f0:f7:7b:38:35:73:a3:20:9e:85:
                    67:1a:22:57:60:11:6c:d1:10:60:13:7f:90:5b:c5:
                    d1:9b:07:d0:ab:bf:8f:09:02:4d:15:64:cb:b9:72:
                    40:3f:89:ec:8a:62:cc:60:71:a8:a2:22:06:dd:7c:
                    d3:91:ba:2b:1e:45:19:be:d1:29:31:5b:ba:a2:5a:
                    b8:76:14:24:e2:15:cc:16:a1:9e:e1:bc:59:4e:f7:
                    a5:20:3d:f4:72:a9:c7:8a:29:43:e0:0b:da:fe:1b:
                    b4:a5:88:42:38:98:0c:01:37:8b:5e:34:a8:52:dd:
                    19:47:29:2d:bb:23:04:e4:e8:dc:9e:bc:eb:d5:91:
                    dc:e6:3c:8d:97:e9:a1:c3:d7:a8:71:a6:ef:58:a0:
                    70:bc:54:82:39:91:f6:e6:d6:f0:4a:51:00:fe:75:
                    6b:cf:84:d7:9f:14:31:da:eb:e3:bd:13:06:89:d6:
                    3b:86:6c:bd:1f:cf:f8:64:12:b1:23:e6:2e:5a:09:
                    f2:f0:68:17:10:5b:4b:02:91:f3:9e:ec:03:cc:7d:
                    2e:91:d2:8c:06:33:eb:8a:b0:9e:02:48:61:f8:c6:
                    24:4f:ad
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         7b:16:f2:a7:3c:bf:26:92:35:a4:b8:23:6a:c8:7b:61:ba:e1:
         78:09:d7:17:8a:0a:43:16:33:66:8f:db:d7:40:e5:68:48:cc:
         6b:4a:08:08:30:4a:9c:17:c9:a8:bb:4f:fa:0d:9e:23:2f:45:
         e0:4a:d1:94:81:74:67:d7:5a:b7:99:10:72:f3:50:1a:3b:c7:
         b1:70:e4:27:6f:e9:a2:9b:65:31:d0:b8:aa:16:da:d0:af:eb:
         78:cf:74:a5:81:ca:cb:40:cb:80:3f:51:66:18:b8:c3:79:48:
         c0:be:7d:15:83:4b:08:b5:e5:69:b3:6c:e5:16:9a:4f:ce:da:
         a3:85:9a:5a:66:60:0a:e4:5c:48:e6:c1:7e:f2:a9:62:a5:b5:
         d9:f2:d0:a0:46:b8:96:ec:7f:2d:c6:45:e0:30:cf:c7:94:c8:
         41:ae:6e:6a:66:ff:d5:9f:f9:9f:25:c2:4a:5b:68:97:ae:49:
         60:64:8d:01:26:8b:0c:ae:12:21:78:20:24:51:81:fc:72:0d:
         4b:75:8d:f1:20:6b:80:14:e1:1d:04:5f:da:3b:8e:dd:2e:61:
         39:e0:e1:56:83:5c:3c:a8:19:dc:fa:93:04:c0:a5:22:e4:e7:
         f1:fe:48:dc:4d:68:b6:0c:58:e7:db:6b:8e:b9:79:ab:df:f4:
         21:59:41:28:4f:7e:b5:c3:75:e8:28:ac:c2:f1:4d:51:ed:df:
         4c:cf:d9:20:8b:8d:36:bb:56:a2:a7:d2:9a:fb:28:7f:03:b6:
         fc:fb:2a:e7:8f:9f:c2:d3:42:38:e8:85:b9:dc:b1:11:45:13:
         ab:e7:5c:ad:d3:09:72:f1:01:ed:f4:1e:e0:65:85:24:06:f1:
         75:d7:23:e4:53:eb:f4:c5:32:13:27:71:22:88:8c:a3:08:13:
         b3:cf:11:24:61:af:c7:b0:4b:d9:64:2a:2d:16:53:c9:1a:5e:
         56:6f:ab:0a:91:83:f1:41:b0:cb:fb:7d:05:27:b9:7d:12:11:
         d0:b3:d2:e3:e2:da:73:4e:22:6b:22:1a:9f:d8:09:e7:56:f6:
         d6:c5:0d:ba:45:c0:29:63:a9:23:bf:95:70:43:cb:e4:57:08:
         6f:6f:23:d8:3f:e0:d3:f2:df:29:52:39:0b:6e:38:55:41:6a:
         5d:d2:8b:53:f6:9a:8d:8a:e9:23:47:ee:30:66:6e:b5:96:db:
         42:c9:39:13:89:e4:bc:38:6b:65:ab:ef:d8:b2:12:1d:cb:dc:
         90:5a:1f:82:01:77:31:0c:7b:37:c8:f0:3b:20:67:d3:96:42:
         5b:1c:b0:3e:b3:93:de:a0:17:26:c1:83:ef:8d:0e:54:89:e2:
         c9:bc:e0:c9:ee:0c:c5:99

That test cert content (not used for anything so it's fine to add here):

-----BEGIN CERTIFICATE-----
MIIFPDCCAyQCCQD7JnLvQmbIKDANBgkqhkiG9w0BAQsFADBgMQswCQYDVQQGEwJV
UzELMAkGA1UECAwCTlkxETAPBgNVBAcMCE5ldyBZb3JrMQ0wCwYDVQQKDARUZXN0
MQwwCgYDVQQLDANGb28xFDASBgNVBAMMC3d3dy5iYXIuY29tMB4XDTIyMDcwMTE5
MTA1NVoXDTMyMDYyODE5MTA1NVowYDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5Z
MREwDwYDVQQHDAhOZXcgWW9yazENMAsGA1UECgwEVGVzdDEMMAoGA1UECwwDRm9v
MRQwEgYDVQQDDAt3d3cuYmFyLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC
AgoCggIBAMPKG1Y72KCA/fheRMEKSY4qVutAmWvTbTRWt486wBoX25oa0ULpGeDv
Uf/1zddTmsDxepKucxeYZeZkKYd/dPDMErlgOr1yIfh8w/d7ThXMkS26G6NF3UJE
CDg5mr6rZtABytZ9uOXuMyUlXHAg7jlRrXiiK3qhiS/141nUrBVhYCm0AfNLIXC/
HGXLmatylgJvHXNdXiHq93gnKMAItxXoEjcTh6/OjwGVgzX7rQI8M14VANtO/jUR
lP/xH+KMZ9ucVzAXDCHiIANFXMEJUwFjujhl6AYnwI/qDTDeEsRox1EludKHd0G6
KelGzORjhup9qCp+VgR4pKRnleuKJSKAXAy8VNixfXqB+sAcxbFZb0xgYa+b7CyZ
t8i9KwlOqK/w93s4NXOjIJ6FZxoiV2ARbNEQYBN/kFvF0ZsH0Ku/jwkCTRVky7ly
QD+J7IpizGBxqKIiBt1805G6Kx5FGb7RKTFbuqJauHYUJOIVzBahnuG8WU73pSA9
9HKpx4opQ+AL2v4btKWIQjiYDAE3i140qFLdGUcpLbsjBOTo3J6869WR3OY8jZfp
ocPXqHGm71igcLxUgjmR9ubW8EpRAP51a8+E158UMdrr470TBonWO4ZsvR/P+GQS
sSPmLloJ8vBoFxBbSwKR857sA8x9LpHSjAYz64qwngJIYfjGJE+tAgMBAAEwDQYJ
KoZIhvcNAQELBQADggIBAHsW8qc8vyaSNaS4I2rIe2G64XgJ1xeKCkMWM2aP29dA
5WhIzGtKCAgwSpwXyai7T/oNniMvReBK0ZSBdGfXWreZEHLzUBo7x7Fw5Cdv6aKb
ZTHQuKoW2tCv63jPdKWBystAy4A/UWYYuMN5SMC+fRWDSwi15WmzbOUWmk/O2qOF
mlpmYArkXEjmwX7yqWKltdny0KBGuJbsfy3GReAwz8eUyEGubmpm/9Wf+Z8lwkpb
aJeuSWBkjQEmiwyuEiF4ICRRgfxyDUt1jfEga4AU4R0EX9o7jt0uYTng4VaDXDyo
Gdz6kwTApSLk5/H+SNxNaLYMWOfba465eavf9CFZQShPfrXDdegorMLxTVHt30zP
2SCLjTa7VqKn0pr7KH8Dtvz7KuePn8LTQjjohbncsRFFE6vnXK3TCXLxAe30HuBl
hSQG8XXXI+RT6/TFMhMncSKIjKMIE7PPESRhr8ewS9lkKi0WU8kaXlZvqwqRg/FB
sMv7fQUnuX0SEdCz0uPi2nNOImsiGp/YCedW9tbFDbpFwCljqSO/lXBDy+RXCG9v
I9g/4NPy3ylSOQtuOFVBal3Si1P2mo2K6SNH7jBmbrWW20LJOROJ5Lw4a2Wr79iy
Eh3L3JBaH4IBdzEMezfI8DsgZ9OWQlscsD6zk96gFybBg++NDlSJ4sm84MnuDMWZ
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQDDyhtWO9iggP34
XkTBCkmOKlbrQJlr0200VrePOsAaF9uaGtFC6Rng71H/9c3XU5rA8XqSrnMXmGXm
ZCmHf3TwzBK5YDq9ciH4fMP3e04VzJEtuhujRd1CRAg4OZq+q2bQAcrWfbjl7jMl
JVxwIO45Ua14oit6oYkv9eNZ1KwVYWAptAHzSyFwvxxly5mrcpYCbx1zXV4h6vd4
JyjACLcV6BI3E4evzo8BlYM1+60CPDNeFQDbTv41EZT/8R/ijGfbnFcwFwwh4iAD
RVzBCVMBY7o4ZegGJ8CP6g0w3hLEaMdRJbnSh3dBuinpRszkY4bqfagqflYEeKSk
Z5XriiUigFwMvFTYsX16gfrAHMWxWW9MYGGvm+wsmbfIvSsJTqiv8Pd7ODVzoyCe
hWcaIldgEWzREGATf5BbxdGbB9Crv48JAk0VZMu5ckA/ieyKYsxgcaiiIgbdfNOR
uiseRRm+0SkxW7qiWrh2FCTiFcwWoZ7hvFlO96UgPfRyqceKKUPgC9r+G7SliEI4
mAwBN4teNKhS3RlHKS27IwTk6NyevOvVkdzmPI2X6aHD16hxpu9YoHC8VII5kfbm
1vBKUQD+dWvPhNefFDHa6+O9EwaJ1juGbL0fz/hkErEj5i5aCfLwaBcQW0sCkfOe
7APMfS6R0owGM+uKsJ4CSGH4xiRPrQIDAQABAoICAFIQCyc4zFJx3oFUmcLDmhLx
Yh8Wo3eTValjU2rAODZ3m37bEKzX4RXrmVYMY+uFyk5mqHaEkg7ovIU1pEZ7wLsQ
/iQe1mADT4PEX6sCTrdKV+tvnxfvN48Hzt4qHtceJmi255rBdVdCYMOIBlelJdR2
jJwMdObggAzPJizHvBz2BIVWQp8D9Y85OyDblpxK2d9JTXHGR6c1weXlm+npfTu8
Wq4lDHE+xafnT+i3hUtziiA783D4igoaRXaFTVT6fqmIvO6WKmqlSb+1OexbCSSC
9VGgPQEfdlGby02ZNwqF1E0J3/oC95RgACMbntN3d9wfMPYmn80d9/xEwozICUnQ
av+FXzmdyGuvzRln9g/6fUsngc7DYoVkXB1niwryCNP1SjSHOHQaZ18cJB9Lcgwv
+7BGocODjtk3YxT5DCIAoM+yL7F9+teWOwivgBLBxu03om8dIelmqpmXXpYFnlp5
qJRyM7Zh6eB4obi2+Pbq9syQJPje2y6lKYtPyfjmWiTdi2vH2Eo0Jur4hRMJywgJ
KhGtSS6U6l0ONAUEf/Qz0bG+2NqDc0nXkKhlmrvaq8o2WN0zDOkejQ5lKlpbJs14
BHg4q7Ld4Ox1Ku/+UIjS4+9hOJit6KeXW2FBQq6iQyc4l+2bYMSa7Xm+35mTg4KP
vDHMoOJ7aIyV809BNfJhAoIBAQD/jtLke6vanbwajCieDOodf+G2b+uP+YX0vk/b
zUDB8rLdGCfsuP5QurDvxn6qgAXu6iCcgjh5dT7LUY/5f7O78AfeNbTUXl1fWTYF
1GD/75j6t0mHduiwfbPb7cJtM6onDrO6c5eBZqB5guo2qaPfhyGamMFa0rCrPhEe
WlWMBn2rIq3E/zBePUDGA62ql/k0ZNwW1trnoUfWoTd+2NJD15YQAP60gcz5h7pU
aYr9WggqkQ2MlsWyVBvUrLC3EsXmhdHZQ1uGrDOUMzGA8dYVMksWpGXTPz3Cs5HQ
XdW7xNRUY1m+JfGGw2cOeS6hUBS+MI+hV6SWFMhYLFZHraJ1AoIBAQDEINBhECkv
nWcKQbToNxsfvawGJyCWphVOclcZmLwPiD0e7IOo2//xYHF/o+lkr66WtVkiYpC0
1e02oJ8DO/sJMirMsqUJzXB+Da1P71Ifn93HopRH+yxYz0esfXIr09nIfB8yUSFL
QjYOn2IsfX4QQkVbIyK/Ei+/0UhES/mIxga0LwFTHsABdZqCMO04/8qLp9/xWsa2
Da0aQjCl2M9pJyI18f9xU6TjKVS6jAaWbOyb6ATIcMu8A8hVAXtfhe3J64Pjxu30
zcZgPcg9JX6qaQtGES22lxT1knIMPAwcbyDEX0kFmGfzp9aaU0gxK6nZPGyHFvDP
I+T3hsAnRuFZAoIBAQCTPzbSQVO+zf2xCirpDiXiw2lJFemTdkExyGn5CqWSWoWm
kKGIavrVhwwly6f2E3fXtfz+UKW7jcuVmLQtHOBxnBbcePNtHcYNBzjyWPFe5i2t
3+e07tWZw4PbZ34ML6WNhZIXDz0vaPNzr/PFdS8fvxukmPlnnFgBAu0YIRKDcMX9
0ihvhl3XHtHwhfQNhC0pfCe33SoaJjpfN1rjcoiMYG5tYDVhyhxRdlvtATb9TV/B
LvywkbrtSjsS6kGT6nQuo9JKrd3RlezOKkUOt8kTQ3wyWpz0/85VHeQZO8fHvdxd
lzUBM5mbfYEmvLoaZirdfttzKi2B8A2TiFBaUnh1AoIBACyma8otK7MzWmnzW4Qi
R4KKY8di0QB6/w7E6R9iri27Dm8j2TVwIGyLUEuGvXO2q0CUOJ6jYd5JdZ+iIZFS
7mvBiAzh/sCkjJ+l7XbtPP0GIC96y9c3T3lId0DFrmBb8pC58UA7SCEiCbx6fJc0
WqbI75E4BokJUuWQd6tUwJJqwsvyh5KsxXmPlIVfLiJz2WDB7zG7QNHhey8nMnwP
R+jgh+FQHp9mM3ujxS5TAozsb1FvGOepBjjfg5QWqkes7d2ySGxqtXGuByOR7Sw+
hP+2umJRNt8CtjrSBDTU1s0CX6tVJgKoF8JYy3qGy0CmeByErRB0II6H1XCktMo8
UMECggEBAPi0J8liLDxQIlA736hYsXXPQcl47AgxE9vveY/nVmREqiwb+zdkmwGe
+xEN7KcWD0KHNbLcjcB/82PbPqAcrZtLZzFa2az28iRavgA17fkw7kghhh3v6bZX
PyFbd2Ogi1B5cmay8zZHzeXe5jvLZ2+99RD+M/mf+eUBRTvJx1HT4Ox0p6+Z8t6y
5iKuNKMh/2WALrCbhd5D5n5mBgUrFJZ1c6zFHFbIdlms2RpXEwF7P/aYM7WjlyzJ
47/HqbtjYGQZ+DlkKs8dCjfHB5qSsj8GkqteBePVedFD4JLMWiZ2ruO3A5VnVFIG
ZShbp6suPVlOybMOiSTvMzCcorPY9t0=
-----END PRIVATE KEY-----

Generated using the attached command and appending the .key to the .crt to make the .pem:

openssl req -newkey rsa:4096 \
            -x509 \
            -sha256 \
            -days 3650 \
            -nodes \
            -out server.crt \
            -keyout server.key \
            -subj "/C=US/ST=NY/L=New York/O=Test/OU=Foo/CN=www.bar.com"

[sgnn7]

USE_THREAD= not USE_THREAD=0 on your make line

Change makes the GDB debugging easier but same crash still occurs

... [don't] rewrite CFLAGS

No effect - same crash still happens

use ADDLIB="-Wl,-rpath=${ssl_dir_prefix}/lib" instead of changing the LDFLAGS

No effect - same crash still happens

0xfeeefeeefeeefeee pattern

From what I can tell this is previously-freed memory.

---

I'm still trying a few more things and I still need to investigate the ref counts.

[sgnn7]

For reproducibility, OpenSSL 1.0.2u shared libs are built using this command in MSYS2 MinGW x64:

    ./config --prefix="${dist_dir}" \
             --openssldir="${ssl_dir_prefix}/${SSL_ARTIFACTS_DIR}" \
             -no-asm -no-ssl2 -no-ssl3 -no-comp \
            -ldl -Wl,--disable-dynamicbase \
             shared zlib
  make depend
  make install_sw

And this relevant patch (for debugging and building in MINGW):

--- a/Configure
+++ b/Configure
@@ -610,7 +610,7 @@ my %table=(
 # handling, one can't seriously consider its binaries for using with
 # non-mingw64 run-time environment. And as mingw64 is always consistent
 # with itself, Applink is never engaged and can as well be omitted.
-"mingw64", "gcc:-mno-cygwin -DL_ENDIAN -O3 -Wall -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE::-D_MT:MINGW64:-lws2_32 -lgdi32 -lcrypt32:SIXTY_FOUR_BIT RC4_CHUNK_LL DES_INT EXPORT_VAR_AS_FN:${x86_64_asm}:mingw64:win32:cygwin-shared:-D_WINDLL:-mno-cygwin:.dll.a",
+"mingw64", "gcc:-mno-cygwin -fomit-frame-pointer --debug -fno-inline -DL_ENDIAN -O0 -g -Wall -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -DDEBUG_FINGERPRINT_PREMAIN::-D_MT:MINGW64:-lws2_32 -lgdi32 -lcrypt32 -lbcrypt:SIXTY_FOUR_BIT RC4_CHUNK_LL DES_INT EXPORT_VAR_AS_FN:${x86_64_asm}:mingw64:win32:cygwin-shared:-D_WINDLL:-mno-cygwin:.dll.a",

[sgnn7]

@wlallemand I've updated the thread here but just a minor update that it's OpenSSL 1.0.2u as the version showing the issue, not 1.0.1u (I misremembered the version from the script constant).

[wlallemand]

Are you able to try with an openssl version still maintained? 1.0.2 is out of support for more than 2 years, I don't know if you are facing an OpenSSL bug or something related to your toolchain, or a combination of the two. But you might want to try a supported version like the 1.1.1.

Also, according to our CI there is a package openssl in msys2, maybe you should try to use it instead of trying to build it. https://packages.msys2.org/package/mingw-w64-x86_64-openssl

[sgnn7]

Hey @wlallemand,

you might want to try a supported version like the 1.1.1.

I fully understand where you are coming from but sadly OpenSSL 1.0.2 is an internal requirement that isn't flexible so a switch to a different version isn't feasible at this time. However, given that I suspect this bug is not related to OpenSSL itself, I would think that we would surface the same issue with a newer library.

Also, according to our CI there is a package openssl in msys2, maybe you should try to use it instead of trying to build it.

One of the targets of the built artifacts is a container so it won't have any of the toolchains and package managers plus there's a few things that don't work in OpenSSL with default flags that upstream maintainers set so a self-contained build is preferable.

---

@wlallemand I will try to reproduce this bug with OpenSSL 1.1.1 dev package from MSYS2 on Windows. If I can manage to hit the same issue with a supported version from upstream build, I think at that point we would know for sure if it's an issue in HAProxy or not.

As a follow-up question, do you know off the top of your head if HAProxy handles various versions of OpenSSL differently in the code or is it all using the generic API?

[wlallemand]

We made the code the same for loading files, but we had to copy some of the missing functions for old versions. For example the SSL_CTX_up_ref() was copied. (It only does a CRYPTO_add).

https://github.com/haproxy/haproxy/blob/master/include/haproxy/openssl-compat.h#L289

What's odd is that this part was heavily used on a lot of plateforms. I checked the openssl 1.0.2u code and I don't see any exception for windows in the SSL_CTX_free() code.

We could have a real bug in our code which is easily triggered on windows, but I can't see where the problem could be by reading the code. I'll try to reproduce your build once I have access to a windows machine.

[wlallemand]

I just installed the whole thing on a win10 pro. Using the same haproxy version and gcc-11.3.0 from msys2.

make line: make -j8 TARGET=cygwin USE_OPENSSL=1 USE_THREAD= DEBUG_FULL=1

With the openssl 1.1.1 provided by msys2 and seems to run correctly.

[sgnn7]

Thank you for the replies - they're much appreciated!

In the meantime since my last message I also built HAProxy against:

• MSYS2 pre-built OpenSSL 1.1.1o package artifacts like you did
• Manually compiled upstream OpenSSL 1.1.1p source artifacts

Both seem to not crash when loading a frontend certificate like OpenSSL v1.0.2u does.

Because there is the compatibility layer that you linked (openssl-compat.h) which I still need to look into, I think we are down to either OpenSSL 1.0.2u at fault or the compatibility layer. I'll keep investigating 1.0.2 specifically and let you know what I find. If you have any other ideas around where to look for clues, I'd be very interested to hear them.

[wlallemand]

I spend some time trying to figure out if there was a problem with your openssl-1.0.2 build.

I rebuilt an old msys2 package using a commit which still had openssl-1.0.2 (it was 1.0.2p msys2/MINGW-packages@feb0f5b ) so I used the whole msys2 toolchain without tinkering the build options... and I've got the same segfault at the same place.

At least I can try to debug it now. :-)

I tried to swap the lines this way to be sure we do the SSL_CTX_up_ref() before the free, so it's always incremented before:

$ git diff
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 7f74977e1..8d0bc0e49 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -2992,8 +2992,9 @@ void ssl_sock_load_cert_sni(struct ckch_inst *ckch_inst, struct bind_conf *bind_

        /* replace the default_ctx if required with the instance's ctx. */
        if (ckch_inst->is_default) {
-               SSL_CTX_free(bind_conf->default_ctx);
+               printf("ckch_inst: %p, default_ctx:%p\n", ckch_inst->ctx, bind_conf->default_ctx);
                SSL_CTX_up_ref(ckch_inst->ctx);
+               SSL_CTX_free(bind_conf->default_ctx);
                bind_conf->default_ctx = ckch_inst->ctx;
        }
 }

And... it still crashed in the SSL_CTX_free()... So I removed the first SSL_CTX_free() which is done in ckch_inst_new_load_store() which made it worked. So it make me think something is wrong with the SSL_CTX_up_ref()

I patched the SSL_CTX_up_ref() to displays the counter and I got this:

SSL_CTX_up_ref: ctx: 0x40d2e0 ref: 1
SSL_CTX_up_ref: ctx: 0x40d2e0 ref: 2
SSL_CTX_up_ref: ctx: 0x40d2e0 ref: 3
ckch_inst: 0x40d2e0, default_ctx:0x40d2e0
SSL_CTX_up_ref: ctx: 0x40d2e0 ref: -17891601

What is really odd is that the SSL_CTX_new() creates an object with a references set to 1. So it's not supposed to be 1 after the SSL_CTX_up_ref, I checked with gdb and it's indeed created with 0... It does not make sense at all, at this point I don't trust anything anymore.