HAProxy 3.0.5 integrated with openssl 1.1.1w does not allow TLS1.3 connections with RSA-PSS+SHA256 signature algorithm.

[rohithjayak]

Detailed Description of the Problem

Hi,
We have integrated haproxy 3.0.5 with openssl1.1.1w and when we try to trigger an openssl s_client connection to few of the ports openend by haproxy for ex: 8443, 443 we are unable to get a successful connection, this happens specifically when the client used sigalg as RSA-PSS+SHA256. With the latest HAProxy, the openssl is unable to find a suitable certificate or pKey which is matching the specified signature scheme (specifically while using RSA-PSS+SHA256) and in turn fails with SSL_R_NO_SUITABLE_SIGNATURE_ALGORITHM error.

[root@node root]# openssl s_client -connect 10.xx.yy.zz:8443 -sigalgs RSA-PSS+SHA256 -tls1_3
CONNECTED(00000003)
140178270058304:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1544:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 191 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

This was working with haproxy 2.2.32 integrated with same openssl1.1.1w.
But with 3.0.5, looks like there are lots of changes with the SSL initialization and certificate generation when compared with 2.2.32, so we just wanted to confirm if this is a known limitation with the latest HAProxy? And any idea why the RSA certificate and private key is not getting loaded into the SSL object?

Please note:

When we use the sigalg as RSA-PSS+SHA256:RSA+SHA384, we are able to see that the TLS connections are working fine. Also it works fine when we use ECDSA+SHA384 but it does not work when we just pass a single sigalg(i.e., specifically RSA-PSS+SHA256).

Expected Behavior

The TLS connection should go fine with the below s_client command:

openssl s_client -connect 10.yy.xx.zz:8443 -sigalgs RSA-PSS+SHA256 -tls1_3

Steps to Reproduce the Behavior

Below openssl s_client command to any of the ports opened via haproxy
openssl s_client -connect 10.yy.xx.zz:8443 -sigalgs RSA-PSS+SHA256 -tls1_3

Do you have any idea what may have caused this?

Looks like there are lots of changes with the SSL initialization and certificate generation within haproxy 3.0.5 when compared with 2_2_32, so would like to understand if any of these changes is not honoring the RSA-PSS+SHA256.

Do you have an idea how to solve the issue?

Looks like ECDSA certificate is getting loaded by default and the connection goes fine while using ECDSA+SHA384 sigalg, so tried adding only the rsa certificate specifically to the default-crt option in the bind line instead of the generic certificate name as follows. When i specifically added the RSA certificate in the bind line as follows, the TLS connection worked with RSA-PSS+SHA256.

bind 0.0.0.0:8443 tfo ssl default-crt /usr/local/platform/.security/tomcat/keys/HAProxy_tomcat.pem**.rsa** npn http/1.1

What is your configuration?

haproxy.conf
`
global
  maxconn 32768
  nbthread 4
  maxsslrate 500
  maxsslconn 18000
  tune.bufsize 32768
  tune.ssl.lifetime 3600
  tune.ssl.default-dh-param 2048
  log /dev/log local5

  stats socket ipv4@127.0.0.1:9999 level admin
  stats socket /var/run/haproxy.sock mode 666 level admin
  stats timeout 2m

  ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:AES256-SHA:AES128-SHA:DHE-RSA-AES128-SHA
  ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
  ssl-default-bind-options prefer-client-ciphers ssl-min-ver TLSv1.2  ssl-max-ver TLSv1.3
  ssl-default-bind-curves X25519:P-256:P-384:P-521

...
...
...

# This opens port 443 and 8443, and redirects it to 84 Proxy
listen ssl

  bind 0.0.0.0:443   tfo ssl default-crt /usr/local/platform/.security/tomcat/keys/HAProxy_tomcat.pem npn http/1.1

  bind :::443 v6only tfo ssl default-crt /usr/local/platform/.security/tomcat/keys/HAProxy_tomcat.pem npn http/1.1

  bind 0.0.0.0:8443   tfo ssl default-crt /usr/local/platform/.security/tomcat/keys/HAProxy_tomcat.pem npn http/1.1

  bind :::8443 v6only tfo ssl default-crt /usr/local/platform/.security/tomcat/keys/HAProxy_tomcat.pem npn http/1.1

`
Please note: we have both RSA and ECDSA certs in the keys location.

Output of haproxy -vv

HAProxy version 3.0.5-8e879a5 2024/09/19 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.5.html
Running on: Linux 4.18.0-513.11.1.el8_9.x86_64 #1 SMP Wed Jan 17 02:00:40 EST 2024 x86_64
Build options :
  TARGET  = custom
  CC      = cc
  CFLAGS  = -O2 -g -fwrapv
  OPTIONS = USE_EPOLL=1 USE_THREAD=1 USE_LINUX_SPLICE=1 USE_OPENSSL=1 USE_CPU_AFFINITY=1
  DEBUG   =

Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ENGINE +EPOLL -EVPORTS -GETADDRINFO -KQUEUE -LIBATOMIC -LIBCRYPT -LINUX_CAP +LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO +THREAD -THREAD_DUMP -TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1w.7.2.555
Running on OpenSSL version : OpenSSL 1.1.1w.7.2.555
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): no
Built with gcc compiler version 8.5.0 20210514 (Red Hat 8.5.0-20)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

Additional Information

No response

[wlallemand]

Hello,

There were some changes in the way HAProxy loads certificates indeed, this might be a corner case not well handled in the new selection algorithms, I don't see why this does not behave like it should with RSA-PSS+SHA256:RSA+SHA384, I need to make some tests about it.

When using a certificate like this with foobar.pem.ecdsa and foobar.pem.rsa:

With previous versions on HAProxy crt "foobar.pem" was loading both .ecdsa and .rsa in the same SSL context, and the first SSL context of a bind line becomes the default context.

With recent versions, HAProxy will create 2 contextes with the .ecdsa and the .rsa, and put both as default.

The problem is that if you don't test with the -servername option of s_client HAProxy is not able to made the selection with the SNI. Without a SNI only the default contexts are used. That's why there is a new "default-crt" keyword which helps to set new defaults context. But this is still supposed to work since both contextes are default, but there might be something missing.

[wlallemand]

I identified some limitation in the actual selection code, which was designed when TLSv1.3 was not a thing. It lacks additional checks on the sigalgs when doing TLSv1.3. I'll work on a fix to handle this.

However, I'm still surprised about "RSA-PSS+SHA256:RSA+SHA384" working, but not "RSA-PSS+SHA256" nor "RSA+SHA384" alone... Looks like something else is going on.

[wlallemand]

However, I'm still surprised about "RSA-PSS+SHA256:RSA+SHA384" working, but not "RSA-PSS+SHA256" nor "RSA+SHA384" alone... Looks like something else is going on.

This is in fact normal for "RSA+384" alone because PSS is required for TLSv1.3.

[wlallemand]

So what's going on is that the clienthello code was made for TLSv1.2 with these in mind:

https://datatracker.ietf.org/doc/html/rfc5246#section-7.4.1.4.1

      enum {
          none(0), md5(1), sha1(2), sha224(3), sha256(4), sha384(5),
          sha512(6), (255)
      } HashAlgorithm;

      enum { anonymous(0), rsa(1), dsa(2), ecdsa(3), (255) }
        SignatureAlgorithm;

      struct {
            HashAlgorithm hash;
            SignatureAlgorithm signature;
      } SignatureAndHashAlgorithm;

      SignatureAndHashAlgorithm
        supported_signature_algorithms<2..2^16-2>;

But TLSv1.3 extended it to that:
https://datatracker.ietf.org/doc/html/rfc8446#section-4.2.3

 SignatureSchemeList value:

      enum {
          /* RSASSA-PKCS1-v1_5 algorithms */
          rsa_pkcs1_sha256(0x0401),
          rsa_pkcs1_sha384(0x0501),
          rsa_pkcs1_sha512(0x0601),

          /* ECDSA algorithms */
          ecdsa_secp256r1_sha256(0x0403),
          ecdsa_secp384r1_sha384(0x0503),
          ecdsa_secp521r1_sha512(0x0603),

          /* RSASSA-PSS algorithms with public key OID rsaEncryption */
          rsa_pss_rsae_sha256(0x0804),
          rsa_pss_rsae_sha384(0x0805),
          rsa_pss_rsae_sha512(0x0806),

          /* EdDSA algorithms */
          ed25519(0x0807),
          ed448(0x0808),

          /* RSASSA-PSS algorithms with public key OID RSASSA-PSS */
          rsa_pss_pss_sha256(0x0809),
          rsa_pss_pss_sha384(0x080a),
          rsa_pss_pss_sha512(0x080b),

          /* Legacy algorithms */
          rsa_pkcs1_sha1(0x0201),
          ecdsa_sha1(0x0203),

          /* Reserved Code Points */
          private_use(0xFE00..0xFFFF),
          (0xFFFF)
      } SignatureScheme;

      struct {
          SignatureScheme supported_signature_algorithms<2..2^16-2>;
      } SignatureSchemeList;

We are only checking 1byte to dermine if this was RSA, ECDSA or DSA, but the PSS one does have the first byte to 0x01, so they never match the RSA-PSS signatures.

This patch must fix the issue: 2baf085

[wtarreau]

William, I'm about to work on the release, don't you want to push that patch to master so that it's in that release ?

[wlallemand]

@wtarreau I just needed the CI before, I'm pushing this to master.

[wtarreau]

Thank you!

[wlallemand]

@rohithjayak could you check with the current master if that's behaving correctly for you? So we could work on backports in previous releases.