Certificate request message signing algorithm rsa_pkcs1 is ignored if TLS 1.3 and TLS 1.2 are set and mTLS is used with ssl in the state.

[voluntas]

This problem occurred with tlsv1.3 and tlsv1.2 specified in versions. So I changed it to tlsv1.2 only and the problem was solved.

Perhaps the client specifies both TLS 1.3 and 1.2, but the signature algorithm selection process on the client side is TLS 1.3, even though the server side has selected TLS 1.2, and the rsa_pkcs1 process I think they are being discouraged.

---

Describe the bug

When using TLS 1.3 and TLS 1.2 with ssl, with mTLS (client authentication),
Certificate Request message from the server contains rsa_pkcs1 as the signature algorithm,
However, the certificate signed with rsa_pkcs1 is not used and an empty Certificate message is sent.

I am thinking that the following part may be wrong, but I have not found the problem part.
https://github.com/erlang/otp/blob/master/lib/ssl/src/ssl_handshake.erl#L1668-L1748

To Reproduce

The client passes a minimum of options to ssl:connect. The certificate is signed with sha384WithRSAEncryption.

-module(mtls).

-feature(maybe_expr, enable).

-export([run/0]).

run() ->
    maybe
        {ok, _Started} ?= application:ensure_all_started(ssl),
        TlsOpts = [{versions, ['tlsv1.3', 'tlsv1.2']},
                   {verify, verify_none},
                   {certfile, <<"cert.pem">>},
                   {keyfile, <<"key.pem">>}],
        {ok, _Socket} ?= ssl:connect("localhost", 4433, TlsOpts, 5000),
        ok
    else
        Reason ->
            io:format("Error: ~p~n", [Reason]),
            error
    end.

Server is OpenSSL with mTLS enabled, TLS 1.2 enforced,
The server uses OpenSSL with mTLS enabled, TLS 1.2 enforced, and a narrower signature algorithm.
The same certificate is used for both client and server for verification purposes.

$ openssl s_server -accept 4433 -cert cert.pem -key key.pem -Verify 1 -tls1_2 -cipher 'ECDHE-RSA-AES128-GCM-SHA256' -client_sigalgs rsa_pkcs1_sha384

When executed, the following error occurs.

1> mtls:run().
=NOTICE REPORT==== 22-Dec-2023::16:03:22.781490 ===
TLS client: In state cipher received SERVER ALERT: Fatal - Handshake Failure

Error: {error,
           {tls_alert,
               {handshake_failure,
                   "TLS client: In state cipher received SERVER ALERT: Fatal - Handshake Failure\n"}}}
error

Expected behavior

When TLS 1.3 and TLS 1.2 are specified in versions and mTLS is used, when a Certificate Request message is sent with rsa_pkcs1 included, a certificate signed by rsa_pkcs1 is included in the Certificate message

Affected versions

OTP-26.2.1

Additional context

The verification certificates cert.pem and key.pem used are on the following Gist.

https://gist.github.com/voluntas/47b95f54069f9728189041583c20aab7

Incidentally, it works if the signature algorithm includes rsa_pss_rsae; we found this problem because some servers did not include rsa_pss_rsae in the Certificate Request message.

Only rsa_pkcs1_sha384 is included in Certificate Request in TLS 1.2

Client Certificate is empty in TLS 1.2

[IngelaAndin]

FROM RFC 8446:

RSASSA-PKCS1-v1_5 algorithms:  Indicates a signature algorithm using
      RSASSA-PKCS1-v1_5 [[RFC8017](https://datatracker.ietf.org/doc/html/rfc8017)] with the corresponding hash algorithm
      as defined in [[SHS](https://datatracker.ietf.org/doc/html/rfc8446#ref-SHS)].  These values refer solely to signatures
      which appear in certificates (see [Section 4.4.2.2](https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.2.2)) and are not
      defined for use in signed TLS handshake messages, although they
      MAY appear in "signature_algorithms" and
      "signature_algorithms_cert" for backward compatibility with
      TLS 1.2.

So you need to include some other signature_algs that can be used for the protocol messages.

[voluntas]

Thanks for the quick response.

Sorry for the confusing explanation.

I feel that there is a problem with the behavior when TLS 1.3 and 1.2 are specified in "versions" and 1.2 is used, is this behavior now expected?

When both 1.3 and 1.2 are specified and TLS 1.2 is used, the behavior seems to be 1.3 behavior.

We believe that TLS 1.3 and 1.2 behavior respectively is fine.

• When only 'tlsv1.2' is specified, it succeeds.
• If only 'tlsv1.3' is specified, pss_rsae is needed instead of pkcs1

Is it possible that if I specify both tlsv1.3 and tlsv1.2, I need to specify signature algs?

[IngelaAndin]

If TLS-1.3 shall be negotiated you will need both rsa_pss_rsae and rsa_pkcs1_sha384 to be supported to be able to get a connection with that certificate.

[voluntas]

Thanks for the reply.

The issue I am having is that when both TLS 1.3 and TLS 1.2 are specified in ssl:connect versions and TLS 1.2 is selected, the certificate is empty.

It is not known why this is not the same behavior as when only TLS 1.2 is specified.

• Use a certificate whose Signature Algorithm is sha384WithRSAEncryption and whose Public Key Algorithm is rsaEncryption
• Set 'tlsv1.3' and 'tlsv1.2' to ssl:connect versions
• Select TLS 1.2 for the server
• Server Certificate Request message contains only rsa_pkcs1
• The ssl library thinks that it does not match the signature algorithm and sends an empty Certificate message. I think this is the bug.

Since the choice here is TLS 1.2, I think that if rsa_pkcs1 is included in the Certificate Request message, then the configured client certificate with sha384WithRSAEncryption should be sent.

Is this a bug or a specification of the ssl library?

When only TLS 1.2 is specified, if rsa_pkcs1 is sent from the server, the certificate is sent.

If both TLS 1.3 and TLS 1.2 versions are specified and TLS 1.2 is used, will the certificate not be sent even if rsa_pkcs1 is sent from the server?

If this is the specification, I will close this issue.
If specifying two versions, TLS 1.3 and TLS 1.2, is itself a problem, please let me know.

[voluntas]

I found the problem.

It seems that when both tlsv1.3 and tlsv1.2 are specified for versions, even though the pattern match is done with ?TLS_1_2 in the select_hashsign function, SupportedHashSigns is for TLS 1.3.
Therefore, {sha384, rsa} is not found and the certificate is empty.

Here is the line
https://github.com/erlang/otp/blob/OTP-26.2.1/lib/ssl/src/ssl_handshake.erl#L1684

2> ssl:connect("localhost", 4433, [{versions, ['tlsv1.3', 'tlsv1.2']}, {verify, verify_none}, {server_name_indication, "localhost"}, {customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, {certfile, <<"localhost.pem">>}, {keyfile, <<"localhost-key.pem">>}], 5000).
SupportedHashSigns | [eddsa_ed25519,eddsa_ed448,ecdsa_secp521r1_sha512,
                      ecdsa_secp384r1_sha384,ecdsa_secp256r1_sha256,
                      rsa_pss_pss_sha512,rsa_pss_pss_sha384,
                      rsa_pss_pss_sha256,rsa_pss_rsae_sha512,
                      rsa_pss_rsae_sha384,rsa_pss_rsae_sha256,
                      rsa_pkcs1_sha512,rsa_pkcs1_sha384,rsa_pkcs1_sha256,
                      {sha512,ecdsa},
                      {sha384,ecdsa},
                      {sha256,ecdsa}]

2> ssl:connect("localhost", 4433, [{versions, ['tlsv1.2']}, {verify, verify_none}, {server_name_indication, "localhost"}, {customize_hostname_check, [{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}, {certfile, <<"localhost.pem">>}, {keyfile, <<"localhost-key.pem">>}], 5000).
SupportedHashSigns | [{sha512,ecdsa},
                      rsa_pss_pss_sha512,rsa_pss_rsae_sha512,
                      {sha512,rsa},
                      {sha384,ecdsa},
                      rsa_pss_pss_sha384,rsa_pss_rsae_sha384,
                      {sha384,rsa},
                      {sha256,ecdsa},
                      rsa_pss_pss_sha256,rsa_pss_rsae_sha256,
                      {sha256,rsa}]

Let me try to find out why TLS 1.3 SupportedHashSigns is used here.

My guess is that if the SupportedHashSigns in this section uses the values from ClientHello, then when TLS 1.2 is selected, things will go wrong.

[voluntas]

I found that signature_algs only handles the first version.
https://github.com/erlang/otp/blob/OTP-26.2.1/lib/ssl/src/ssl.erl#L2069

Therefore, when TLS 1.3 and TLS 1.2 are specified, even if TLS 1.2 is adopted, the signature_algs of TLS 1.3 will be adopted and an empty certificate will be sent because {sha384,rsa} is not found.

Is this behavior intended? If it is intended, I will stop specifying more than one for versions and close this issue.

My personal opinion is that if both TLS 1.3 and TLS 1.2 are specified and TLS 1.2 is selected, the signature algorithm for TLS 1.2 should be obtained again when selecting the signature algorithm for the Certificate Request message.

[voluntas]

I don't think this is correct, but I rewrote the code as follows and it works.

https://github.com/erlang/otp/blob/OTP-26.2.1/lib/ssl/src/ssl_handshake.erl#L1679-L1685

select_hashsign(#certificate_request{
                   hashsign_algorithms = #hash_sign_algos{
                                            hash_sign_algos = HashSigns},
                   certificate_types = Types},
                Cert,
                _SupportedHashSigns0,
		?TLS_1_2) ->
    %% Force getting the TLS 1.2 signature algorithm again.
    SupportedHashSigns = ssl:signature_algs(default, 'tlsv1.2’),

[IngelaAndin]

There is a problem that some algorithms have different names in TLS-1.3 and TLS-1.2 context but have the same value on the wire.
I think we solved at least some such issue already, perhaps there is some issue left?! If so I will look into it and fix it after Christmas holidays.

[voluntas]

Yes, it appears so.

pkcs1 is {sha384, rsa} in TLS 1.2, but in TLS 1.3 it is treated as rsa_pkcs1_sha384. Therefore, if TLS 1.3 and TLS 1.2 are specified and TLS 1.2 is adopted, the error is that {sha384, rsa} is not found in the list of signature algorithms for TLS 1.3.

Sorry if this has been fixed in master or maint, since this is OTP 26.2.1.

I've worked around it by selecting TLS 1.2 only, so take your time!

[IngelaAndin]

@voluntas #7997 should solve the issue "tm" correct way. Seems I missed a place where the function handling the conversion should be called.

[voluntas]

@IngelaAndin Pull-Request confirmed, thanks for the fix!