ERL-763: TLS handshake fails for RSA based end-entity certificate signed by EC based CA with TLS insufficient security alert

[OTP-Maintainer]

Original reporter: urbanserj
Affected version: OTP-21.1
Fixed in version: OTP-21.2
Component: ssl
Migrated from: https://bugs.erlang.org/browse/ERL-763

---

Erlang server is failing to complete TLS handshake and it terminates TLS handshake with TLS alert 71 - insufficient_security (https://tools.ietf.org/html/rfc5246#section-7.2.2). 

TLS hierarchy setup:

{code}
Root CA (EC based private key)
  - Intermediate CA 1 (EC based private key)
  - Intermediate CA 2 (EC based private key)
    - End-entity certificate (RSA based private key)
{code}

OpenSSL commands to verify that the OpenSSL implementation works with provided certificate hierarchy:

{code}
openssl s_server -key $(pwd)/dcos-net.key -cert $(pwd)/dcos-net.crt -CAfile $(pwd)/ca-bundle.crt -verify_depth 10 -cipher ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 -tls1_2 -verify 10 -accept 4433 -Verify 10
openssl s_client -connect localhost:4433 -CAfile $(pwd)/ca-bundle.crt -key $(pwd)/dcos-net.key -cert $(pwd)/dcos-net.crt -cipher ECDHE-RSA-AES256-GCM-SHA384
{code}

Erlang SSL Client works well with openssl s_server:

{code}
1> ssl:start(), {ok, [[{server, _}, {client, SSLOptions}]]} = file:consult("tls.config"), ssl:connect("localhost", 4433, SSLOptions).
{ok,{sslsocket,{gen_tcp,#Port<0.6>,tls_connection,undefined},
               [<0.104.0>,<0.103.0>]}}
{code}

Let’s start two erlang nodes with distribution protocol over TLS:

{code}
erl -setcookie test -name foo@127.0.0.1 -proto_dist inet_tls -ssl_dist_optfile $(pwd)/tls.config
erl -setcookie test -name bar@127.0.0.1 -proto_dist inet_tls -ssl_dist_optfile $(pwd)/tls.config
{code}

Ping {{foo}} node from {{bar}} node:

{code}
(bar@127.0.0.1)1> net_adm:ping('foo@127.0.0.1').
=INFO REPORT==== 31-Oct-2018::17:06:42.821584 ===
TLS client: In state hello received SERVER ALERT: Fatal - Insufficient Security

Pang
{code}

{code}
(foo@127.0.0.1)1> =INFO REPORT==== 31-Oct-2018::17:06:18.396240 ===
TLS server: In state hello at tls_handshake.erl:200 generated SERVER ALERT: Fatal - Insufficient Security - no_suitable_ciphers

=INFO REPORT==== 31-Oct-2018::17:06:42.808669 ===
TLS server: In state hello at tls_handshake.erl:200 generated SERVER ALERT: Fatal - Insufficient Security - no_suitable_ciphers
{code}

Try to connect to {{foo}} node using openssl client:

{code}
$ epmd -names
epmd: up and running on port 4369 with data:
name bar at port 52691
name foo at port 52689
$ openssl s_client -connect localhost:52689 -CAfile $(pwd)/ca-bundle.crt -key $(pwd)/dcos-net.key -cert $(pwd)/dcos-net.crt -cipher ECDHE-RSA-AES256-GCM-SHA384
CONNECTED(00000005)
4469548652:error:1400442F:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert insufficient security:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.200.4/libressl-2.6/ssl/ssl_pkt.c:1205:SSL alert number 71
{code}

Please find attached all certificates and tls configuration.

[OTP-Maintainer]

ingela said:

I believe the bug should be fixed by this patch

{code}
diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl
index 0e22d63..54c04c1 100644
--- a/lib/ssl/src/ssl_cipher.erl
+++ b/lib/ssl/src/ssl_cipher.erl
@@ -932,7 +932,7 @@ filter_suites_pubkey(ec, Ciphers, _, OtpCert) ->
                                    ec_ecdhe_suites(Ciphers)),
     filter_keyuse_suites(keyAgreement, Uses, CiphersSuites, ec_ecdh_suites(Ciphers)).
 
-filter_suites_signature(rsa, Ciphers, {3, N}) when N >= 3 ->
+filter_suites_signature(_, Ciphers, {3, N}) when N >= 3 ->
      Ciphers;
 filter_suites_signature(rsa, Ciphers, Version) ->
     (Ciphers -- ecdsa_signed_suites(Ciphers, Version)) -- dsa_signed_suites(Ciphers, Version);

{code}

[OTP-Maintainer]

urbanserj said:

[~ingela] It works well, all our integration tests have passed.

Thank you so much for the quick fix!