Handshake fails with 3.0.2 & 1.1.1n, but succeeds with 1.1.1

[lojic]

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

I'm unable to establish an SSL connection using OpenSSL 3.0.2 on Ubuntu 22.04, but I have no issues, with the same command, using OpenSSL 1.1.1 on Ubuntu 18.04. I'll start with the curl command output, in case it's helpful:

curl -v -f -u <username>:<password> https://<my-host>/path/file > file

* Connected to <my-host> (<ip-addr>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Unknown (21):
{ [5 bytes data]
* TLSv1.3 (IN), TLS alert, internal error (592):
{ [2 bytes data]
* error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

Using openssl s_client -msg -security_debug_verbose -connect <my-host>:443

The main error is:

805B5017597F0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1584:SSL alert number 80

The full stdout and stderr is in this gist

Happy to provide whatever other info I can, I've been googling & researching for hours, and I've tried dozens of arguments to s_client and curl to no avail. I wouldn't be surprised if the server (not mine) is misconfigured; however, since it works w/o problems on OpenSSL 1.1.1, it seems like I should be able to get it to work with 3.0.2.

Thanks!

[lojic]

Here is the SSL-Session successfully established with the OpenSSL 1.1.1 s_client:

SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 000200...<elided>
    Session-ID-ctx: 
    Master-Key: A2B...<elided>
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1666880841
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

[lojic]

Out of curiosity, I tried this from my Raspberry Pi OpenSSL 1.1.1n 15 Mar 2022 and it failed similarly to the 3.0.2 with the following error:

1996427328:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:../ssl/record/rec_layer_s3.c:1543:SSL alert number 80

The 1.1.1 version that succeeded is: OpenSSL 1.1.1 11 Sep 2018

[t8m]

The server is sending internal error alert. That indicates it cannot handle something the OpenSSL-1.1.1n and 3.0.2 versions send in the ClientHello that it is not able to handle properly. As the alert is internal error the problem is most probably on the server side.

You could try to eliminate some of the supported groups, signature algorithms, ciphersuites and even the TLS-1.3 version as the server does not seem to support it.

[lojic]

Since it works with 1.1.1 and fails with 1.1.1n and 3.0.2, is it possible to get a diff of what they're sending? I don't control the server, so I'll need to solve it from the client side.

I have tried many, many options such as locking down to TLS1.2 and specifying the same cipher that was used in the successful connection, but none of those attempts were successful.

Host is: extracts.nccourts.org in case that's helpful.

[mattcaswell]

Use the -trace argument to s_client and it will dump out verbose information on the handshake messages it is sending. Compare the first message ("ClientHello") for the working and non-working versions.

[naciohr]

Hello,

I'm getting the same issue. For what I've observed, openssl v3 starts the handshake on TLS1.0 always whatever specified with -tlsv1x option.

root@7e651c66d765:/# openssl s_client -connect <server> -trace -no_tls1
CONNECTED(00000003)
Sent Record
Header:
  Version = TLS 1.0 (0x301)
  Content Type = Handshake (22)
  Length = 316
    ClientHello, Length=312
      client_version=0x303 (TLS 1.2)
      Random:
[...]

root@7e651c66d765:/# curl -vvvv  -trace "ldaps://<server>" --tlsv1.2
[...]
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection

However, openssl version 1.1.1 starts with the requested tls version.

root@c7f0e0ff26ea:/# curl -vvvv "ldaps://<server>" --tlsv1.0
[...]
* TCP_NODELAY set
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (IN), TLS handshake, Server hello (2):
* TLSv1.0 (IN), TLS handshake, Certificate (11):
* TLSv1.0 (OUT), TLS alert, Server hello (2):
[...]

root@c7f0e0ff26ea:/# curl -vvvv "ldaps://<>" --tlsv1.2
[...]
* TCP_NODELAY set
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
[...]

Thanks and regards,

[mattcaswell]

Header:
Version = TLS 1.0 (0x301)

This is the record layer version and is ignored by all correct implementations. It has no bearing on the offered TLS versions or what TLS version actually gets negotiated. I would be very surprised if that had any relationship with an internal error alert response from the server.

[naciohr]

Correct. However, even if it should not be considered, I specified tlsv1.2 and it used 1.3 in curl. You should see the details capturing the traffic with wireshark.

I had to downgrade to openssl 1.1.1t to make the application work.

[mattcaswell]

I specified tlsv1.2 and it used 1.3 in curl. You should see the details capturing the traffic with wireshark.

This seems to be quite a different issue to the one initially described by the OP. Please create a new issue for that and supply the wireshark capture.

[fmoc]

OpenSSL 3.1.2 is apparently affected by this bug, too. See owncloud/client#11172.

[t8m]

OpenSSL 3.1.2 is apparently affected by this bug, too. See owncloud/client#11172.

I do not see 3.1.2 mentioned there at all.

[fmoc]

Sorry. I tested this by hand with our AppImage builds, too, which ship 3.1.2, but also with the Ubuntu packages which use 3.0.2. I just forgot to mention this explicitly in our issue. The AppImages can be found here.

[lojic]

I ran openssl s_client -msg -security_debug_verbose -connect <host>:443 -trace

Here is a gist with stdout and stderr.

Is there any other info I can provide to help solve this issue?

@mattcaswell I tried issuing that command on the working server, as you suggested, but I received the following error:

s_client: Option unknown option -trace
s_client: Use -help for summary.

So maybe -trace was added to a later version of openssl? I tried the -help option as prompted, but didn't see any option analogous to trace.

[kroeckx]

I don't see the output of a successful connection yet. I assume that the server is configured with some limited list of supported group, and only supports one we no longer enable by default in newer versions for security reasons.

[lojic]

It doesn't seem like that should result in an internal error though, right?

[t8m]

Please note that the server reports an internal error to the client and we are talking here about the change of the client's OpenSSL version.

So basically the server is buggy. There is no bug in OpenSSL (at least on the client that is).