OpenSSL 3.0.0alpha16 interop issue with 0.9.8zh - Diffie-Hellman invalid public key #15465
Comments
mattcaswell
added
triaged: bug
Thanks @rainerjung - that would be very helpful |
|
I can not reproduce with 0.9.8 s_client, but with a very simple programmatic client using 0.9.8. I am convinced, that the problem is due to the missing padding (DH_compute_key_padded(()) in OpenSSL before 1.0.2 introduced in this commit first in 1.0.2: OpenSSL 3 sets DH_R_INVALID_PUBKEY in crypto/dh/dh_key.c line 392 when the size of the public key is not correct. Although the error only happens spuriously it happens roughly 1/256 of the attempts, which would be a good fit to the average likelyness, that the key has at least one byte less as size. The stricter check in 3 was added in 9aaecbf as a side effect of FFDHE support. Since the padding is mentioned in section 4.2.8.1 of RFC 8446 and was introduced in 1.0.2 as a client in 2013 the interop break with older OpenSSL might be OK. A very verbose dump of s_server shows the following differences between a working handshake and a breaking one: Working: SSL_accept:SSLv3/TLS write server done Broken: SSL_accept:SSLv3/TLS write server done
|
|
This works as reproducer for me and openssl 1.1.1l is involved. Server side with openssl 3.0.1: client side openssl 1.1.1l: Once a while server reports: Not seeing this between 3.0.1 <--> 3.0.1 and 1.1.1l <--> 1.1.1l. (but I wasn't testing these for longer than few minutes). Only when 3.0.1 is server and 1.1.1l is a client (not vice versa). Real life cases where this happens: Nagios nrpe client (openssl 1.1.1l) checking nrpe server (openssl 3.0.1) and sometimes reporting: "CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 1.2.3.4: 1" Client side: Server side: |
t8m
added
branch: 3.0
It does not make sense to check the size because this function can be used in other contexts than in TLS-1.3 and the value might not be padded to the size of p. However it makes sense to do the partial pubkey check because there is no valid reason having the pubkey value outside the 1 < pubkey < p-1 bounds. Fixes openssl#15465
|
Fix in #17630 |
It does not make sense to check the size because this function can be used in other contexts than in TLS-1.3 and the value might not be padded to the size of p. However it makes sense to do the partial pubkey check because there is no valid reason having the pubkey value outside the 1 < pubkey < p-1 bounds. Fixes #15465 Reviewed-by: Paul Dale <pauli@openssl.org> (Merged from #17630) (cherry picked from commit 2c0f7d4)
I ran into an interop problem when doing release testing for httpd 2.4.48 using OpenSSL 3.0.0alpha16 in the server and OpenSSL 0.9.8zh in the client. First I made TLS 1.0 work by adding "@SECLEVEL=0" at the end of the SSLCipherSuite config lines, but I still see sporadic failures:
SSL Library Error: error:02800066:Diffie-Hellman routines::invalid public key
I think they might have to do with restricting the allowed params for DH in OpenSSL 3. So it seems sometimes 0.9.8 generates DH keys that do not fulfil the stricter requirements of version 3. That means using such old client software will most of the time succeed with an TLS 1.0 handshake, but sporadically it will fail. I am not arware of a workaround.
Of course TLS 1.0 should no longer be used, but since it is supported in OpenSSL 3, the behavior of sporadic interop failures is a bit problematic and should at least be documented.
With a bit of help how to add more debug info, I could do a custom httpd build. But I will try to reproduce with s_server and s_client first and hope there's enough debug flags there to understand the used DH params.
The text was updated successfully, but these errors were encountered: