OpenSSL incompatibilities with SChannel

[Lotendan]

Hello,

I have set up a secure SSL connection between a Mariadb client and a Mariadb server.
Upon upgrade from OpenSSL 3.0.6 to 3.0.7, the Windows-based client now refuses to connect to the server with an SSL error:
SSL connection error: no cipher match. Error 0x80090331(SEC_E_ALGORITHM_MISMATCH)

Files are attached certificates-and-keys.zip and generated as follows:

# CA cert
$ openssl req -x509 -newkey rsa:2048 -keyout ca.key -out ca.crt -days 365 -nodes -subj '/C=EU/ST=Europe/O=test/CN=test.com/' -sha256

# Server cert
$ openssl req -newkey rsa:2048 -days 365 -nodes -keyout server.key -subj '/C=EU/ST=Europe/O=server test/CN=server.test.com/' -sha256 > server.csr
$ openssl x509 -req -in server.csr -days 365 -CA ca.crt -CAkey ca.key > server.crt

# Client cert
$ openssl req -newkey rsa:2048 -days 365 -nodes -keyout client.key -subj '/C=EU/ST=Europe/O=client test/CN=client.test.com/' -sha256 > client.csr
$ openssl x509 -req -in client.csr -days 365 -CA ca.crt -CAkey ca.key > client.crt

OpenSSL version on the server: OpenSSL 3.0.7 1 Nov 2022
Server: MariaDB 10.9.4 running on ArchLinux 01SEP 2022
OpenSSL conf file: openssl.cnf.zip
Client: HeidiSQL (linked with MariaDB C Connector, itself linked with SChannel)
Client system: Windows 10 22H2 19045.2486
Wireshark capture: wireshark_capture.zip

As you can see the client abruptly aborts the connection for some reason.
The client and the server are able to find a common cipher algorithm, but I suspect that it cannot find a common signature algorithm.
Adding more verbose logging on SChannel via the registry doesn't give more information; the system simply tells about a "bad handshake".
The logs of the server are not helpful either. The server only sees an aborted connection from the client.
Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '192.168.1.61' (This connection closed normally without authentication).
Using other clients (Python, etc...) fails as well as they are all linked with SChannel on Windows.

Similar issues

• Using CipherString = DEFAULT:@SECLEVEL=1 as recommended here did not help.
• Similar issue posted in parallel on the client: Mariadb 10.6 - unable to connect via SSL HeidiSQL/HeidiSQL#1426

Can you please help with the issue? Getting desperate about this after having spent days. Only workaround was to disable SSL requirement.
Thanks a lot and best regards

[t8m]

• Using CipherString = DEFAULT:@SECLEVEL=1 as recommended here did not help.

Could you please try to use CipherString = DEFAULT:@SECLEVEL=0 ?

[mattcaswell]

Is it possible to get a wireshark capture of a working server (i.e. with OpenSSL 3.0.6)...it might help identify the differences. The wireshark capture you provided doesn't show anything untoward that I can identify up until the point that the client abruptly closes the connection.

[Lotendan]

Thanks, it works with CipherString = DEFAULT:@SECLEVEL=0
Attached Wireshark capture of a working server still with OpenSSL 3.0.7: good_seclevel0.zip

The server seems to advertise more signature algorithms:

BAD

Signature Hash Algorithms (20 algorithms)
[...]

GOOD

Signature Hash Algorithms (23 algorithms)
[...]
    Signature Algorithm: ecdsa_sha1 (0x0203)
    Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
    Signature Algorithm: SHA1 DSA (0x0202)

Unless I'm not looking at the right place and the signature algorithms have nothing to do with this problem?

SECLEVEL=0 is not an acceptable solution though. Is there anything that you can relate and that has changed in the latest versions that would be responsible for this?
Is there anything else I could do with the CA/certificates to comply with a higher level of security?
What do you recommend?

[mattcaswell]

It almost certainly is related tot he sig algs. In particular the added sig algs all use SHA1 which is widely considered to be broken. Probably the aborting client is relying on them.

I'm slightly surprised that this behaviour changed between 3.0.6 and 3.0.7 though. SHA1 was supposed to be disabled in the default security level in 3.0.0, so I would have expected this to already be broken in 3.0.6. I'm wondering what has changed to cause this.

[mattcaswell]

Interestingly there is an overlap between between the sigalgs supported by the client and those supported by the server.

For example when it comes to declaring support for signature verification the client claims to support "rsa_pkcs1_sha256" and the server also claims to be able to support verifying this algorithm. But when it comes to signature generation it seems that the client is only willing to support sha1 based sig algs. This seems like quite a significant deficiency on the client side.

[mattcaswell]

Even more curious is that I am currently unable to find a difference between 3.0.6 and 3.0.7 that would explain this. In my testing it seems that a 3.0.6 server will not claim support for SHA1 based sig algs at the default security level.

E.g. running s_server like this with 3.0.6:

openssl s_server -verify 5 -no_tls1_3

Results in the following sig algs being advertised in the CertificateRequest:

        ecdsa_secp256r1_sha256 (0x0403)
        ecdsa_secp384r1_sha384 (0x0503)
        ecdsa_secp521r1_sha512 (0x0603)
        ed25519 (0x0807)
        ed448 (0x0808)
        rsa_pss_pss_sha256 (0x0809)
        rsa_pss_pss_sha384 (0x080a)
        rsa_pss_pss_sha512 (0x080b)
        rsa_pss_rsae_sha256 (0x0804)
        rsa_pss_rsae_sha384 (0x0805)
        rsa_pss_rsae_sha512 (0x0806)
        rsa_pkcs1_sha256 (0x0401)
        rsa_pkcs1_sha384 (0x0501)
        rsa_pkcs1_sha512 (0x0601)
        ecdsa_sha224 (0x0303)
        rsa_pkcs1_sha224 (0x0301)
        dsa_sha224 (0x0302)
        dsa_sha256 (0x0402)
        dsa_sha384 (0x0502)
        dsa_sha512 (0x0602)

So, no mention of SHA1. Modifying the command to this:

openssl s_server -verify 5 -no_tls1_3 -cipher "DEFAULT:@SECLEVEL=0"

Results in the SHA1 sigalgs appearing:

        ecdsa_secp256r1_sha256 (0x0403)
        ecdsa_secp384r1_sha384 (0x0503)
        ecdsa_secp521r1_sha512 (0x0603)
        ed25519 (0x0807)
        ed448 (0x0808)
        rsa_pss_pss_sha256 (0x0809)
        rsa_pss_pss_sha384 (0x080a)
        rsa_pss_pss_sha512 (0x080b)
        rsa_pss_rsae_sha256 (0x0804)
        rsa_pss_rsae_sha384 (0x0805)
        rsa_pss_rsae_sha512 (0x0806)
        rsa_pkcs1_sha256 (0x0401)
        rsa_pkcs1_sha384 (0x0501)
        rsa_pkcs1_sha512 (0x0601)
        ecdsa_sha224 (0x0303)
        ecdsa_sha1 (0x0203)
        rsa_pkcs1_sha224 (0x0301)
        rsa_pkcs1_sha1 (0x0201)
        dsa_sha224 (0x0302)
        dsa_sha1 (0x0202)
        dsa_sha256 (0x0402)
        dsa_sha384 (0x0502)
        dsa_sha512 (0x0602)

So, basically, it is currently unclear how this ever worked for you with OpenSSL 3.0.6. It should be behaving the same way as 3.0.7 in this regard. Unless there was some other difference in configuration between the two versions??

[Lotendan]

This was bugging me so I went into my backups and extracted the server back.
I fiddled a lot with the different files just to realize that actually the server was linked with the latest OpenSSL but the version that was dlopen'ed was the following version:
OpenSSL 1.1.1q 5 Jul 2022
Maybe this makes more sense and explains the issue more accurately!

[mattcaswell]

Maybe this makes more sense and explains the issue more accurately!

Yes! If it was actually using 1.1.1q then that would explain a lot. SHA1 was still available at the default security level then. That was changed from 3.0.0.

Unfortunately, it doesn't help a lot solving your problem. The client insists on using SHA1 for signing even though it is perfectly capable of using other digests for verifying. If you don't have control of the client, and you want to allow the client to sign using SHA1, then I think the only option left is to lower the default security level to allow it.

[Lotendan]

Thanks for the support then. I think we can consider this as closed then.

I need to revert to the authors of the client library and check whether is it possible to no longer link with SChannel on Windows because it is likely to use weak/deprecated signature algorithms.

Else maybe this is a bug in SChannel because as you stated there is indeed an overlap but it is not picked by the client. This is less likely to be addressed though.

We can see in this page from Microsoft the support from the different Microsoft crypto providers.

Maybe last question: is there a way to try to force the server and OpenSSL to expose some of the more robust signature algorithms, and try to increase the overlap so hopefully another algo is picked i.e. some of these as advertised by the client:

Signature Hash Algorithms
    Signature Algorithm: rsa_pss_rsae_sha256 (0x0804)
    Signature Algorithm: rsa_pss_rsae_sha384 (0x0805)
    Signature Algorithm: rsa_pss_rsae_sha512 (0x0806)
    Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
    Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
    Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
    Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
    Signature Algorithm: rsa_pkcs1_sha512 (0x0601)
    Signature Algorithm: ecdsa_secp521r1_sha512 (0x0603)

[mattcaswell]

If you compare that list with the list I gave above in #20138 (comment) you'll see that they shoull all already be being advertised. Those are all in the default sigalgs list.

[Lotendan]

Indeed I wasn’t paying attention. I’ll close this for now.
Thanks all!
If you have any idea where I could file a bug report with SChannel I’ll gladly take it.

[mattcaswell]

If you have any idea where I could file a bug report with SChannel I’ll gladly take it.

No idea. Sorry.

[Lotendan]

Note: Jira issue created here with MariaDB C Connector: https://jira.mariadb.org/browse/CONC-639