ossl_get_channel_binding fails with ML-DSA certificates with "Could not find digest algorithm UNDEF (NID 0)"

[rcritten]

I did this

I'm working on adding support for ML-DSA certs/keys to certmonger (X.509 certificate tracking and renewal) which uses libcurl to make HTTP requests. Requests are failing with CURLE_SSL_INVALIDCERTSTATUS.

Apparently OpenSSL has moved away from NID and they are not defined for ML-DSA at all. There is some mention of that in openssl/openssl@3216dc1

This is causing EVP_get_digestbynid(algo_nid) to return NULL and the function bails causing the connection to fail.

I expected the following

No response

curl/libcurl version

curl 8.18.0

operating system

Fedora 44

[icing]

Hmm, you did not say what you expect.

Shall we document that this does not work for PQC algorithms in OpenSSL? Do you think we should just not establish a channel binding in that case? Do you want to work on a channel binding solution that solves the problems for PQC?

[rcritten]

You raise an interesting question. My expectation was that channel binding be made to work with PQC in order to retain a stronger security posture. I don't have enough domain knowledge of openssl to know what is required for this. I had naively thought that the current code could be extended for PQC. I imagine it could be also substantially more work (e.g. new code from whole cloth).

[icing]

Since the channel binding is only in use for the infamous NEGOTIATE authentication, I have no idea what the proper way forward with PQC is. I simply have no test system to verify how that would possibly work.

Unless someone with knowledge comes forward, failing such a request seems the only safe thing to do.

[abbra]

The problem is visible with PQC algorithms but it is not limited to them. OpenSSL 3.5+ does not provide NIDs to any algorithm which is not defined in the library and is only available through a provider. For example, in FIPS mode the default provider is replaced by the FIPS one and then all algorithms come from this loadable module and will have nulled NIDs as well.

[icing]

I took a stab at finding the digest of a cert's signature without using NID in #20734. I cannot test this, so it would be helpful if you could give it a try in your environment. Thanks.

[rcritten]

FWIW I found a post on the IETF mailing list pointing out that tls-server-end-point channel binding for ML-DSA is undefined in the RFC. I was not able to find any follow-up to this thread.

[rcritten]

Some further information RFC 5299 in section 4.1 reads:

   o  if the certificate's signatureAlgorithm uses no hash functions or
      uses multiple hash functions, then this channel binding type's
      channel bindings are undefined at this time (updates to is channel
      binding type may occur to address this issue if it ever arises).

This is the case for ML-DSA.

Alexander Bokovoy provided a standalone program that demonstrates how to detect in OpenSSL when a signature algorithm has no hash in PR 20734. I have a very rough port of this into lib/vtls/openssl.c and propose that the channel binding be skipped for this class of certificate. In other words, if this condition is detected, ossl_get_channel_binding returns CURLE_OK with no header set until such time that this becomes defined behavior in an RFC.

[icing]

Thanks for the explainer. Will adapt the PR to not make it fail, as suggested.

[icing]

@rcritten updated #20734 with my version of the excellent example from @abbra .