crypto/x509: Certificate.isValid should reject certificates with UnknownPublicKeyAlgorithm

[rolandshoemaker]

A certificate with UnknownPublicKeyAlgorithm is not reasonably useful, since it cannot be used to verify any signatures, isValid should therefore reject it.

The one argument against this is for certificates which use key types which we don't support in the standard library, but for which third-party support does exist. In this case they could parse the key from the RawSubjectPublicKeyInfo, and use that to verify signatures, but that seems likely to be an incredibly uncommon use case (which I am not particularly sure actually currently exists).

Related to #66166, this would prevent putting certificates in the pool which can only reasonably be used by themselves.

[AGWA]

I assume you mean UnknownPublicKeyAlgorithm rather than UnknownSignatureAlgorithm?

Verifying a leaf certificate with an unknown public key algorithm doesn't seem that bizarre to me (though it's certainly not useful in the WebPKI), and removing support would probably break someone. Are there any justifications for this besides #66166? CertPool.Add could just check for UnknownPublicKeyAlgorithm itself, right?

[rolandshoemaker]

I assume you mean UnknownPublicKeyAlgorithm rather than UnknownSignatureAlgorithm?

🤦, yes.

Verifying a leaf certificate with an unknown public key algorithm doesn't seem that bizarre to me (though it's certainly not useful in the WebPKI), and removing support would probably break someone. Are there any justifications for this besides #66166? CertPool.Add could just check for UnknownPublicKeyAlgorithm itself, right?

For roots and intermediates I think the argument is clearer, since we cannot reasonably build any chains with them, but I think you can make an equally compelling argument (although perhaps slightly less strong) argument for leaves, in that the certificate itself may be verified, but what is the certificate then to be used for? It cannot be used for TLS, since you need to verify the signature during the handshake, you can't use it for code signing, or S/MIME, because you would need to verify the signature, etc etc.

While I don't think this is extremely high priority, I think since we do explicitly target web PKI, and since isValid is only used during chain building, doing this lets us remove a little complexity that needs to be considered currently (#65390 is an example of how this has bitten us recently).