crypto/rsa: allow hash.Hash for OAEP and MGF1 to be specified independently

[nmiyake]

rsa.EncryptOAEP and rsa.DecryptOAEP both take a hash.Hash as input, and this hash function is used as the hash function for both OAEP and the MGF1 XOR. However, an option should be provided to specify the hash function for OAEP and MGF1 separately, as it is permissible for the hash functions for these two operations to be different.

This is pertinent for compatibility with other languages and RSA implementations, as the Sun JDK's implementation of the RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING provider uses SHA-256 for OAEP but SHA-1 for MGF1. As it currently stands, the rsa package in Go is not compatible with this encryption mode in Java.

For reference, the OpenSSL API also allows for the hash functions for OAEP and MGF1 to be specified separately: https://github.com/openssl/openssl/blob/master/crypto/rsa/rsa_oaep.c#L44, const EVP_MD *md, const EVP_MD *mgf1md.

[andybons]

@FiloSottile

[brandonweeks]

As another data point, Android P has introduced usage of RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING.

https://developer.android.com/reference/android/security/keystore/WrappedKeyEntry

[shuxiao9058]

the same issue.

[ribeiroit]

Hi guys, I could help to implement the resolution for this issue, but I need your help to decide what's the best scenario for this.

Do you think the OEAPOptions is the best way?

Thank you

[FiloSottile]

OAEP is one of those unfortunate standards with a million parameters, which the Go standard library intentionally culls to a reasonable subset. For example, there is no good reason I know of to use different hashes at different stages of the algorithm.

On balance, I'm mostly ok with adding a field to OEAPOptions for decryption, as it won't disturb common users. However, adding a whole new function would be more invasive, and I don't think it would be worth it.

Would having support for independent hashes only for decryption make sense for your use cases, or would it be useless without encryption? (It makes sense to me also because the library is already asymmetrical due to crypto.Decrypter not having an encryption counterpart, and decryption is more likely to be about compatibility with a third party.)

[ribeiroit]

I got your point, but thinking in interoperability maybe is the deal here. I think that it could reduce the fricction along another implementations. As the use case that I exposed related to android.

I think that makes sense use for both encrypt/decrypt.

Well, this is my understood at reading RFC 3447, starting chapter "7. Encryption schemes" and keeping up beyond this section. But, looking to the implementation that we have at the golang, I think that is a gonna be just a decision to update of the usage context and allow the encryption besides the decryption.

Thanks