RSA algorithm incompatible with other languages

[gdearment]

the sun crypto provider's implementation of RSA/ECB/OAEPWithSHA-256AndMGF1Padding I believe has a bug it in, which causes it to not be compatible with other languages (like Go). I have verified Go's RSA/ECB/OAEP-SHA-256 cipher is incapable of decrypting a value encrypted with java, given the same key.

This issue appears to outline the problem, but it is quite old: https://bugs.openjdk.java.net/browse/JDK-7038158

We were able to verify that java's RSA/ECB/OAEPWithSHA-1AndMGF1Padding works fine between java and Go. So as a proposed path forward, I'd suggest we update the java's cipher to use something that is compatible across languages.

[nmiyake]

What's the simplest way to repro this? Curious to poke around a bit and see what's going on.

[nmiyake]

BLUF: current incompatibility stems from the fact that Java's RSA/ECB/OAEPWithSHA-1AndMGF1Padding defaults to OAEP-SHA-256/MGF1-SHA-1, while Golang uses OAEP-SHA-256/MGF1-SHA-256 (Go's rsa package is implemented in such a way that the same hash function is used for OAEP and MGF1, even though that's not a strict requirement of the protocol). We can work around this on the Go side for legacy compatibility, but may make sense to make the parameters more standard on the Java side going forward (that is, produce/consume OAEP-SHA-256/MGF1-SHA-256 in Java ECV).

---

OK, got a chance to take a look, and figured out what's going on. This isn't a bug per se, but arises from somewhat unconventional defaults.

OAEP and MGF1 padding are separate concerns, but they both require hash functions. However, the hash functions they use do not need to be the same.

The Sun crypto provider's implementation of RSA/ECB/OAEPWithSHA-256AndMGF1Padding defaults to using SHA-256 for OAEP, but still uses SHA-1 for MGF1 padding. However, Go's implementation of OAEP encrypt/decrypt in their rsa package only takes one hash function and uses it for both OAEP and MGF1. Thus, our Java implementation uses OAEP-SHA-256/MGF1-SHA-1, while Go uses OAEP-SHA-256/MGF1-SHA-256, and these two are not compatible. As Greg observed, the SHA-1 padding works because then everything lines up.

We can make the Go side compatible with Java by making it use OAEP-SHA-256/MGF1-SHA-1. Unfortunately, currently the rsa package doesn't allow this to be configured -- I opened golang/go#19974 on Golang since this probably should be a parameter. In the meantime, it is possible to copy Go's RSA implementation (it's in a single file) and make that parameter configurable.

The other option would be to configure the Java side to produce OAEP-SHA-256/MGF1-SHA-256. We can do this with the following code (example is for decryption, but applies the same for encryption as well):

OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-256"), PSource.PSpecified.DEFAULT);
cipher.init(Cipher.DECRYPT_MODE, privateKey, oaepParams);

I confirmed that both options outlined above produce inputs/outputs that are compatible between languages.

For legacy purposes, we should make sure that Golang can read old keys. However, if we're standardizing encryption algorithm parameters moving forward knowing that we'll eventually switch over (as we're doing in #56 for AES), it may make sense to switch the Java code to use the more widely compatible OAEP-SHA-256/MGF1-SHA-256 parameters.