public_key:decrypt_private OTP bug

[victorolinasc]

Hi @potatosalad !

I've found an odd bug on OTP public_key module that makes RSA-OAEP broken here on JOSE.

The bug report is here. What happens is that public_key:decrypt_private/3 is not respecting the options passed to it. So, RSA-OAEP breaks here. Options is ignored.

The fix would be to switch it to use crypto:private_decrypt directly. To do that we need to unwrap the private key because crypto accepts a list of integers instead of the record. For a two-prime RSA private key what I am currently doing is:

{:RSAPrivateKey, _, n, e, d, p1, p2, e1, e2, c, _} = priv_key
:crypto.private_decrypt(:rsa, encrypted_key, [e, n, d, p1, p2, e1, e2, c], [rsa_padding: :rsa_pkcs1_oaep_padding, rsa_oaep_md: :sha256, rsa_mgf1_md: :sha256])

I hope this helps. Since I am not that familiar with Erlang tools (common test and so on) I haven't tried opening a PR for this fix.

[victorolinasc]

Same thing happens to public_key:encrypt_public/3

[potatosalad]

@victorolinasc If I wait long enough to reply, maybe all bugs like this will have been fixed during the delay 😄

This has been fixed as of OTP 22 thanks to the bug you filed in ERL-878.