.NET Core 3.1 RSA decryption with 3k certificates failed on Linux environments due to OAEP padding

[Maxyeah]

Description

While running our en/decryption test with .Net Core 3.1 we stepped into an Issue on Linux environments.

When using 3k certificate key sizes and Aes256Sha256RsaPss security policy(RsaPaddingMode.OaepSha256), decryption failed with "Error occurred while decoding OAEP padding".

On Windows or .Net 6 on Linux everything is working fine.
Are there any known restrictions?

Reproduction Steps

Encrypt message with rsa cryptoprovider with 3k certificate and RSAEncryptionPadding.OaepSHA256 on Linux and .Net 3.1
Decrypt message with rsa cryptoprovider with 3k certificate and RSAEncryptionPadding.OaepSHA256 on Linux and .Net 3.1

Expected behavior

Message will be decrypted

Actual behavior

Decryption fails with "Error occurred while decoding OAEP padding"

Regression?

No response

Known Workarounds

No response

Configuration

• .Net Core 3.1.420
• Debian 11
• Openssl 1.1.1n
• x64

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

While running our en/decryption test with .Net Core 3.1 we stepped into an Issue on Linux environments.

When using 3k certificate key sizes and Aes256Sha256RsaPss security policy(RsaPaddingMode.OaepSha256), decryption failed with "Error occurred while decoding OAEP padding".

On Windows or .Net 6 on Linux everything is working fine.
Are there any known restrictions?

Reproduction Steps

Encrypt message with rsa cryptoprovider with 3k certificate and RSAEncryptionPadding.OaepSHA256 on Linux and .Net 3.1
Decrypt message with rsa cryptoprovider with 3k certificate and RSAEncryptionPadding.OaepSHA256 on Linux and .Net 3.1

Expected behavior

Message will be decrypted

Actual behavior

Decryption fails with "Error occurred while decoding OAEP padding"

Regression?

No response

Known Workarounds

No response

Configuration

• .Net Core 3.1.420
• Debian 11
• Openssl 1.1.1n
• x64

Other information

No response

Author:	Maxyeah
Assignees:	-
Labels:	area-System.Security
Milestone:	-

[vcsjones]

Interestingly, a 4096-bit key works, 3072 does not.

Bisecting, it looks like this was fixed for .NET 6 by 6aa4d59 in #50063.

Small repro. (Fails in .NET Core 3.1 and .NET 5, passes in .NET 6)

using RSA rsa = RSA.Create(3072);
byte[] encrypted = rsa.Encrypt(new byte[] { 1, 2 ,3 }, RSAEncryptionPadding.OaepSHA256);
byte[] decrypted = rsa.Decrypt(encrypted, RSAEncryptionPadding.OaepSHA256);
Assert.Equal(new byte[] { 1, 2, 3}, decrypted);

@bartonjs does anything ring a bell here?

If anything, it seems at least there is an opportunity to test OAEP encryption using different key sizes.

[bartonjs]

That change would have moved us off of the managed implementation of OAEP, so there must be a logic bug there. (Which would mean other SHA-2 OAEPs would fail, but SHA-1 probably works)

[vcsjones]

Makes sense. I can take a look at this since presumably the managed implementation is still used on other platforms.

[vcsjones]

So, to summarize, this is a bug that affects key sizes that are not a power-of-two. 2048 and 4096 keys are powers of two, while 3072 is not.

1. Non-power-of-two key sizes do not work for RSA OAEP decryption with SHA2 on OpenSSL platforms (Linux) in .NET Core 3.1. This is not applicable for .NET 6+ because the managed depadding is not used on this platform.
2. Non-power-of-two key sizes do not work for RSA OAEP decryption with SHA2 on Android in .NET 6 and .NET 7 previews. It inherited the same bug from OpenSSL. This is not applicable for .NET Core 3.1 as there was no Android support.
3. .NET 7 will address the issue for Android by removing use of the managed OAEP depadding, similarly to what was done for OpenSSL in .NET 6.

Some PRs have been opened for potential servicing, but are still not approved: that's up to the team that decides if issues meet the servicing requirements.

[Maxyeah]

Thanks to @vcsjones that information helps us how to handle this.

[danmoseley]

@jeffhandley following up re being in last 6 month phase of 3.1 lifecycle.

[jeffhandley]

The issue as submitted does not describe enough impact to support backporting this to .NET Core 3.1. @Maxyeah, if this issue is substantially impacting a production application (as opposed to tests), you don't have a workaround, and you can't immediately move to .NET 6.0, please let us know more of that context for us to reconsider.

We have approved the fix for port to 6.0.

[vcsjones]

I believe this issue has been resolved then.

1. The .NET Core 3.1 OpenSSL fix was not accepted per @jeffhandley's comment.
2. .NET 6 and up are already fixed for OpenSSL.
3. The .NET 6 Android fix was accepted and will be in 6.0.9.
4. The .NET 7 Android fix will be in .NET 7.0-preview7.

I'm going to close this, but @Maxyeah if you would like to re-open it to further discuss the 3.1 fix as @jeffhandley indicated please feel free to do so.