Could not use CreateFromEncryptedPemFile to load certificates when using dotnet 7 in FIPS environment.

[Samcoder2000]

Description

Hi there, we are seeing this error when using dotnet 7 to load certificates.

Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:020000AB:rsa routines::invalid keypair
   at Interop.Crypto.DecodePkcs8PrivateKey(ReadOnlySpan`1 source, EvpAlgorithmId algorithmId)
   at System.Security.Cryptography.RSAOpenSsl.ImportParameters(RSAParameters parameters)
   at System.Security.Cryptography.RSA.ImportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 password, ReadOnlySpan`1 source, Int32& bytesRead)
   at System.Security.Cryptography.PemKeyHelpers.ImportEncryptedPem[TPass](ReadOnlySpan`1 input, ReadOnlySpan`1 password, ImportEncryptedKeyAction`1 importAction)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.ExtractKeyFromEncryptedPem[TAlg](ReadOnlySpan`1 keyPem, ReadOnlySpan`1 password, Func`1 factory, Func`2 import)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.CreateFromEncryptedPem(ReadOnlySpan`1 certPem, ReadOnlySpan`1 keyPem, ReadOnlySpan`1 password)
   at SimpleWebServer.Program.Main(String[] args) in /app/SimpleWebServer/Program.cs:line 17
   at SimpleWebServer.Program.<Main>(String[] args)

I have created two simple applications: one is using C#, the other is using C++ to demonstrate this issue. Both applications are using same certificates. Could you pls help us to figure out how to solve this issue? Thanks.
dotnet.tar.gz

Reproduction Steps

For C# application:

1. Unzip the file.
2. cd ./dotnet-issue/SimpleWebServer
3. ./simplewebserver build
4. docker run -d --name simplewebserver --publish 443:10001 simplewebserver
5. docker logs simplewebserver.
   Should be able to see the issue.

For C++ application:

1. Unzip the file.
2. cd ./dotnet-issue/SimpleWebServer
3. ./simplewebserver build-cpp
4. docker run -d --name simplewebservercpp --publish 443:8080 simplewebservercpp
5. docker logs simplewebservercpp.
   The application is running without issue.

Expected behavior

Dotnet application should be able to load the certificates.

Actual behavior

Dotnet application could not load the certificates.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Hi there, we are seeing this error when using dotnet 7 to load certificates.

Unhandled exception. Interop+Crypto+OpenSslCryptographicException: error:020000AB:rsa routines::invalid keypair
   at Interop.Crypto.DecodePkcs8PrivateKey(ReadOnlySpan`1 source, EvpAlgorithmId algorithmId)
   at System.Security.Cryptography.RSAOpenSsl.ImportParameters(RSAParameters parameters)
   at System.Security.Cryptography.RSA.ImportEncryptedPkcs8PrivateKey(ReadOnlySpan`1 password, ReadOnlySpan`1 source, Int32& bytesRead)
   at System.Security.Cryptography.PemKeyHelpers.ImportEncryptedPem[TPass](ReadOnlySpan`1 input, ReadOnlySpan`1 password, ImportEncryptedKeyAction`1 importAction)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.ExtractKeyFromEncryptedPem[TAlg](ReadOnlySpan`1 keyPem, ReadOnlySpan`1 password, Func`1 factory, Func`2 import)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.CreateFromEncryptedPem(ReadOnlySpan`1 certPem, ReadOnlySpan`1 keyPem, ReadOnlySpan`1 password)
   at SimpleWebServer.Program.Main(String[] args) in /app/SimpleWebServer/Program.cs:line 17
   at SimpleWebServer.Program.<Main>(String[] args)

I have created two simple applications: one is using C#, the other is using CPP to demonstrate this issue. Both applications are using same certificates. Could you pls help us to figure out how to solve this issue? Thanks.
dotnet.tar.gz

Reproduction Steps

For C# application:

1. Unzip the file.
2. cd ./dotnet-issue/SimpleWebServer
3. ./simplewebserver build
4. docker run -d --name simplewebserver --publish 443:10001 simplewebserver
5. docker logs simplewebserver.
   Should be able to see the issue.

For C++ application:

1. Unzip the file.
2. cd ./dotnet-issue/SimpleWebServer
3. ./simplewebserver build-cpp
4. docker run -d --name simplewebservercpp --publish 443:8080 simplewebservercpp
5. docker logs simplewebserver.
   The application is running without issue.

Expected behavior

Dotnet application should be able to load the certificates.

Actual behavior

Dotnet application could not load the certificates.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

Author:	Samcoder2000
Assignees:	-
Labels:	area-System.Security
Milestone:	-

[vcsjones]

I can take a look since I have a FIPS development and debugging environment set up.

[vcsjones]

I can reproduce it for FIPS. This fails in the SP800-56b validation that is only part of the FIPS module.

.NET runs your key through EVP_PKEY_check which results in it checking the consistency of your private key, and the FIPS module. You can reproduce this outside of .NET.

Using the private key from your example, on a FIPS mode Linux installation:

openssl pkey -in privatekey.pem -check

Gives me

Key is invalid
20549D95FFFF0000:error:020000AB:rsa routines:ossl_rsa_sp800_56b_check_keypair:invalid keypair:crypto/rsa/rsa_sp800_56b_check.c:430

Your C++ project does not fail because it does not call EVP_PKEY_check on the key.

Okay, so the next natural question is "Why doesn't EVP_PKEY_check like my key?"

Fortunately with a debug build of OpenSSL, I got a line number. Looking at the source of rsa_sp800_56b_check.c:430 I got:

    ret = ossl_rsa_check_prime_factor(rsa->p, rsa->e, nbits, ctx)
          && ossl_rsa_check_prime_factor(rsa->q, rsa->e, nbits, ctx)
          && (ossl_rsa_check_pminusq_diff(r, rsa->p, rsa->q, nbits) > 0)
          /* (Step 6): Check the private exponent d */
          && ossl_rsa_check_private_exponent(rsa, nbits, ctx)
          /* 6.4.1.2.3 (Step 7): Check the CRT components */
          && ossl_rsa_check_crt_components(rsa, ctx);
    if (ret != 1)
        ERR_raise(ERR_LIB_RSA, RSA_R_INVALID_KEYPAIR);

So your private key is failing one of those checks from the FIPS module. Stepping through it, your key is failing in ossl_rsa_check_private_exponent.

The check is basically this:

ret = (ret
          /* LCM(p - 1, q - 1) */
          && (ossl_rsa_get_lcm(ctx, rsa->p, rsa->q, lcm, gcd, p1, q1,
                               p1q1) == 1)
          /* (Step 6a) d < LCM(p - 1, q - 1) */
          && (BN_cmp(rsa->d, lcm) < 0)
          /* (Step 6b) 1 = (e . d) mod LCM(p - 1, q - 1) */
          && BN_mod_mul(r, rsa->e, rsa->d, lcm, ctx)
          && BN_is_one(r));

and the private exponent is failing at step 6a.

---

Summarizing: your key does not pass OpenSSL's FIPS check. This can be demonstrated on the command line with pkey -check using a FIPS-enforced OpenSSL configuration, so this is not .NET specific. Your C++ application does not perform EVP_PKEY_check which is why it is able to run successfully.

The solution to this is to use a key that meets FIPS requirements if you are going to use your key in a FIPS restricted environment.

[Samcoder2000]

@vcsjones Thank you so much for the detail explanation. I will try to see if my c++ application can reproduce the same error here.

[vcsjones]

@Samcoder2000 I modified your C++ application to reproduce it. The source for it is here:

https://gist.github.com/vcsjones/fb1abcf5fdbbabf357519a42de4233b8

With the changes, the container fails to start with

docker run -it --rm > sha256:619e62eea23bcfcb7a13b686f9e2665506913c786f71a1e2ae5e04ba88dc2b73
Port:8080
Private key failed validation.
20C0C6ADFFFF0000:error:020000AB:rsa routines:ossl_rsa_sp800_56b_check_keypair:invalid keypair:crypto/rsa/rsa_sp800_56b_check.c:430:

The same error as .NET.

[Samcoder2000]

@vcsjones I can reproduce this error from the modified c++ application. I will test our production certificate. Will report back. Thank you very much.

[vcsjones]

@Samcoder2000 given that this can be reproduced outside of .NET in both the OpenSSL CLI and in your C++ application, I am going to close this issue as the issue is outside of .NET.

Please feel free to re-open this issue if you think otherwise or there are unaddressed issues with .NET’s behavior.