CryptographicException results when attempting to protect data after configuring with ProtectKeysWithCertificate

[porcus]

We have an Asp.NET MVC 5 app targeting DNX 4.5.1, and whenever we attempt to use the ProtectKeysWithCertificate method to configure the keys to be encrypted to a specified X509 certificate before being persisted to storage, we get a CryptographicException, with the message: Unable to retrieve the decryption key.

This is how data protection is being configured for the time being in our Startup class. (Code that I presume is irrelevant has been removed for brevity.)

        string _contentRootPath;

        public Startup(IHostingEnvironment hostingEnvironment)
        {
            _contentRootPath = hostingEnvironment.ContentRootPath;
        }

        public IServiceProvider ConfigureServices(IServiceCollection services)
        {
            var pfxFileInfo = new DirectoryInfo(_contentRootPath).EnumerateFiles("*.pfx").Single();
            var x509Cert = new X509Certificate2(File.ReadAllBytes(pfxFileInfo.FullName), "supersecretpassword");
            services.AddDataProtection()
                .SetApplicationName("hrselfservice")
                .ProtectKeysWithCertificate(x509Cert)
                .PersistKeysToFileSystem(new DirectoryInfo(_contentRootPath));
        }

For the purposes of testing, a self-signed pfx file exists at the location corresponding to the ContentRootPath defined above.

The following controller effectively allows me to reproduce the exception we are getting:

    public class DataProtectionController : Controller
    {
        private readonly IDataProtector _dataProtector;

        public DataProtectionController(IDataProtectionProvider dataProtectionProvider) 
        {
            _dataProtector = dataProtectionProvider.CreateProtector("TestDataProtection");
        }

        [HttpGet]
        public bool TestDataProtection()
        {
            var originalData = "This is only a test";
            var protectedData = _dataProtector.Protect(originalData);
            var unprotectedData = _dataProtector.Unprotect(protectedData);
            return unprotectedData == originalData;
        }
    }

When the Protect method (above) is called, a new "key" file (e.g. key-4c76c163-3842-42ae-a502-d1c92e9f5aa0.xml) is created (with encrypted data) alongside the .pfx file, and then the following exception is thrown, presumably while it's trying to use data from the newly created key file:

System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyEncryptorDelegate>b__0()
   at System.Lazy`1.CreateValue()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.get_Value()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptorInstance()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.KeyHolder.GetEncryptorInstance(Boolean& isRevoked)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.get_DefaultAuthenticatedEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   at Microsoft.AspNetCore.DataProtection.DataProtectionExtensions.Protect(IDataProtector protector, String plaintext)

According to the documentation, I believe we're using the API correctly, but perhaps we're missing something about how the X509 certificate is supposed to be set up for decryption of key data. Can you help confirm that we're using the API correctly?

It's worth noting that if we comment out the .ProtectKeysWithCertificate(x509Cert) line in the above example, the key file is created (without the encrypted key value, of course), and the Protect and Unprotect methods of IDataProtector work without throwing any exceptions.

And BTW, we're using version "1.0.0-rc2-20254" of Microsoft.AspNetCore.DataProtection.

[muratg]

cc @blowdart

[blowdart]

For decryption the certificate must be in the certificate store.It's a limitation of how EncryptedXml works.

[chadmyers]

@blowdart How does that work in a load balanced environment?

As I understand it, we need to make sure the same key/cert is in the machine store (or user store?) on each machine in the load balanced and in a pfx file on the filesystem. Does that sound correct?

[porcus]

Thanks for the speedy response, @blowdart! I must have missed that crucial point when reading over the docs. I'll give that a shot.

[blowdart]

It might not actually be in the docs. Oops.

[blowdart]

@chadmyers You don't have to have the PFX on the file system. Shove it in the store, then pull it out of the store via the X509CertificateStore class. That is, frankly, safer, as you don't have a file floating around, and you don't have to put a password anywhere.

[chadmyers]

@blowdart I was going to submit a PR for the docs once I understand how it works. I can help there if you need/want it

[blowdart]

Heh. The docs are all kinds of special. The docs repo has instructions on how to get started.

[porcus]

I was finally able to get this to work -- but not quite as easily as I'd hoped. Here's what I did...

1. Added the certificate to the local machine store.
2. Copied the thumbprint from the certificate info in the store.
3. Set up data protection in the app using the following:

public const string X509_CERTIFICATE_THUMBPRINT = "b2e65f7fa52306b55c3ca7c6a4dada1a32650754";

public IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddDataProtection()
        .SetApplicationName("hrselfservice")
        .ProtectKeysWithCertificate(X509_CERTIFICATE_THUMBPRINT);
}

When I ran the app, I got an InvalidOperationException in Microsoft.AspNetCore.DataProtection.dll with the message A certificate with the thumbprint 'b2e65f7fa52306b55c3ca7c6a4dada1a32650754' could not be found. I couldn't figure out what I hadn't configured properly. After all, the certificate was definitely in the store, and I had the correct thumbprint. I then realized I could use an alternate approach:

public const string X509_CERTIFICATE_THUMBPRINT = "b2e65f7fa52306b55c3ca7c6a4dada1a32650754";

public IServiceProvider ConfigureServices(IServiceCollection services)
{
    X509Store store = new X509Store(StoreLocation.LocalMachine);
    store.Open(OpenFlags.ReadOnly);
    X509Certificate2Collection x509Certificate2Collection = store.Certificates.Find(X509FindType.FindByThumbprint, X509_CERTIFICATE_THUMBPRINT, false);
    X509Certificate2 x509Cert = x509Certificate2Collection.Count > 0 ? x509Certificate2Collection[0] : null;
    store.Close();

    services.AddDataProtection()
        .SetApplicationName("hrselfservice")
        .ProtectKeysWithCertificate(x509Cert);
}

The latter approach worked! It was able to find the cert by its thumbprint and register it for key protection. And that time around, the key file generated had an encrpyted value as expected, and no CryptographicException resulted when attempting to use the key. Success!

My final question is simply this: Why did calling ProtectKeysWithCertificate(string thumbprint) with a seemingly valid thumbprint not work for me? Are there some additional requirements around how/where the certificate is supposed to be set up in the certificate store that I am not aware of? I don't want to use this latter approach that seems to work if there are some drawbacks to using it of which I'm not aware.

[chadmyers]

I believe the issue that @porcus ran into (thumbprint not found) is the same or related to #125

[tillig]

I was going to store my certificate in Azure Key Vault rather than the certificate store on the machine, so this isn't too cool. I get that's a limitation of EncryptedXml, but still less than stellar. It means I either need to set the certificate as part of a provisioning step or I need to do something probably pretty custom to allow the certificate to come from somewhere not in the certificate store.

[autodidaddict]

I second @tillig 's thoughts. There's nothing cloud native about requiring something to be physically installed on a machine (our machines should all be disposable and ephemeral, not pets) in order for people to have farm-friendly data protection, which is required for horizontally scalable apps that participate in virtually any kind of secure transactions.

[tillig]

I worked around this by creating a custom IXmlEncryptor and IXmlDecryptor pair that uses the manual asymmetric encryption/decryption mechanism outlined on MSDN. It's not a lot of code and might be something interesting to provide out of the box to work around these sorts of challenges.

[tillig]

For reference, I posted my custom encryptor/decryptor here.