Unable to retrieve the decryption key when using X509 certificate to protect keys

[urbanhusky]

I am using .ProtectKeysWithCertificate(X509Certificate2) to protect the keys at rest with an X509 certificate. Creation and encryption of the initial key works fine - but it cannot be decrypted.
Subsequent restarts just add more keys that fail to decrypt.

AspNetCore.All v2.0.3

DPAPI configuration:

services.AddDbContext<DataProtectionDbContext>(
    opts =>
    {
        var dpapiMigrationsAssembly = typeof(DataProtectionDbContext).GetTypeInfo().Assembly.GetName().Name;
        opts.UseSqlServer(dpapiConnectionString, b => b.MigrationsAssembly(dpapiMigrationsAssembly));
    },
    ServiceLifetime.Transient); // I don't think that Scoped would be a good idea when the repository is most likely registered as a singleton

var intermittentBuilder = services.BuildServiceProvider();
services.AddDataProtection()
    .ProtectKeysWithCertificate(GetCertificate()) // GetCertificate() loads an X509Certificate2 from disk
    .AddKeyManagementOptions(options => options.XmlRepository = new SqlDatabaseXmlRepository(intermittentBuilder)) // custom IXmlRepository, needs to resolve DataProtectionDbContext hence passing IServiceProvider

Errors:

Microsoft.EntityFrameworkCore.Infrastructure:Information: Entity Framework Core 2.0.1-rtm-125 initialized 'DataProtectionDbContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer' with options: MigrationsAssembly=Senseforce.Authentication.Web 
Microsoft.EntityFrameworkCore.Database.Command:Information: Executed DbCommand (5ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT [k].[XmlData]
FROM [dpapi].[DataProtectionKeys] AS [k]
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager:Information: Creating key {1f208812-c07e-4c45-b231-ab7923ea4bbd} with creation date 2017-11-27 12:55:08Z, activation date 2017-11-27 12:55:08Z, and expiration date 2018-02-25 12:55:08Z.
Microsoft.EntityFrameworkCore.Infrastructure:Information: Entity Framework Core 2.0.1-rtm-125 initialized 'DataProtectionDbContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer' with options: MigrationsAssembly=Senseforce.Authentication.Web 
Microsoft.EntityFrameworkCore.Database.Command:Information: Executed DbCommand (24ms) [Parameters=[@__friendlyName_0='?' (Size = 450)], CommandType='Text', CommandTimeout='30']
SELECT TOP(2) [k].[FriendlyName], [k].[XmlData]
FROM [dpapi].[DataProtectionKeys] AS [k]
WHERE [k].[FriendlyName] = @__friendlyName_0
Microsoft.EntityFrameworkCore.Database.Command:Information: Executed DbCommand (1ms) [Parameters=[@p0='?' (Size = 450), @p1='?' (Size = -1)], CommandType='Text', CommandTimeout='30']
SET NOCOUNT ON;
INSERT INTO [dpapi].[DataProtectionKeys] ([FriendlyName], [XmlData])
VALUES (@p0, @p1);
Microsoft.EntityFrameworkCore.Infrastructure:Information: Entity Framework Core 2.0.1-rtm-125 initialized 'DataProtectionDbContext' using provider 'Microsoft.EntityFrameworkCore.SqlServer' with options: MigrationsAssembly=Senseforce.Authentication.Web 
Microsoft.EntityFrameworkCore.Database.Command:Information: Executed DbCommand (0ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT [k].[XmlData]
FROM [dpapi].[DataProtectionKeys] AS [k]
Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager:Error: An exception occurred while processing the key element '<key id="1f208812-c07e-4c45-b231-ab7923ea4bbd" version="1" />'.

System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver:Warning: Key {1f208812-c07e-4c45-b231-ab7923ea4bbd} is ineligible to be the default key because its CreateEncryptor method failed.

System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key)
Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver:Warning: Key {1f208812-c07e-4c45-b231-ab7923ea4bbd} is ineligible to be the default key because its CreateEncryptor method failed.

System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DeferredKey.<>c__DisplayClass1_0.<GetLazyDescriptorDelegate>b__0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.CreateValue()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyBase.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key)

[urbanhusky]

What I've been able to figure out so far is:

• XML is encrypted via CertificateXmlEncryptor
• CertificateXmlEncryptor uses EncryptedXml.Encrypt(XmlElement, X509Certificate2)
  • …which adds KeyInfoX509Data for a certificate to the XML
    • The certificate used is stored, in full, raw, in the <X509Certificate> element.
      • This tells me that the certificate must have a protected private key
    • <CipherData> contains a random RSA session key, encrypted with the certificate's public key
• The actual key is encrypted with the randomly generated RSA session key
• Encrypted XML is stored in repository
• Encrypted XML can be read from repository
• Encrypted XML defines that the encrypted secret is to be decrypted with Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor
  • EncryptedXmlDecryptor uses System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
    • calls GetDecryptionKey(EncryptedData, null), which most likely calls DecryptEncryptedKey(EncryptedKey)
    • …which calls Utils.BuildBagOfCerts(KeyInfoX509Data, CertUsageType.Decryption)
      • …which builds a X509IssuerSerial
      • and uses X509Store to load the certificate
        • which renders using .ProtectKeysWithCertificate(X509Certificate2) abolutely useless because the certificate has to be loaded from the cert store anyway (which is not really feasible for our use case - docker swarm, different certs for each environment + having to set up each dev-environment etc.)

Why do you even offer that overload?

Example encrypted XML:

<key id="f7523279-3986-40d5-b909-64a1e1a0bc72" version="1">
  <creationDate>2017-11-27T10:39:19.6595798Z</creationDate>
  <activationDate>2017-11-27T10:39:19.092618Z</activationDate>
  <expirationDate>2018-02-25T10:39:19.092618Z</expirationDate>
  <descriptor deserializerType="Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=2.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60">
    <descriptor>
      <encryption algorithm="AES_256_CBC" />
      <validation algorithm="HMACSHA256" />
      <encryptedSecret decryptorType="Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor, Microsoft.AspNetCore.DataProtection, Version=2.0.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60" xmlns="http://schemas.asp.net/2015/03/dataProtection">
        <EncryptedData Type="http://www.w3.org/2001/04/xmlenc#Element" xmlns="http://www.w3.org/2001/04/xmlenc#">
          <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#aes256-cbc" />
          <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
            <EncryptedKey xmlns="http://www.w3.org/2001/04/xmlenc#">
              <EncryptionMethod Algorithm="http://www.w3.org/2001/04/xmlenc#rsa-1_5" />
              <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                <X509Data>
                  <X509Certificate><!-- redacted --></X509Certificate>
                </X509Data>
              </KeyInfo>
              <CipherData>
                <CipherValue><!-- redacted --></CipherValue>
              </CipherData>
            </EncryptedKey>
          </KeyInfo>
          <CipherData>
            <CipherValue><!-- redacted --></CipherValue>
          </CipherData>
        </EncryptedData>
      </encryptedSecret>
    </descriptor>
  </descriptor>
</key>

[blowdart]

This is a failing in the underlying framework unfortunately. We can encrypt with a cert loaded from a PFX, but not decrypt. Decrypt is limited to search the store for a matching certificate. We could look at deleting that overload, but it's not going to help you much if you say you can't use the certificate store, and it'd have to be two major versions away, one to mark it as obsolete, then another to remove it.

[urbanhusky]

Thank you

[natemcmaster]

abolutely useless because the certificate has to be loaded from the cert store anyway (which is not really feasible for our use case - docker swarm, different certs for each environment + having to set up each dev-environment etc.)

If you have the x509certificate file on disk, you should be able to load it into the store so EncryptedXml can use it. Try adding this when loading the certificate:

var store = new X509Store(StoreName.My, StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadWrite);
store.Add(new X509Certificate2(filePathToYourCert, certFilePassword, X509KeyStorageFlags.Exportable));
store.Close();

See dotnet/aspnetcore#2321 (comment)

[natemcmaster]

Fixed in #299