Add Azure Storage / Azure Key Vault extensibility to DataProtection #92

Closed
GrabYourPitchforks opened this Issue Sep 8, 2015 · 53 comments

Comments

Projects
None yet
9 participants
@GrabYourPitchforks
Contributor

GrabYourPitchforks commented Sep 8, 2015

These solve two different but related problems. The DataProtection stack requires that all machines in the environment are able to point to the same key repository so that they can share key information. Out-of-box the DataProtection stack has support for the file system (including UNC paths) and the Windows registry. By adding support for Azure Storage, applications running within distributed environments would be able to use that as an alternative repository.

Once all machines agree on a key repository, they're faced with the problem of key protection at rest, since we generally don't want the keys sitting around in plaintext in the repository. The DataProtection stack has built-in support for using Windows DPAPI, CNG DPAPI, or an X.509 certificate. Adding support for Azure Key Vault would offer a mechanism for easing the burden of secret management in a distributed environment, and it would complement support for Azure Storage.

Not sure if this would be better suited as a sample application, but throwing the idea out there.

@muratg muratg added this to the 1.0.0 backlog milestone Sep 9, 2015

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg Sep 29, 2015

Member

I don't think this will fit in V1. We should probably write a sample around it when we document these things better.

Member

muratg commented Sep 29, 2015

I don't think this will fit in V1. We should probably write a sample around it when we document these things better.

@muratg muratg modified the milestones: Backlog, 1.0.0 backlog Sep 29, 2015

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Oct 13, 2015

👍 I think this is somewhat critical given that so much is moving to Azure right now. A very common scenario will be using Antiforgery with forms in web farm apps across Azure VM's, which creates and validates tokens with the data protection system. Azure Files might cut it for a network share (?) but the docs state that Core CLR cannot use the X.509 certificate bits to secure the keys on the share (if I'm reading that correctly).

Even if it were very manual at this point, a sample for hacking up an Azure Key Vault-to-Data Protection IKeyManager or XmlKeyManager implementation would be very helpful in the interim. AFAIK there is no sample anywhere of how this is done, correct?

👍 I think this is somewhat critical given that so much is moving to Azure right now. A very common scenario will be using Antiforgery with forms in web farm apps across Azure VM's, which creates and validates tokens with the data protection system. Azure Files might cut it for a network share (?) but the docs state that Core CLR cannot use the X.509 certificate bits to secure the keys on the share (if I'm reading that correctly).

Even if it were very manual at this point, a sample for hacking up an Azure Key Vault-to-Data Protection IKeyManager or XmlKeyManager implementation would be very helpful in the interim. AFAIK there is no sample anywhere of how this is done, correct?

@GrabYourPitchforks

This comment has been minimized.

Show comment
Hide comment
@GrabYourPitchforks

GrabYourPitchforks Oct 13, 2015

Contributor

@guardrex There's a workaround for doing certificate-based encryption if you're on Core CLR. However, it requires that you be on Windows 8.1 / Windows Server 2012 R2. See https://docs.asp.net/en/latest/security/data-protection/implementation/key-encryption-at-rest.html#certificate-based-encryption-with-windows-dpapi-ng.

You're correct in that there's currently no sample for using Azure Key Vault.

Contributor

GrabYourPitchforks commented Oct 13, 2015

@guardrex There's a workaround for doing certificate-based encryption if you're on Core CLR. However, it requires that you be on Windows 8.1 / Windows Server 2012 R2. See https://docs.asp.net/en/latest/security/data-protection/implementation/key-encryption-at-rest.html#certificate-based-encryption-with-windows-dpapi-ng.

You're correct in that there's currently no sample for using Azure Key Vault.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Oct 13, 2015

@GrabYourPitchforks Thanks ... I'll give that a shot. I'm going to try this with Azure File Storage.

@GrabYourPitchforks Thanks ... I'll give that a shot. I'm going to try this with Azure File Storage.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Oct 13, 2015

@GrabYourPitchforks This isn't entirely surprising: I setup an Azure File Storage share with a key directory and confirmed the share was working on the VM, then used this ...

configure.PersistKeysToFileSystem(new DirectoryInfo(@"\\<my-storage-account-name>.file.core.windows.net\dataprotection\keys\"));

... and it looks like its choking on the age-old problem of permissions. With .NET/IIS working on ApplicationPoolIdentity or NetworkService, it doesn't seem to have permission to access the share, so navigating to a view with Antiforgery configured, it fails with ...

IOException: The parameter is incorrect
    System.IO.Win32FileSystem.CreateDirectory(String fullPath)
    Microsoft.AspNet.DataProtection.Repositories.FileSystemXmlRepository.<GetAllElementsCore>d__15.MoveNext()
    System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
    System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
    Microsoft.AspNet.DataProtection.Repositories.FileSystemXmlRepository.GetAllElements()
    Microsoft.AspNet.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNet.DataProtection.KeyManagement.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow)
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)

Yeah, you have these flashbacks to the good 'ole days of impersonation! Yuck. I'll investigate making the share available to the app further; but given your familiarity with the packages and implementation, do you have any tips?

@GrabYourPitchforks This isn't entirely surprising: I setup an Azure File Storage share with a key directory and confirmed the share was working on the VM, then used this ...

configure.PersistKeysToFileSystem(new DirectoryInfo(@"\\<my-storage-account-name>.file.core.windows.net\dataprotection\keys\"));

... and it looks like its choking on the age-old problem of permissions. With .NET/IIS working on ApplicationPoolIdentity or NetworkService, it doesn't seem to have permission to access the share, so navigating to a view with Antiforgery configured, it fails with ...

IOException: The parameter is incorrect
    System.IO.Win32FileSystem.CreateDirectory(String fullPath)
    Microsoft.AspNet.DataProtection.Repositories.FileSystemXmlRepository.<GetAllElementsCore>d__15.MoveNext()
    System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
    System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
    Microsoft.AspNet.DataProtection.Repositories.FileSystemXmlRepository.GetAllElements()
    Microsoft.AspNet.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNet.DataProtection.KeyManagement.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow)
    Microsoft.AspNet.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)

Yeah, you have these flashbacks to the good 'ole days of impersonation! Yuck. I'll investigate making the share available to the app further; but given your familiarity with the packages and implementation, do you have any tips?

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Oct 15, 2015

Let's say I go with a custom IXmlRepository and use Azure Table Storage. Something like this:

In ConfigureServices(...) of Startup.cs

services.AddDataProtection();
var serviceDescriptor = new ServiceDescriptor(typeof(IXmlRepository), typeof(ICustomXmlRepository), ServiceLifetime.Singleton);
services.Add(serviceDescriptor);

XmlRepository.cs [The FetchAll() and Insert() are methods for grabbing and saving the keys. This app will have its own Azure Table for the keys, and multiple instances of the app will be hitting the table.]

using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Xml.Linq;
using Microsoft.AspNet.DataProtection.Repositories;
using ShorehamPublic.Models;

namespace MyApp.Repositories
{
    public class ICustomXmlRepository : IXmlRepository
    {
        public IReadOnlyCollection<XElement> GetAllElements()
        {
            IList<XElement> keyList = new List<XElement>();
            IEnumerable<DataProtectionKey> keyEntities = DataProtectionKey.FetchAll().Result;
            foreach (var key in keyEntities)
            {
                keyList.Add(XElement.Parse(key.Data));
            }
            IReadOnlyCollection<XElement> readElements = new ReadOnlyCollection<XElement>(keyList);
            return readElements;
        }

        public async void StoreElement(XElement element, string friendlyName)
        {
            await DataProtectionKey.Insert(element.ToString());
        }
    }
}
  1. This seems to work ok in light testing, but what would the gotchas be for having instances of the app using this approach?
  2. If this is the only app (with multiple instances) using a table of keys, I don't need to supply an application name in configuration, correct?
  3. I noted <!-- Warning: the key below is in an unencrypted form. --> above the key value. What's the best way to handle this? Can I encrypt with out-of-the-box methods/configuration, or should I encrypt and decrypt the whole XElement in my table storage model methods?
  4. What about removing the table entities? I've created the entities with a timestamp RowKey. I presume I should clear the old keys on my own ... keys older than 90 days can be deleted from the table?
  5. Did I make a royal code smell anywhere? ... E.g., Did I register the custom IXmlRepository correctly? Does the implementation of the custom repository look ok?

Let's say I go with a custom IXmlRepository and use Azure Table Storage. Something like this:

In ConfigureServices(...) of Startup.cs

services.AddDataProtection();
var serviceDescriptor = new ServiceDescriptor(typeof(IXmlRepository), typeof(ICustomXmlRepository), ServiceLifetime.Singleton);
services.Add(serviceDescriptor);

XmlRepository.cs [The FetchAll() and Insert() are methods for grabbing and saving the keys. This app will have its own Azure Table for the keys, and multiple instances of the app will be hitting the table.]

using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Xml.Linq;
using Microsoft.AspNet.DataProtection.Repositories;
using ShorehamPublic.Models;

namespace MyApp.Repositories
{
    public class ICustomXmlRepository : IXmlRepository
    {
        public IReadOnlyCollection<XElement> GetAllElements()
        {
            IList<XElement> keyList = new List<XElement>();
            IEnumerable<DataProtectionKey> keyEntities = DataProtectionKey.FetchAll().Result;
            foreach (var key in keyEntities)
            {
                keyList.Add(XElement.Parse(key.Data));
            }
            IReadOnlyCollection<XElement> readElements = new ReadOnlyCollection<XElement>(keyList);
            return readElements;
        }

        public async void StoreElement(XElement element, string friendlyName)
        {
            await DataProtectionKey.Insert(element.ToString());
        }
    }
}
  1. This seems to work ok in light testing, but what would the gotchas be for having instances of the app using this approach?
  2. If this is the only app (with multiple instances) using a table of keys, I don't need to supply an application name in configuration, correct?
  3. I noted <!-- Warning: the key below is in an unencrypted form. --> above the key value. What's the best way to handle this? Can I encrypt with out-of-the-box methods/configuration, or should I encrypt and decrypt the whole XElement in my table storage model methods?
  4. What about removing the table entities? I've created the entities with a timestamp RowKey. I presume I should clear the old keys on my own ... keys older than 90 days can be deleted from the table?
  5. Did I make a royal code smell anywhere? ... E.g., Did I register the custom IXmlRepository correctly? Does the implementation of the custom repository look ok?
@GrabYourPitchforks

This comment has been minimized.

Show comment
Hide comment
@GrabYourPitchforks

GrabYourPitchforks Oct 19, 2015

Contributor

/cc @blowdart

I reworked the existing Azure extensions for DataProtection I've been working on and migrated the project to https://github.com/GrabYourPitchforks/DataProtection.Azure/tree/dev. Feel free to give that a shot if you're looking for something quick and dirty. Maybe one day it can be contributed back into the main project. :)

In particular, the system ends up being more reliable if you use blobs instead of tables. Since the data is XML-based it makes more sense for the data to be in a file (blob) format rather than as a key/value pair in a table. This also makes read operations much faster since you'll end up reading all available data in a single operation rather than traversing all entries in a table.

To your other questions: if there's only one application pointed at a repository, you don't need to worry about the app name. You also don't need to worry about cleaning up old keys. The system's automatic key management will never delete old keys since deleting a key would render existing encrypted data undecipherable.

You can use the existing key encryption methods like certs. The system is designed such that repositories and key encryption methods are separate concerns and can be mixed-and-matched freely.

I'm also working on Azure Key Vault support in the project mentioned above, but it's not there yet. The combination of Azure Storage + Azure Key Vault would be a nice end-to-end "everything's hosted in the cloud and is automatic" story.

Contributor

GrabYourPitchforks commented Oct 19, 2015

/cc @blowdart

I reworked the existing Azure extensions for DataProtection I've been working on and migrated the project to https://github.com/GrabYourPitchforks/DataProtection.Azure/tree/dev. Feel free to give that a shot if you're looking for something quick and dirty. Maybe one day it can be contributed back into the main project. :)

In particular, the system ends up being more reliable if you use blobs instead of tables. Since the data is XML-based it makes more sense for the data to be in a file (blob) format rather than as a key/value pair in a table. This also makes read operations much faster since you'll end up reading all available data in a single operation rather than traversing all entries in a table.

To your other questions: if there's only one application pointed at a repository, you don't need to worry about the app name. You also don't need to worry about cleaning up old keys. The system's automatic key management will never delete old keys since deleting a key would render existing encrypted data undecipherable.

You can use the existing key encryption methods like certs. The system is designed such that repositories and key encryption methods are separate concerns and can be mixed-and-matched freely.

I'm also working on Azure Key Vault support in the project mentioned above, but it's not there yet. The combination of Azure Storage + Azure Key Vault would be a nice end-to-end "everything's hosted in the cloud and is automatic" story.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Oct 19, 2015

@GrabYourPitchforks That's great. Thanks.

Since the data is XML-based it makes more sense for the data to be in a file (blob) format rather than as a key/value pair in a table. This also makes read operations much faster since you'll end up reading all available data in a single operation rather than traversing all entries in a table.

That table storage version I did seems to work, but I didn't stress test it yet. I feel you though on the "traversing entities," which is why I envision a single table per app for this and used a common PartitionKey for all entities in the table (i.e., to pull back all entities in one bite from a single edge server). I'm also taking the entire Xml and just plopping it into a property of the entity. I'm not trying to parse it and not dealing with key-value pairs. I'll do a stress test on it and see how it holds up.

The system's automatic key management will never delete old keys since deleting a key would render existing encrypted data undecipherable.

This is very interesting! The only use I have for this for the moment is for Antiforgery. If I recall correctly, the Antiforgery header/cookie system is setup for 90-day expiration out-of-the-box (?); so if I were only using Data Protection for Antiforgery, I guess I could kill old keys at 90-days for sure ... but that would be a strange case where someone opens a webform and leaves it open for as long as 89 days and finally POST's back to the server hoping for an Antiforgery validation to work. In any case, it would be easy for me to drop old keys, since my RowKey is actually a timestamp and easy to filter/delete on periodically.

I'm also working on Azure Key Vault support in the project mentioned above, but it's not there yet. The combination of Azure Storage + Azure Key Vault would be a nice end-to-end "everything's hosted in the cloud and is automatic" story.

Yeah, that's going to be excellent. That will be the gold standard for Azure-hosted apps IMHO. I'll give your DataProtection.Azure a shot, and I'll keep an eye out for Azure Storage + Azure Key Vault.

One ? though ... about the <!-- Warning: the key below is in an unencrypted form. --> note above the key value in the Xml. What's the best way to handle this if I want it encrypted? Is there something in-the-box, or should I just encrypt/decrypt the whole Xml payload just before dropping into storage and just before handing back to Data Protection?

@GrabYourPitchforks That's great. Thanks.

Since the data is XML-based it makes more sense for the data to be in a file (blob) format rather than as a key/value pair in a table. This also makes read operations much faster since you'll end up reading all available data in a single operation rather than traversing all entries in a table.

That table storage version I did seems to work, but I didn't stress test it yet. I feel you though on the "traversing entities," which is why I envision a single table per app for this and used a common PartitionKey for all entities in the table (i.e., to pull back all entities in one bite from a single edge server). I'm also taking the entire Xml and just plopping it into a property of the entity. I'm not trying to parse it and not dealing with key-value pairs. I'll do a stress test on it and see how it holds up.

The system's automatic key management will never delete old keys since deleting a key would render existing encrypted data undecipherable.

This is very interesting! The only use I have for this for the moment is for Antiforgery. If I recall correctly, the Antiforgery header/cookie system is setup for 90-day expiration out-of-the-box (?); so if I were only using Data Protection for Antiforgery, I guess I could kill old keys at 90-days for sure ... but that would be a strange case where someone opens a webform and leaves it open for as long as 89 days and finally POST's back to the server hoping for an Antiforgery validation to work. In any case, it would be easy for me to drop old keys, since my RowKey is actually a timestamp and easy to filter/delete on periodically.

I'm also working on Azure Key Vault support in the project mentioned above, but it's not there yet. The combination of Azure Storage + Azure Key Vault would be a nice end-to-end "everything's hosted in the cloud and is automatic" story.

Yeah, that's going to be excellent. That will be the gold standard for Azure-hosted apps IMHO. I'll give your DataProtection.Azure a shot, and I'll keep an eye out for Azure Storage + Azure Key Vault.

One ? though ... about the <!-- Warning: the key below is in an unencrypted form. --> note above the key value in the Xml. What's the best way to handle this if I want it encrypted? Is there something in-the-box, or should I just encrypt/decrypt the whole Xml payload just before dropping into storage and just before handing back to Data Protection?

@GrabYourPitchforks

This comment has been minimized.

Show comment
Hide comment
@GrabYourPitchforks

GrabYourPitchforks Oct 20, 2015

Contributor

Call any of the ProtectKeysWith* APIs, such as ProtectKeysWithCertificate, during the configuration routine. You can mix and match ProtectKeysWith* and PersistKeysTo* calls. The system is designed such the data repository doesn't know anything about how keys are encrypted at rest, and the key protection mechanism doesn't know anything about how the key files are stored.

Contributor

GrabYourPitchforks commented Oct 20, 2015

Call any of the ProtectKeysWith* APIs, such as ProtectKeysWithCertificate, during the configuration routine. You can mix and match ProtectKeysWith* and PersistKeysTo* calls. The system is designed such the data repository doesn't know anything about how keys are encrypted at rest, and the key protection mechanism doesn't know anything about how the key files are stored.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Oct 20, 2015

@GrabYourPitchforks Good grief. Sorry about that ... we just touched on that above. Thanks for your help ... I'm good now.

@GrabYourPitchforks Good grief. Sorry about that ... we just touched on that above. Thanks for your help ... I'm good now.

@blowdart

This comment has been minimized.

Show comment
Hide comment
@blowdart

blowdart Nov 13, 2015

Member

@muratg We should revisit this for RC2. People are asking about it more and more.

Member

blowdart commented Nov 13, 2015

@muratg We should revisit this for RC2. People are asking about it more and more.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Nov 13, 2015

👍 for Azure Table Storage as one of the offerings.

👍 for Azure Table Storage as one of the offerings.

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg Nov 13, 2015

Member

I like @GrabYourPitchforks' extension that uses blob storage. We should consider getting it in our branch. Putting this in RC2 for now to consider in RC2.

Member

muratg commented Nov 13, 2015

I like @GrabYourPitchforks' extension that uses blob storage. We should consider getting it in our branch. Putting this in RC2 for now to consider in RC2.

@muratg muratg modified the milestones: 1.0.0-rc2, Backlog Nov 13, 2015

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Nov 13, 2015

@muratg I know ... it's a "file"-type structure/format. I was just saying if multiple options could exist, it would be nice to have a choice.

@muratg I know ... it's a "file"-type structure/format. I was just saying if multiple options could exist, it would be nice to have a choice.

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg Nov 13, 2015

Member

@guardrex agreed!

Azure Key Vault support would also be nice BTW.

Member

muratg commented Nov 13, 2015

@guardrex agreed!

Azure Key Vault support would also be nice BTW.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Nov 13, 2015

@muratg Oh, yes, of course. I really like Azure Key Vault. Would love to use that for this.

@muratg Oh, yes, of course. I really like Azure Key Vault. Would love to use that for this.

@cwe1ss

This comment has been minimized.

Show comment
Hide comment
@cwe1ss

cwe1ss Mar 8, 2016

👍 on Azure Key Vault.

Is there any progress update on this topic?

cwe1ss commented Mar 8, 2016

👍 on Azure Key Vault.

Is there any progress update on this topic?

@muratg muratg modified the milestones: 1.0.0, 1.0.0-rc2 Mar 21, 2016

@dfederm

This comment has been minimized.

Show comment
Hide comment
@dfederm

dfederm May 6, 2016

Ah, so this is why my users have to log in again every time I slotswap... Any way for Azure Websites to just support this out of the box across slots, or at least sticky to the slot?

dfederm commented May 6, 2016

Ah, so this is why my users have to log in again every time I slotswap... Any way for Azure Websites to just support this out of the box across slots, or at least sticky to the slot?

@cwe1ss

This comment has been minimized.

Show comment
Hide comment
@cwe1ss

cwe1ss May 19, 2016

We just tried to set this up on a custom VM (running Azure Service Fabric) and an Azure file storage share, but dealing with the permissions is just a huge PITA so I gave up. I will try to use @GrabYourPitchforks 's azure blob storage provider now.

I really think you should provide at least one official cluster-aware storage provider for RTM - preferable based on some Azure stuff to have a good Azure story.

If you don't provide something here, I can already see the countless hours wasted by people trying to get their UNC share permissions working :-(

cwe1ss commented May 19, 2016

We just tried to set this up on a custom VM (running Azure Service Fabric) and an Azure file storage share, but dealing with the permissions is just a huge PITA so I gave up. I will try to use @GrabYourPitchforks 's azure blob storage provider now.

I really think you should provide at least one official cluster-aware storage provider for RTM - preferable based on some Azure stuff to have a good Azure story.

If you don't provide something here, I can already see the countless hours wasted by people trying to get their UNC share permissions working :-(

@cwe1ss

This comment has been minimized.

Show comment
Hide comment
@cwe1ss

cwe1ss May 19, 2016

I upgraded @GrabYourPitchforks 's code to RC2 in case anyone needs it: GrabYourPitchforks/DataProtection.Azure#1

cwe1ss commented May 19, 2016

I upgraded @GrabYourPitchforks 's code to RC2 in case anyone needs it: GrabYourPitchforks/DataProtection.Azure#1

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg Jun 29, 2016

Member

@pakrym Could you investigate this one?

Member

muratg commented Jun 29, 2016

@pakrym Could you investigate this one?

@dfederm

This comment has been minimized.

Show comment
Hide comment
@dfederm

dfederm Aug 24, 2016

I'm not sure how blob storage is inherently more secure than providing a string, especially since the string could come from somewhere secure. For example a slot setting. I do see how a hard coded literal key would be less secure, but it won't always be a literal.

For storage anyway you need to have some way to get an access key to the machine so that it can go talk to blob storage (or keyvault) to get the secret. It's basically using a secret to get a secret. So why not just have the original secret be the one that's used?

I realize that using slot settings may not scale to a ton of secrets, but for simple apps with just a handful of secrets, why add the extra layer?

dfederm commented Aug 24, 2016

I'm not sure how blob storage is inherently more secure than providing a string, especially since the string could come from somewhere secure. For example a slot setting. I do see how a hard coded literal key would be less secure, but it won't always be a literal.

For storage anyway you need to have some way to get an access key to the machine so that it can go talk to blob storage (or keyvault) to get the secret. It's basically using a secret to get a secret. So why not just have the original secret be the one that's used?

I realize that using slot settings may not scale to a ton of secrets, but for simple apps with just a handful of secrets, why add the extra layer?

@gdoron

This comment has been minimized.

Show comment
Hide comment
@gdoron

gdoron Aug 24, 2016

Well, after almost an hour of trying to implement the UseThisBloodyKeyForeverSecurityIsOverratedBro, I realized you made it impossible 😄 because I still need to implement a repository to store the keys for when I change slots on my azure web app.

I hope the official solution will be shipped soon, it's really missing.
I'll just not change slots until it'll be shipped 😢

gdoron commented Aug 24, 2016

Well, after almost an hour of trying to implement the UseThisBloodyKeyForeverSecurityIsOverratedBro, I realized you made it impossible 😄 because I still need to implement a repository to store the keys for when I change slots on my azure web app.

I hope the official solution will be shipped soon, it's really missing.
I'll just not change slots until it'll be shipped 😢

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Aug 24, 2016

Data Protection creates and rotates.

I do key rotation using the DP bits. I use a master key and replace the key encryption and key decryption bits to use my master key.

services.AddSingleton<IXmlRepository, CustomXmlRepository>();
services.AddSingleton<IXmlEncryptor, CustomXmlEncryptor>();
services.AddSingleton<IXmlDecryptor, CustomXmlDecryptor>();

This avoids me having to use certs on servers. Replacing the repository with Azure Table Storage prevents me having to use a network share. I'm happy with what I have and will keep an 👀 on development to see if/when I can switch over to MS bits.

Data Protection creates and rotates.

I do key rotation using the DP bits. I use a master key and replace the key encryption and key decryption bits to use my master key.

services.AddSingleton<IXmlRepository, CustomXmlRepository>();
services.AddSingleton<IXmlEncryptor, CustomXmlEncryptor>();
services.AddSingleton<IXmlDecryptor, CustomXmlDecryptor>();

This avoids me having to use certs on servers. Replacing the repository with Azure Table Storage prevents me having to use a network share. I'm happy with what I have and will keep an 👀 on development to see if/when I can switch over to MS bits.

@blowdart

This comment has been minimized.

Show comment
Hide comment
@blowdart

blowdart Aug 24, 2016

Member

@gdoron The slots thing azure knows about (and are hopefully working on)

Member

blowdart commented Aug 24, 2016

@gdoron The slots thing azure knows about (and are hopefully working on)

@gdoron

This comment has been minimized.

Show comment
Hide comment
@gdoron

gdoron Aug 24, 2016

(and are hopefully working on)

@blowdart From my experience that's not obviously granted, I couple of months ago emailed Scott Gu for constantly getting an (unstyled IIS) 404 on Azure registration!, he forwarded the email to several PMs and they emailed me, it will be fixed in several weeks...

gdoron commented Aug 24, 2016

(and are hopefully working on)

@blowdart From my experience that's not obviously granted, I couple of months ago emailed Scott Gu for constantly getting an (unstyled IIS) 404 on Azure registration!, he forwarded the email to several PMs and they emailed me, it will be fixed in several weeks...

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Aug 24, 2016

@blowdart As an aside tho: If you only register replacements for IXmlEncryptor and IXmlDecryptor ... not the repository IXmlRepository ... Why does it throw a .....
pasted image at 2016_08_24 11_08 pm
Apparently, they must all be replaced together.

@blowdart As an aside tho: If you only register replacements for IXmlEncryptor and IXmlDecryptor ... not the repository IXmlRepository ... Why does it throw a .....
pasted image at 2016_08_24 11_08 pm
Apparently, they must all be replaced together.

@gdoron

This comment has been minimized.

Show comment
Hide comment
@gdoron

gdoron Aug 24, 2016

Just to be sure, even the AzureBlobStorageProvider that is being worked on won't work when I change slots unless Azure will fix that stuff, right?

Or did I get it wrong?

gdoron commented Aug 24, 2016

Just to be sure, even the AzureBlobStorageProvider that is being worked on won't work when I change slots unless Azure will fix that stuff, right?

Or did I get it wrong?

@cwe1ss

This comment has been minimized.

Show comment
Hide comment
@cwe1ss

cwe1ss Aug 24, 2016

I'd hope people would use a separate blob store for keys :)

That's very optimistic thinking. :) I'd use Atwood's JavaScript law and say "Everything that can be stored in one storage account, will eventually be stored in one Storage account". 😄

Plus KeyVault doesn't support saying "Give me all versions of this secret" which is also problematic for us, as we want to say "Give me all keys" and get them, without having to use a paging interface.

True, working with secrets is not very straight forward unfortunately.

However, one huge advantage of key vault is, that you can use certificate based authentication. This somehow removes the issue of "providing a secret to get a secret" as mentioned by @dfederm .

cwe1ss commented Aug 24, 2016

I'd hope people would use a separate blob store for keys :)

That's very optimistic thinking. :) I'd use Atwood's JavaScript law and say "Everything that can be stored in one storage account, will eventually be stored in one Storage account". 😄

Plus KeyVault doesn't support saying "Give me all versions of this secret" which is also problematic for us, as we want to say "Give me all keys" and get them, without having to use a paging interface.

True, working with secrets is not very straight forward unfortunately.

However, one huge advantage of key vault is, that you can use certificate based authentication. This somehow removes the issue of "providing a secret to get a secret" as mentioned by @dfederm .

@cwe1ss

This comment has been minimized.

Show comment
Hide comment
@cwe1ss

cwe1ss Aug 24, 2016

Seems like one secret can have up to 25kb so maybe you can combine/compress stuff?

cwe1ss commented Aug 24, 2016

Seems like one secret can have up to 25kb so maybe you can combine/compress stuff?

@dfederm

This comment has been minimized.

Show comment
Hide comment
@dfederm

dfederm Aug 24, 2016

@cwe1ss cert-based auth is still a secret, it's just a different way of delivering the secret. And unless I'm mistaken, I don't think Azure Websites supports installing certs on the machines anyway.

One thing I might have missed with the slot setting thing though is that if the data protection stuff also rotates the key itself then it wouldn't work since slot settings are read-only from the app (unless you do some shenanigans like call the management APIs from the app itself, but that requires extra secrets as well anyway). So storage/keyvault works since it allows the app to not only get the secret but also set it when rotating it.

dfederm commented Aug 24, 2016

@cwe1ss cert-based auth is still a secret, it's just a different way of delivering the secret. And unless I'm mistaken, I don't think Azure Websites supports installing certs on the machines anyway.

One thing I might have missed with the slot setting thing though is that if the data protection stuff also rotates the key itself then it wouldn't work since slot settings are read-only from the app (unless you do some shenanigans like call the management APIs from the app itself, but that requires extra secrets as well anyway). So storage/keyvault works since it allows the app to not only get the secret but also set it when rotating it.

@cwe1ss

This comment has been minimized.

Show comment
Hide comment
@cwe1ss

cwe1ss Aug 24, 2016

Of course, but it's using a different medium and no one can commit it to Source control (at least you must be pretty stupid to do so 😄). But handling certificates with expirations etc is also quite some work unfortunately.
I'm mostly using Service Fabric which allows to install certificates. But as far as I know you can upload certificates to Azure Web Apps - I think you must also set some magic appSettings key to actually be able to use them though.

cwe1ss commented Aug 24, 2016

Of course, but it's using a different medium and no one can commit it to Source control (at least you must be pretty stupid to do so 😄). But handling certificates with expirations etc is also quite some work unfortunately.
I'm mostly using Service Fabric which allows to install certificates. But as far as I know you can upload certificates to Azure Web Apps - I think you must also set some magic appSettings key to actually be able to use them though.

@blowdart

This comment has been minimized.

Show comment
Hide comment
@blowdart

blowdart Aug 24, 2016

Member

Blob storage will work between slots. And Web Apps do support certs.

Member

blowdart commented Aug 24, 2016

Blob storage will work between slots. And Web Apps do support certs.

@gdoron

This comment has been minimized.

Show comment
Hide comment
@gdoron

gdoron Sep 6, 2016

@gdoron The slots thing azure knows about (and are hopefully working on)

@blowdart FYI I was told by an Escalation Engineer from the ASFS program that if it will be implemented it probably won't happen in the near future.
He explained to me that WebApp should be agnostic to the application and platform it's being used for, so WebApp shouldn't handle ASP.NET Core differently.

He suggested I won't wait for the fix and use the code from the PR: #163

I think it's reasonable and makes sense.

gdoron commented Sep 6, 2016

@gdoron The slots thing azure knows about (and are hopefully working on)

@blowdart FYI I was told by an Escalation Engineer from the ASFS program that if it will be implemented it probably won't happen in the near future.
He explained to me that WebApp should be agnostic to the application and platform it's being used for, so WebApp shouldn't handle ASP.NET Core differently.

He suggested I won't wait for the fix and use the code from the PR: #163

I think it's reasonable and makes sense.

@blowdart

This comment has been minimized.

Show comment
Hide comment
@blowdart

blowdart Sep 6, 2016

Member

Yup. That'll will become part of the next core release.

As for the WebApp being agnostic, well I'd disagree, but it's Azure's choice.

Member

blowdart commented Sep 6, 2016

Yup. That'll will become part of the next core release.

As for the WebApp being agnostic, well I'd disagree, but it's Azure's choice.

@henningst

This comment has been minimized.

Show comment
Hide comment
@henningst

henningst Feb 24, 2017

I have just spent a few hours debugging a System.OperationCanceledException: The operation was canceled. which turned out to be caused by this:

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
      An unhandled exception has occurred: The antiforgery token could not be decrypted.
System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {9725081b-7caf-4642-ae55-93cf9c871c36} was not found in the key ring.

I now understand that this is happening because I'm deploying to a separate staging slot and then swapping to production to avoid downtime. I though the keys stored in /ASP.NET/DataProtection-Keys/ were synchronized between all sites, but this is obviously not the case for slots.

Will this be handled for me by the Web App platform in the future? If so, when?

What is the current recommended approach to handle this issue? I've seen a blob storage implementation here, but I would feel more comfortable if there was an "official" implementation I could use.

I have just spent a few hours debugging a System.OperationCanceledException: The operation was canceled. which turned out to be caused by this:

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
      An unhandled exception has occurred: The antiforgery token could not be decrypted.
System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {9725081b-7caf-4642-ae55-93cf9c871c36} was not found in the key ring.

I now understand that this is happening because I'm deploying to a separate staging slot and then swapping to production to avoid downtime. I though the keys stored in /ASP.NET/DataProtection-Keys/ were synchronized between all sites, but this is obviously not the case for slots.

Will this be handled for me by the Web App platform in the future? If so, when?

What is the current recommended approach to handle this issue? I've seen a blob storage implementation here, but I would feel more comfortable if there was an "official" implementation I could use.

@guardrex

This comment has been minimized.

Show comment
Hide comment
@guardrex

guardrex Feb 24, 2017

@henningst There's an official implementation for Azure Blob Storage. If you would like an Azure Table Storage implementation, you can hack your own key repository, but that's not recommended by MS.

@henningst There's an official implementation for Azure Blob Storage. If you would like an Azure Table Storage implementation, you can hack your own key repository, but that's not recommended by MS.

@henningst

This comment has been minimized.

Show comment
Hide comment
@henningst

henningst Feb 24, 2017

Exactly what I was looking for! Thanks @guardrex!

Exactly what I was looking for! Thanks @guardrex!

@muratg muratg modified the milestones: 2.0.0-preview2, 2.0.0-preview1 Apr 21, 2017

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg May 9, 2017

Member

For the KeyVault extensibility we're waiting on some KeyVault SDK changes. Removing this from 2.0.0-preview2 for now to bring it back to triage.

Member

muratg commented May 9, 2017

For the KeyVault extensibility we're waiting on some KeyVault SDK changes. Removing this from 2.0.0-preview2 for now to bring it back to triage.

@muratg muratg removed this from the 2.0.0-preview2 milestone May 9, 2017

@muratg muratg added 1 - Ready and removed 0 - Backlog labels Aug 29, 2017

@muratg muratg added this to the 2.1.0 milestone Aug 29, 2017

@muratg

This comment has been minimized.

Show comment
Hide comment
@muratg

muratg Aug 29, 2017

Member

We should have requirements in by now putting this in 2.1 milestone and assigning it to @pakrym.

cc @glennc @DamianEdwards @blowdart

Member

muratg commented Aug 29, 2017

We should have requirements in by now putting this in 2.1 milestone and assigning it to @pakrym.

cc @glennc @DamianEdwards @blowdart

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment