ASP.NET core 2.2: tips for configuring MongoDB as a custom key ring storage

[EnricoMassone]

Hi,

we are developing an ASP.NET core 2.2 web application using the cookie authentication without ASP.NET identity. We followed this docs in order to set it up.

We are facing some troubles with the ASP.NET core data protection system when hosting our application inside a web farm. Our requirements are a bit complex because we must guarantee all of the followings hosting scenarios:

• Azure app service on a single data center
• Azure app services distributed over multiple data centers (active-active scenario)
• deployment inside a kubernetes cluster

In the first release of our application we weren't aware of the interaction between data protection system and cookies authentication, we tested extensively on Azure app services and everything worked well. This is due to the default behavior of the data protection system under Azure app services I guess.

Problems arised when we started some testing on a kubernetes cluster and we scaled out the application pod. In this scenario we cannot rely on the ASP.NET core's defaults and we need a central shared key ring storage. The same holds true for the Azure multi data center scenario I guess.

Our plan is using MongoDB as shared key ring storage, we would like some tips over the following subjects:

• some existing implementations (such as this one) has a mechanism to remove revoked and expired keys each time a new key is stored (I think its done to save space on disk). Is there any issue in doing so ? I guess that revoked or expired keys can safely be removed because they are not useful any longer, but I can't find any official guideline on the Microsoft docs.
• we plan to avoid any kind of key encryption at rest, at least initially. I guess that, given our requirements, the only feasible solution for us is using a shared X.509 certificate distributed to all
  the machines involved. Do you suggest to explore alternatives in case we decide to adopt key encryption at rest ?
• in the final part of this blog post the author warns that when the application is scaled out to multiple instances, by default all of the nodes act as key generator. He also suggests to investigate the possibility of disabling the key generation for the "secondary nodes" and leave it enabled only for the "primary node". Is this really necessary ? Is it possible to safely stick to the defaults and leave all of the nodes as key generators ? Our application is designed so that all of the nodes are considered equals, we don't really have a built in concept of "primary node" and "secondary nodes", so following the author suggestion could be problematic for us (we would need to introduce an additional configuration to identify the primary node and we would like to avoid that).

Thanks for helping us.

Kind regards

[blowdart]

Removal of keys safety depends on how you are using data protection. If you're using it on your own data, for things you want to unprotect later, then no, don't remove keys. If it's just the default asp.net uses then you can remove ones that expired a month ago safely.

If you don't want to use Azure then x509certificates are the only way to protect at rest. Do not skip this step.

Disabling multiple nodes isn't necessary. A problem arises if you fire up a bunch at exactly the same time, so when you start, bring up one node only, send it a request or two until a key is created and then you're fine. We've patched this in 3.0

[EnricoMassone]

Removal of keys safety depends on how you are using data protection. If you're using it on your own data, for things you want to unprotect later, then no, don't remove keys. If it's just the default asp.net uses then you can remove ones that expired a month ago safely.

If you don't want to use Azure then x509certificates are the only way to protect at rest. Do not skip this step.

Disabling multiple nodes isn't necessary. A problem arises if you fire up a bunch at exactly the same time, so when you start, bring up one node only, send it a request or two until a key is created and then you're fine. We've patched this in 3.0

Hi and thanks for the suggestions.

We use the keys only for the ASP.NET core authentication cookies, we don't do anything custom with them. I'm investigating the key removal in order to avoid an infinite growth of the mongo db collection used to store the keys.

Why you said that only the keys expired one month ago can be removed ? I mean, why exactly one month ago ? My understanding is that being expired one minute ago is functionally equivalent to being expired one month ago, but maybe I'm missing a piece here.

For the last point (multiple nodes which concurrently generate keys) you are basically suggesting of coordinating the application startup so that one node is started before all of the others. Doing so the key is basically generated by one node and the concurrency problem is avoid from the very beginning, because all the other nodes will use the key generated by the first one.

Is it possible to avoid this startup procedure by disabling the key generation for all the nodes but one or in any other way that involves only the application code ?

Thanks for all the suggestions

[blowdart]

The keys are really small, and a new one appears every three months by default. I honestly wouldn't worry about it.

New keys get created a few days before its due, and then syncing happens on a lazy basis, hence
wanting to keep an older one around until you're sure everything is synced.

You could disable key generation in all but one node, but what happens if that node goes down? Or you update to 3.0 where we fixed this.

[EnricoMassone]

Thanks for the explanations, now it's clearer and makes sense.

I agree with you, coordinating the startup is way simpler than rotating the primary node when it goes down. Simplistic solutions are not possible here.

[blowdart]

You'll only need to do it once for creating the first key, because your keyring is persistent. After that it'll just work.

[EnricoMassone]

You'll only need to do it once for creating the first key, because your keyring is persistent. After that it'll just work.

Could you please linking here the issue (if one exists) reporting this problem of concurrent key generation so that we can check the details ? Thanks.