Issue unprotecting keys

[ADringer]

Hi all,

I have an issue with keys in one environment, but can't tell what is different about it. As far as I can tell it is set up exactly the same as the others.

I get the following errors occassionaly:

Duende.IdentityServer.Services.KeyManagement.KeyManager Error unprotecting the IdentityServer signing key with kid 6FFCF1327C1CD6C02CA3659C413CCFFB.

Duende.IdentityServer.Services.KeyManagement.KeyManager Error unprotecting the IdentityServer signing key with kid 98EC3334B7548CCC52E91B241C4AEE4F

But those keys are in the database:

When checking what keys are exposed it returns only the latest:

The services are load balanced but as you can see are persisted to a shared database. I'm not sure what else I should be looking at as I can see the keys it's complaining about - any ideas what I should look at next?

It has started today, and last time was the 4th June. I can see a task (we are hosting in AWS Fargate) was started 12 hours ago, but can't see any expiration dates around keys that could cause issues.

Thanks

[ProgrammerAL]

Hi @ADringer, a couple of questions to help diagnose.

• To confirm, are you saying there are multiple environments using this shared database? And if so, how many environments are using the shared database?
• Can you share the code you have that sets up the ASP.NET Data Protection?

With your shared environment, I’m wondering if keys in one environment are being overwritten by an IdentityServer hosted in another environment. When you enable data protection, are you using the same Application Name across all environments? https://learn.microsoft.com/en-us/aspnet/core/security/data-protection/configuration/overview?view=aspnetcore-10.0#set-the-application-name-setapplicationname

[ProgrammerAL]

Thanks for the clarification. The error states .NET is unable to unprotect the signing key, it seems able to access the key. This happens when Data Protection is not configured correctly https://docs.duendesoftware.com/general/data-protection/#common-problems.

Try setting the application name. This ensures each deployed instance in your load balanced environment is using the same key name. builder.Services.AddDataProtection().PersistKeysToDbContext<UserManagementDbContext>().SetApplicationName("IdentityServer");

[wcabus]

Adding an additional note here @ADringer: setting an explicit application name now will invalidate all existing data protection keys, unless you use the same value that's currently being used.

If you want to reduce the risk, you can first try and find the application name for your production environment by adding the code snippet from our upgrade guide listed under "Getting the current application name".

Then, using the value, explicitly configure the application name so that every instance uses the same value, and that the value doesn't change in the future. Problem is, as @ProgrammerAL hinted to, that every instance could have an ever-so-slightly-different application name set right now, as it's derived from where the application lives if it's not configured manually.

[ADringer]

@ProgrammerAL @wcabus thanks both for your help with this.

I've added logging of the default discriminator to see if there could be any differences as dev is hosted in a mix of fargate and fargate spot (the only difference between dev & production) but both logged /app/.

My thinking now is that as it's development environment, developers do occassionally run Identity Server locally but pointing to the dev database, thinking this could be the cause as the app name would be the local folder path.

Questions:

1. What's the trigger for creating a new data protection key? Just checking if it's likely that a developer instance would create a new one.
2. Is it possible to see what the app name is for existing keys? I couldn't see it in the xml in the table.
3. If this is the case, I guess it's either a) use a fixed app name (but will require migration) or b) disable automatic keys for local work?

Thanks

[wcabus]

What's the trigger for creating a new data protection key? Just checking if it's likely that a developer instance would create a new one.

New Data Protection keys get created for different reasons. The most common reasons are:

• There are no Data Protection keys present at all, so a first key needs to be generated. Note that this can also occur if your storage is unavailable.
• The current Data Protection key is nearing its lifetime, so a new one will be created before it becomes the new active key.
• Or, the existing Data Protection key(s) couldn't be read or decrypted, causing no active key to be present.

That last scenario can happen when the Data Protection keys are also encrypted using a certificate or a service like DPAPI on Windows. I don't think this is at play in your case though, since you didn't explicitly use a ProtectKeysWith... method when configuring Data Protection.

In your scenario, I don't think that a new Data Protection key was generated however. IdentityServer did issue new signing keys, because the local development environment uses a different application discriminator value.

Is it possible to see what the app name is for existing keys? I couldn't see it in the xml in the table.

Unfortunately, this is not possible. The app name or app discriminator value is only used at runtime to derive another key from the main Data Protection key. This allows multiple applications on the same server to use the same Data Protection keys, while ensuring that protected data cannot be read by other applications. In practice, key derivation goes even a step further, using a custom purpose string to derive yet another key from the application-level derived key, for example when ASP.NET Core Identity generates a one-time password for multi-factor authentication.

If this is the case, I guess it's either a) use a fixed app name (but will require migration) or b) disable automatic keys for local work?

You can/should definitely configure the application discriminator, and you can set it to /app/ to match the current situation. Migrating to another discriminator is very difficult/hard, so you can also choose to set a custom discriminator value and reset all the protected data for the entire environment.

The issue you've encountered, most likely, was that the local development work used another (default) discriminator value. When IdentityServer tried to read the signing keys, it couldn't unprotect them because the discriminator /app/ didn't match the locally used one, or vice-versa. Setting a shared discriminator will fix that problem.

Another solution is to use a local database for local work, but perhaps that makes it more difficult to share specific configuration or settings.

[ADringer]

@wcabus thanks for the advice, have gone for setting the application name to /app/ to keep it consistent without having to do any migrations in real environments