SPIRE deployments of more than a few servers can create more keys in JWKS than OIDC federating system supports

[keeganwitt]

Using the KMS KeyManager plugin, I currently see 8 keys in my testing (although this might have been caused by pods being replaced without the key_metadata_file being in a persisted volume). Also in my testing, AWS support a maximum of 100 keys in a JWKS endpoint, otherwise it is unable to retrieve the KID from the list to match against the one used in a JWT. Since I have 15 clusters with 2 server instances each, if 4 keys are created each, I'll get 120 keys created, which means I won't be able to use SPIRE for OIDC federation with AWS.

[amartinezfayo]

Using the KMS KeyManager plugin, I currently see 8 keys in my testing (although this might have been caused by pods being replaced without the key_metadata_file being in a persisted volume)

We have an open issue to remove the dependency on the key_metadata_file that KMS plugins rely on: #4375.

[keeganwitt]

Using the KMS KeyManager plugin, I currently see 8 keys in my testing (although this might have been caused by pods being replaced without the key_metadata_file being in a persisted volume)

We have an open issue to remove the dependency on the key_metadata_file that KMS plugins rely on: #4375.

I'll leave this ticket open for now so I can validate, but that will likely resolve my concern (assuming 2 keys per server instance).

[keeganwitt]

Confluent Cloud only supports 15 keys per identity provider. I'm discussing with them whether they can make an exception and what that new limit might be.

[amartinezfayo]

Also in my testing, AWS support a maximum of 100 keys in a JWKS endpoint, otherwise it is unable to retrieve the KID from the list to match against the one used in a JWT.

Is there any chance that there is a resource quota establishing the limit to 100 keys? It seems that the default limit is 100,000 keys: https://docs.aws.amazon.com/kms/latest/developerguide/resource-limits.html#kms-keys-limit.

I'll leave this ticket open for now so I can validate, but that will likely resolve my concern (assuming 2 keys per server instance).

Just to clarify: SPIRE publishes 4 keys (current X509 CA, next X509 CA, current JWT authority, next JWT authority), so there will be 4 keys per server instance. I remember that you mentioned during the last contributors sync that you are only interested in the keys exposed by the SPIRE OIDC Discovery Provider? I may be missing something. What I'm not quite sure is how that could limit the number of keys pushed to AWS KMS by SPIRE Server.

[keeganwitt]

The link you mentioned is about KMS keys. The key limit I'm talking about is when you create an OpenID Identity Provider in AWS and point it to SPIRE's OIDC component, the /keys endpoint can't contain more than 100 keys or else AWS will be unable to find the key to use in JWT validation. This is a limit that is not publicly disclosed and only found by my manual testing.

Yes by "2 keys" I meant what's exposed by the OIDC provider, not the total keys each instance adds to the bundle. I think this is something we confirmed already in the contributor meeting.

[amartinezfayo]

The key limit I'm talking about is when you create an OpenID Identity Provider in AWS and point it to SPIRE's OIDC component

Thanks for the clarification, it does make sense to me now.

So we are keeping this open until #4700 is done and you confirm that this is no longer a concern for you?

[keeganwitt]

So we are keeping this open until #4700 is done and you confirm that this is no longer a concern for you?

I think given what Confluent told me (still waiting to find out how much they can increase the limit), we should keep this open regardless and discuss what options we have when one has to federate with a system with a lower JWKS key limit than the SPIRE deployment requires. There may be other systems I or others have to integrate where this limit is hit and the vendor refuses to increase it.

[azdagron]

Unfortunately, there isn't an immediately clear way to reduce the number of JWT signing keys in use in SPIRE HA environments (two per server is a hard-coded behavior).

It would be good to explore solutions for using SPIRE with OIDC validators with these constraints.

[keeganwitt]

One idea is to add a SPIRE cluster/deployment identifier in the keys in the bundle so that each cluster can filter the bundle the only keys that are from that particular cluster/deployment and configure each cluster/deployment with a separate issuer to know which OIDC endpoint to hit. Would this be a viable option conceptually?