[EKS] [request]: Set IRSA pod identity webhook to use regional STS by default

[mmerickel]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

EKS clusters should default to AWS_STS_REGIONAL_ENDPOINTS=regional instead of legacy for pods using IRSA. It is possible to define this via --sts-regional-endpoint=true on the pod-identity-webhook daemon, however this deployment is not exposed for configuration.

Please set this value by default and/or provide a new EKS addon to support configuring this value.

This issue caused outages with pods using IRSA in us-east-2 during the downtime in us-east-1 which is an unacceptable default.

Which service(s) is this request for?

EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Isolate EKS deployments using IRSA to their regional endpoints instead of relying on the "global" (legacy) STS endpoint when running workloads an AWS-hosted resource tied to a specific region.

Are you currently working around this issue?

Manually defining AWS_STS_REGIONAL_ENDPOINTS=regional environment variable in each pod config.

Additional context

The AWS CLI v2 switched the default to "regional" already. This indicates they felt changing a default was important enough for the community to do it already.

The docs for the webhook state that "regional" is what you want almost always - so why is it left at legacy?

• AWS_STS_REGIONAL_ENDPOINTS default to regional amazon-eks-pod-identity-webhook#130
• change the default to use regional endpoints for sts boto/botocore#2577

[georgejohnis]

You can annotate a service account withsts-regional-endpoints=true to make use of the AWS Security Token Service's regional endpoint on the newer EKS platform versions. Please find the annotation instruction and platform version requirements in EKS doc here.

We are evaluating making the regional STS endpoint as the default in a future EKS version release.

[mmerickel]

@mikestef9 is there more information somewhere on what “Just Shipped” means for this issue?

[mikestef9]

Yes, the documentation link that @georgejohnis mentioned above has instructions

https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html

[mikestef9]

Actually, I'll re-open the issue, as the ask is to change default.

[mikestef9]

Update: We are going to change the default to regional STS endpoint for all clusters, without a need to opt in with annotations on service accounts. This will be available for new clusters soon, and will be rolling out for existing clusters over next several weeks.

[bryantbiggs]

Update: We are going to change the default to regional STS endpoint for all clusters, without a need to opt in with annotations on service accounts. This will be available for new clusters soon, and will be rolling out for existing clusters over next several weeks.

would the STS principal endpoint address for EKS be receiving a similar update or no? Ref: terraform-aws-modules/terraform-aws-eks#1904

[sidewinder12s]

@mikestef9 Did this finish rolling out/would the way to check if you've got the new behavior be to check the pod env vars for the AWS_STS_REGIONAL_ENDPOINTS env var?

[mikestef9]

It's still rolling out, almost done. It will be part of a new EKS platform version, and docs will be updated with the min platform version where this default change has been made.

[georgejohnis]

All newly launched pods on EKS clusters running below platform versions or later will use regional STS endpoint by default. EKS documentation has been updated to reflect this [link].
EKS v1.18 eks.10
EKS v1.19 eks.8
EKS v1.20 eks.4
EKS v1.21 eks.5

We have decided to roll this change out only starting from EKS version 1.22 clusters.

[mikestef9]

Closing as EKS 1.22 is now released