Unable to assume web identity inside EC2 when no region is set in the environment.

[ereOn]

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
All requests fail when executed on an EC2 (EKS) instance trying to use AssumeWebIdentity and AWS_REGION wasn't set.

Version of AWS SDK for Go?
v0.20.0

Version of Go (go version)?
go1.14

To Reproduce (observed behavior)
The following code snippet exhibits the problem:

package main

import (
	"github.com/aws/aws-sdk-go-v2/aws/external"
	"github.com/aws/aws-sdk-go-v2/service/sts"
)

func () main() {
    awsCfg, err := external.LoadDefaultAWSConfig()

    if err != nil {
        panic(fmt.Errorf("unable to load AWS SDK config: %s", err))
    }

    if awsCfg.Region == "" {
        awsCfg.Region = "us-east-1"
    }

    stsClient := sts.New(awsCfg)
    _, err := stsClient.GetCallerIdentityRequest(&sts.GetCallerIdentityInput{}).Send(ctx)

    // This call fails even though the region was set manually. If set before hand via `AWS_REGION`, it works as expected.
    if err != nil {
        panic(fmt.Errorf("failed to fetch AWS STS caller identity: %s", err))
    }
}

Expected behavior
The code should work but doesn't when running on an EC2 instance assuming a web identity.

Additional context
I debugged the situation a bit, and it seems the internal AssumeRole call is performed using a default AWS config. Since the default config on EC2 doesn't fetch the EC2 metadata region (yet? #244), the call fails with UnknownEndpointError, even though a region was set on the configuration. From my understanding, the underlying provider in charge of performing the assume-role call doesn't get the manually set region, and uses an empty one instead.

I could find no way of setting the region on the provider itself.

If I set AWS_REGION from inside the process and reload the default configuration once more, the code starts working again. However, this seems overcomplicated and I assume it is not intentional.

[RedbackThomson]

It appears to me that the ResolveCredentials resolver, which is called in LoadDefaultAWSConfig in the ResolveAWSConfig function already uses the region that is required to have been set previously, or is not being copied across.

This ResolveCredentials provider redirects through the ResolveCredentialChain function to call NewWebIdentityRoleProvider. Here the provider client is initialised, passing a COPY of the configuration aws.Config which has no default region assumed. This is the client that sts uses later to authenticate all requests. Therefore, the override does not make a difference to the authentication provider.

One such method I am attempting is to override the configuration region before calling external.LoadDefaultAWSConfig, so that in the case where no region override is found in any files or environment variables, it was provided previously. However, I have not been able to get this to work.

[meshuga]

I think the answer #492 (comment) might be related with this issue: AWS has multiple partitions, so global endpoint must always be referred in a context of a partition.

[RedbackThomson]

I think the answer #492 (comment) might be related with this issue: AWS has multiple partitions, so global endpoint must always be referred in a context of a partition.

I'm not sure if this is entirely correct. Because sts does not care about regionality, the provider is getting confused when we are passing it the aws partition. However, the regions it lists that the endpoint provides are all regionalised. -

WebIdentityErr: failed to retrieve credentials
caused by: unknown endpoint, could not resolve endpoint, partition: \"aws\", service: \"sts\", region: \"\", known: [eu-north-1 eu-west-2 me-south-1 us-east-1-fips us-west-1-fips us-west-2 ap-south-1 ap-southeast-1 eu-west-1 sa-east-1 us-west-1 us-west-2-fips ap-northeast-2 ap-northeast-1 aws-global ca-central-1 us-east-1 us-east-2 us-east-2-fips ap-east-1 eu-central-1 eu-west-3 ap-southeast-2]

[RedbackThomson]

All new documentation for STS recommends using a regionalised endpoint, and therefore I think there may be an issue within this implementation. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html

[jasdel]

Thanks for reporting this issue. This looks to be a bug in the SDK's implementation of the updated STS endpoint resolving and web identity support.

The STS client used by the web identity token credential provider will use the configured region, if one is set. Though the STS client endpoint lookup no longer allows no region to be specified. This was a bug that was fixed in v0.18.0, and a region must be specified for all clients.

This was missed for STS credential providers though. Those credential provider probably should default to aws-global as the region. But we should investigate if this default behavior would be correct.

[RedbackThomson]

Are you able to suggest a temporary workaround for this issue? I would be able to provide the region at time of configuration. Is there either some way I can pass this through to the resolver, or to force it to use the global region?

[jasdel]

Specifying the region either via environment or shared config is the best workaround for this issue at the moment. When investigating this we're also going to see if it makes sense to allow a "default" fallback region to be passed into the LoadAWSDefaultConfig, if one cannot be determined from the environment or shared config.

you can also specify a region when calling LoadAWSDefaultConfig directly

// Sets the region the configuration will be loaded with. Ignoring any environment,
// or shared config specified region.
cfg, err := external.LoadAWSDefaultConfig(external.WithRegion("us-west-2"))

If the SDK were to add support for a "default" region to fallback to, that would probably look like the following. (Note this is not implemented currently).

// Sets the region to be used if the region cannot be determined via the environment,
// shared config, or some other means.
cfg, err := external.LoadAWSDefaultConfig(external.WithDefaultRegion("us-west-2"))