Unable to deploy to non-default region when using AssumeRole credentials

[scottjbaldwin]

Describe the bug

When trying to deploy a serverless application to a non-default region (i.e. ap-southeast-4) using a profile that uses a session token from sts, the cli is unable to upload the packaged lambda code to the S3 bucket.

Expected Behavior

The lambda code in the zip file should upload and the serverless template should be deployed.
This works perfectly fine if the region is a default region like ap-southeast-2.

Current Behavior

The output of dotnet lambda deploy-serveress first hints an an issue by saying

Warning: Unable to determine region for bucket sbaldwin-dev-mel, assuming bucket is in correct region: The provided token is malformed or otherwise invalid.

and then after compiling and zipping the code, when it attempts to upload the zip file, it errors out with

Uploading to S3. (Bucket: Key: )
Error uploading to in bucket : The provided token is malformed or otherwise invalid.

And exits the deployment.

Reproduction Steps

• Ensure you have set up an Assume Role profile as described in https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html
• Ensure you have a bucket in the region for the deployment code
• Create a new serverless project using dotnet new serverless.AspNetCoreWebAPI --name mel-test --output .\mel-test --region ap-southeast-4 (or equivalent non-default region)
• Attmept to deploy this from the source folder using `dotnet lambda deploy-serverless --profile --region ap-southeast-4 --s3-bucket

Possible Solution

Looking through the source code for the aws extensions for dotnet, I couldn't see anything obvious. I did see that you use the C# sdk, so the issue may be with it's handling of profiles, but because I found the issue in this lbrary, I decided to raise the bug here.

I did check the results of aws s3api get-bucket-location --bucket <mel-bucket> --profile <assumerole-profile> and recieved the expected results:

{
    "LocationConstraint": "ap-southeast-4"
}

Which says to me that the profile and the bucket are set up fine, and given this is the same call that GetBucketRegionAsycn makes, I can only assume that it's some difference between the way the cli handles the profile vs how the C# sdk handles the profile.

Additional Information/Context

I am using MFA for my roles, and I'm using this powershell module AWSCredentialsManager to assist with the AssumeRole and MFA details.
There was an issue with this library using non-default regions initially, but this issue was addressed in version 0.1.6 which is the version of the library I am currently using.

Targeted .NET platform

7.0.201

CLI extension version

Package Id                                     Version      Commands
---------------------------------------------------------------------------------------
amazon.lambda.testtool-3.1                     0.12.7       dotnet-lambda-test-tool-3.1
amazon.lambda.tools                            5.6.3        dotnet-lambda
aws.codeartifact.nuget.credentialprovider      1.0.1        dotnet-codeartifact-creds
microsoft.dotnet-httprepl                      6.0.0        httprepl

Environment details (OS name and version, etc.)

Windows 10

[scottjbaldwin]

Additional information:
This works fine in ap-southeast-4 if using a standard user profile (i.e. long-term credentials, access-key/secret-access-key) for an IAM user, just not when you use the AssumeRole approach.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[scottjbaldwin]

Whoops, accidentally closed the issue. Sorry...

[ashishdhingra]

Hi @scottjbaldwin,

Good afternoon.

Thanks for reporting the issue. Somehow I'm unable to reproduce the issue. Below are the steps I followed:

• Created a new IAM role named testassumerole in my account with the below trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<<account-id>>:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

You may replace arn:aws:iam::<<account-id>>:root above with the IAM user which you want to use for assume role.

• For reproduction purposes, this role also has the AmazonS3FullAccess, AWSCloudFormationFullAccess and below custom policy attached (these permissions are necessary for deploying serverless application using .NET CLI extensions):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:CreateFunction",
                "iam:GetRole",
                "lambda:TagResource",
                "lambda:ListFunctions",
                "apigateway:PUT",
                "lambda:GetFunction",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:AttachRolePolicy",
                "apigateway:DELETE",
                "iam:PassRole",
                "lambda:AddPermission",
                "iam:DetachRolePolicy",
                "apigateway:PATCH",
                "lambda:DeleteFunction",
                "apigateway:POST",
                "apigateway:GET"
            ],
            "Resource": "*"
        }
    ]
}

• In my %USERPROFILE%/.aws/config, configured the new profile testassumerole as below:

[profile testassumerole]
role_arn=arn:aws:iam::<<account-id>>:role/testassumerole
source_profile=default

In this case, my default profile has credentials that gives permissions to assume the role.

• Created an S3 bucket in ap-southeast-2 region (non-default).
• Using Visual Studio 2022, created a new serverless web API project. Modified region in aws-lambda-tools-defaults.json to ap-southeast-2.
• Upgraded to latest version of AWS Lambda Tools for .NET CLI using command dotnet tool update -g Amazon.Lambda.Tools.
• Opened command prompt and navigated to project directory. Executed command dotnet lambda deploy-serverless --profile testassumerole --region ap-southeast-2 --s3-bucket testassumerole-bucketname. It deployed the application successfully:

Amazon Lambda Tools for .NET Core applications (5.6.3)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Enter CloudFormation Stack Name: (CloudFormation stack name for an AWS Serverless application)
teststack
Processing CloudFormation resource AspNetCoreFunction
Initiate packaging of . for resource AspNetCoreFunction
Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder 'D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish'
... dotnet publish "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\." --output "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained False
... publish: MSBuild version 17.5.0-preview-23061-01+040e2a90e for .NET
... publish:   Determining projects to restore...
... publish:   All projects are up-to-date for restore.
... publish:   TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\linux-x64\TestServerlessAssumeRole.dll
... publish:   TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\publish\
Zipping publish folder D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish to C:\Users\ashdhin\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638143917090263972.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll
... zipping: Amazon.Lambda.AspNetCoreServer.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Logging.AspNetCore.dll
... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll
... zipping: appsettings.Development.json
... zipping: appsettings.json
... zipping: aws-lambda-tools-defaults.json
... zipping: TestServerlessAssumeRole
... zipping: TestServerlessAssumeRole.deps.json
... zipping: TestServerlessAssumeRole.dll
... zipping: TestServerlessAssumeRole.pdb
... zipping: TestServerlessAssumeRole.runtimeconfig.json
Created publish archive (C:\Users\<<REDACTED>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638143917090263972.zip).
Lambda project successfully packaged: C:\Users\<<REDACTED>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638143917090263972.zip
Uploading to S3. (Bucket: testassumerole-bucketname Key: TestServerlessAssumeRole/AspNetCoreFunction-CodeUri-Or-ImageUri-638143917090263972-638143917125392654.zip)
... Progress: 51%
... Progress: 100%
Uploading to S3. (Bucket: testassumerole-bucketname Key: TestServerlessAssumeRole/teststack-serverless-638143917134155008.template)
... Progress: 100%
Found existing stack: False
CloudFormation change set created
... Waiting for change set to be reviewed
Created CloudFormation stack teststack

Timestamp            Logical Resource Id                      Status
-------------------- ---------------------------------------- ----------------------------------------
3/14/2023 11:55 AM   teststack                                CREATE_IN_PROGRESS
3/14/2023 11:55 AM   AspNetCoreFunctionRole                   CREATE_IN_PROGRESS
3/14/2023 11:55 AM   AspNetCoreFunctionRole                   CREATE_IN_PROGRESS
3/14/2023 11:55 AM   AspNetCoreFunctionRole                   CREATE_IN_PROGRESS
3/14/2023 11:56 AM   AspNetCoreFunctionRole                   CREATE_COMPLETE
3/14/2023 11:56 AM   AspNetCoreFunction                       CREATE_IN_PROGRESS
3/14/2023 11:56 AM   AspNetCoreFunction                       CREATE_IN_PROGRESS
3/14/2023 11:56 AM   AspNetCoreFunction                       CREATE_COMPLETE
3/14/2023 11:56 AM   ServerlessRestApi                        CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApi                        CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApi                        CREATE_COMPLETE
3/14/2023 11:56 AM   AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApiDeploymentcfb7a37fc3    CREATE_IN_PROGRESS
3/14/2023 11:56 AM   AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS
3/14/2023 11:56 AM   AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS
3/14/2023 11:56 AM   AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApiDeploymentcfb7a37fc3    CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApiDeploymentcfb7a37fc3    CREATE_COMPLETE
3/14/2023 11:56 AM   ServerlessRestApiProdStage               CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApiProdStage               CREATE_IN_PROGRESS
3/14/2023 11:56 AM   ServerlessRestApiProdStage               CREATE_COMPLETE
3/14/2023 11:56 AM   AspNetCoreFunctionRootResourcePermissionProd CREATE_COMPLETE
3/14/2023 11:56 AM   AspNetCoreFunctionProxyResourcePermissionProd CREATE_COMPLETE
3/14/2023 11:56 AM   teststack                                CREATE_COMPLETE
Stack finished updating with status: CREATE_COMPLETE

Output Name                    Value
------------------------------ --------------------------------------------------
ApiURL                         https://<<some-id>>.execute-api.ap-southeast-2.amazonaws.com/Prod/

Please review if you are following different steps or if I'm missing anything.

Thanks,
Ashish

[scottjbaldwin]

@ashishdhingra, ap-southeast-2 is the Sydney region which is a default AWS region. As I said, it works fine in that region, but not in ap-southeast-4 (Melbourne) which is a non-default region.

[ashishdhingra]

@ashishdhingra, ap-southeast-2 is the Sydney region which is a default AWS region. As I said, it works fine in that region, but not in ap-southeast-4 (Melbourne) which is a non-default region.

@scottjbaldwin Could you please elaborate what are you referring to by default region? Is it the region in the credentials/config file? For me, the default region is us-east-2.

[scottjbaldwin]

Note, you will have to enable ap-southeast-4 for your AWS account if you haven't already, but that's how non-default regions work.

[scottjbaldwin]

@ashishdhingra by non-default region, I mean a region that you don't get access to by default in an AWS account, i.e. an opt-in region as per this table https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions:~:text=for%20a%20resource-,Available%20Regions,-Your%20account%20determines

[ashishdhingra]

ing to S3. (Bucket: Key: )
Error uploading to in bucket : The provided token is malformed or otherwise invalid.

@scottjbaldwin Thanks for the inputs, it really helped. It appears to be reproducible for non-default (required to opt-in region). Executing dotnet lambda deploy-serverless --profile testassumerole --region ap-southeast-4 --s3-bucket testassumerole-s3-non-default-region gives error below:

Amazon Lambda Tools for .NET Core applications (5.6.3)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Enter CloudFormation Stack Name: (CloudFormation stack name for an AWS Serverless application)
teststack
Warning: Unable to determine region for bucket testassumerole-s3-non-default-region, assuming bucket is in correct region: The provided token is malformed or otherwise invalid.
Processing CloudFormation resource AspNetCoreFunction
Initiate packaging of . for resource AspNetCoreFunction
Executing publish command
Deleted previous publish folder
... invoking 'dotnet publish', working folder 'D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish'
... dotnet publish "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\." --output "D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained False
... publish: MSBuild version 17.5.0-preview-23061-01+040e2a90e for .NET
... publish:   Determining projects to restore...
... publish:   Restored D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\TestServerlessAssumeRole.csproj (in 389 ms).
... publish:   TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\linux-x64\TestServerlessAssumeRole.dll
... publish:   TestServerlessAssumeRole -> D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\bin\Release\net6.0\publish\
Zipping publish folder D:\source\repros\TestServerlessAssumeRole\TestServerlessAssumeRole\.\bin\Release\net6.0\publish to C:\Users\<<redacted>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474.zip
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll
... zipping: Amazon.Lambda.AspNetCoreServer.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Logging.AspNetCore.dll
... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll
... zipping: appsettings.Development.json
... zipping: appsettings.json
... zipping: aws-lambda-tools-defaults.json
... zipping: TestServerlessAssumeRole
... zipping: TestServerlessAssumeRole.deps.json
... zipping: TestServerlessAssumeRole.dll
... zipping: TestServerlessAssumeRole.pdb
... zipping: TestServerlessAssumeRole.runtimeconfig.json
Created publish archive (C:\Users\<<redacted>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474.zip).
Lambda project successfully packaged: C:\Users\<<redacted>>\AppData\Local\Temp\AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474.zip
Uploading to S3. (Bucket: testassumerole-s3-non-default-region Key: TestServerlessAssumeRole/AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474-638144062032939311.zip)
Error uploading to TestServerlessAssumeRole/AspNetCoreFunction-CodeUri-Or-ImageUri-638144061995484474-638144062032939311.zip in bucket testassumerole-s3-non-default-region: The provided token is malformed or otherwise invalid.

Executing AWS CLI command aws s3api get-bucket-location --bucket testassumerole-s3-non-default-region- --profile testassumerole returns correct region:

{
    "LocationConstraint": "ap-southeast-4"
}

Also using the default profile directly with configured credentials works fine for ap-southeast-4 region.

So the issue appears to be with the underlying .NET SDK that fails to successfully call AmazonS3Client.GetBucketLocationAsync() for assume role flow and fails with The provided token is malformed or otherwise invalid. error. This needs investigation and further review.

[ashishdhingra]

@scottjbaldwin Upon further investigation, Managing AWS STS in an AWS Region mentions that Session tokens from the global STS endpoint are valid only in AWS Regions that are enabled by default. While per the mentioned article, you could change the setting in AWS account, the other workaround is to set environment variable AWS_STS_REGIONAL_ENDPOINTS with value regional in the current session. I tested this and it worked.

In PowerShell session:
$env:AWS_STS_REGIONAL_ENDPOINTS="regional"

In command line (Windows):
set AWS_STS_REGIONAL_ENDPOINTS=regional

Also refer AWS STS Regionalized endpoints where it states that AWS recommends using Regional AWS STS endpoints instead of the global endpoint. So, you may set it at config file level instead.

Please let me know if it works for you. After your confirmation, I would consider this issue as resolved (since it is the default behavior as per above documentation) and convert this issue into Q&A discussion so that it's easily accessible to other users experiencing similar issue.

Thanks,
Ashish

[scottjbaldwin]

Hey @ashishdhingra, thanks so much for diving into this issue and providing the resolution. I really appreciate your diligence on this one.

This does indeed fix the issue, and I am able to deploy my application (although there were still some remaining regional issues, but completely unrelated to this issue). I guess that means I'll need to update my blogpost, as technically it is not so much a bug, as a corner case that people need to be aware of.

Totally support this issue being converted into a Q&A as I am sure I will not be the last person to burn considerable hours on this issue, and I would love to discuss possible paths forward that are not quite as opaque.

Again, thanks for your help.