Internal Server Error when trying to access Multi Region Access Point with Assuming Role.

[kpok]

Describe the bug

I am trying to implement lambda which will assume role and then will send a file to S3 bucket through Multi-region access point.

Lambda throws Internal Server Error exception -> See lambda details in reproduction Steps.

Expected Behavior

File should appear in S3 bucket under the Multi-Region Access Point.

Current Behavior

Below exception is thrown from line "var result = s3Client.PutObjectAsync(putObjectRequest).Result;":

System.AggregateException: One or more errors occurred. (We encountered an internal error. Please try again.) ---> Amazon.S3.AmazonS3Exception: We encountered an internal error. Please try again. ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown. at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler1.HandleAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at System.Threading.Tasks.Task1.get_Result() at IntegrationTests.TestFunction.FunctionHandler()

Reproduction Steps

1. Create Lambda function with .net 6.0 runtime:

And with below code:

public void FunctionHandler()
{
      var s3Endpoint = Environment.GetEnvironmentVariable("S3Endpoint");
      Console.WriteLine($"S3Endpoint: {s3Endpoint}");

      var assumeRoleReq = new AssumeRoleRequest()
      {
          RoleSessionName = "Session1",
          RoleArn = "<<RoleArnCreatedInPoint2>>"
      };

      var client = new Amazon.SecurityToken.AmazonSecurityTokenServiceClient();
      var assumeRoleRes = client.AssumeRoleAsync(assumeRoleReq).Result;
      var s3Client = new AmazonS3Client(assumeRoleRes.Credentials);
      var stream = GenerateStreamFromString("test1");
      var putObjectRequest = new PutObjectRequest()
      {
          BucketName = s3Endpoint,
          Key = "testFromLambda.txt",
          InputStream = stream
      };
      var result  = s3Client.PutObjectAsync(putObjectRequest).Result;
      Console.WriteLine(result);
      Console.WriteLine(result.HttpStatusCode);
}
public static Stream GenerateStreamFromString(string s)
{
    var stream = new MemoryStream();
    var writer = new StreamWriter(stream);
    writer.Write(s);
    writer.Flush();
    stream.Position = 0;
    return stream;
}

2. Create role with S3 full permissions and with trust relationship to allow assume this role by lambda role.
3. Create multi-region access point with below access point policy:

{
	"Statement": [
		{
			"Action": [
				"s3:GetObject",
				"s3:PutObject"
			],
			"Resource": [
				"<<Multi-Region-Access-Point-Arn>>/object/*"
			],
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"<<RoleArnCreatedInPoint2>>"
				]
			}
		}
	]
}

5. Set S3Endpoint environment variable for lambda to Arn of created multi-region access point.
6. Test Lambda.

Possible Solution

No response

Additional Information/Context

The same situation occurs when I try to execute the Lambda Function Code from Console Application on Windows 10 machine while I have aws credentials configured which allows to assume role created in point 2 of reproduction steps.

Also worth to mention that below powershell + AWS CLI code works correctly. This code is doing a similar job as lambda while in this situation the operation ends successfully:

$assumedRole = aws sts assume-role --role-arn $role --role-session-name $roleSessionName --output json --profile $profileName | Out-String | ConvertFrom-Json
$Env:AWS_ACCESS_KEY_ID=$($assumedRole.Credentials.AccessKeyId)
$Env:AWS_SECRET_ACCESS_KEY=$($assumedRole.Credentials.SecretAccessKey)
$Env:AWS_SESSION_TOKEN=$($assumedRole.Credentials.SessionToken)
aws s3api  put-object --body $fileToSend --bucket $multiRegionAccessPointArn --key $fileName

Version of AWS CLI:
aws-cli/2.12.7 Python/3.11.4 Windows/10 exe/AMD64 prompt/off

AWS .NET SDK and/or Package version used

Amazon.Lambda.Core 2.1.0
AWSSDK.Extensions.CrtIntegration 3.7.1.8
AWSSDK.S3 3.7.200
AWSSDK.SecurityToken 3.7.200

Targeted .NET Platform

.NET 6

Operating System and version

Lambda

[ashishdhingra]

Hi @kpok,

Good afternoon.

I'm unsure how you have configured Lambda and roles, here are the steps that I used to reproduce the issue:

• Created role testmraplambda with the AmazonS3FullAccess and CloudWatchFullAccess permissions with the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

• Created MRAP with the following permissions policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<<account-id>>:role/testmraplambda"
                ]
            },
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3::<<account-id>>:accesspoint/<<mrap-alias.mrap>>/object/*"
            ]
        }
    ]
}

• Created simple Lambda function targeting .NET 6 with the following code:

using Amazon;
using Amazon.Lambda.Core;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

namespace TestMRAPLambda;

public class Function
{
    public void FunctionHandler(ILambdaContext context)
    {
        AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
        AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
        AWSConfigs.LoggingConfig.LogMetrics = true;

        var s3Endpoint = "arn:aws:s3::<<account-id>>:accesspoint/<<mrapalias.mrap>>";
        Console.WriteLine($"S3Endpoint: {s3Endpoint}");

        var s3Client = new AmazonS3Client();
        var stream = GenerateStreamFromString("test1");
        var putObjectRequest = new PutObjectRequest()
        {
            BucketName = s3Endpoint,
            Key = "testFromLambda.txt",
            InputStream = stream
        };
        var result = s3Client.PutObjectAsync(putObjectRequest).Result;
        Console.WriteLine(result);
        Console.WriteLine(result.HttpStatusCode);
    }

    public static Stream GenerateStreamFromString(string s)
    {
        var stream = new MemoryStream();
        var writer = new StreamWriter(stream);
        writer.Write(s);
        writer.Flush();
        stream.Position = 0;
        return stream;
    }
}

Also ensured to add reference to AWSSDK.Extensions.CrtIntegration NuGet package. Below is my .csproj file:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
    <!-- This property makes the build directory similar to a publish directory and helps the AWS .NET Lambda Mock Test Tool find project dependencies. -->
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <!-- Generate ready to run images during publishing to improve cold start time. -->
    <PublishReadyToRun>true</PublishReadyToRun>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
    <PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />
    <PackageReference Include="AWSSDK.Extensions.CrtIntegration" Version="3.7.1.8" />
    <PackageReference Include="AWSSDK.S3" Version="3.7.200.1" />
    <PackageReference Include="AWSSDK.SecurityToken" Version="3.7.200.1" />
  </ItemGroup>
</Project>

Reference to AWSSDK.Extensions.CrtIntegration is required for SigV4a signing for MRAP integration.

• While publishing Lambda project, attached role testmraplambda created previously.
• Tested the Lambda from AWS console, it works fine.

I'm unsure why you need to assume a role when you would attach role to Lambda during deployment. Please let me know if I'm missing something from the reproduction steps. Are you using a different role attached to Lambda function?

NOTE: I have hard-coded ARNs for reproduction. This should be avoided in production. Also, we should dispose of Amazon S3 client after use (mat be wrap in using(){} block).

Thanks,
Ashish

[kpok]

Hi @ashishdhingra
The main use case which I try to achieve is to use lambda which is in one AWS account while MRAP and S3 buckets are in second AWS account. Additionally the lambda in first account may need additional permissions for using resources (other then s3) in second AWS account. For that purpose we have been able to use Assume role between accounts. So Lambda in account A assumes role in account B and that assumed role in account B has required permission to send data to MRAP and S3 buckets. This worked fine when we use S3 buckets as S3 endpoint while it fails with the error as above when we try to use Multi-Region access point ARN as S3 Endpoint.

For the bug reproduction I am trying to assume role from another lambda role on the same AWS account but the same error occurs also when I try to do with different accounts.

Regards,
Karol

[ashovlin]

One common "sharp edge" with MRAPs and assumed roles is mentioned on https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointRestrictions.html

To use SigV4A with temporary security credentials—for example, when using AWS Identity and Access Management (IAM) roles—make sure that you request the temporary credentials from a Regional endpoint in AWS Security Token Service (AWS STS), instead of a global endpoint. If you use the global endpoint for AWS STS (sts.amazonaws.com), AWS STS will generate temporary credentials from a global endpoint, which isn't supported by Sig4A. As a result, you'll get an error. To resolve this issue, use any of the listed Regional endpoints for AWS STS.

V3 of the .NET SDK still defaults to the global STS endpoint. Can you check which endpoint the STS request is going to? And if it's still the global endpoint, configure it to go to a regional endpoint instead?

• https://docs.aws.amazon.com/sdkref/latest/guide/feature-sts-regionalized-endpoints.html - config file or environment variable settings
• SecurityTokenServiceConfig exposes a StsRegionalEndpoints setting in code

[kpok]

Hi @ashovlin
Your comment helped. Thank you very much.

I replaced below line:
var client = new Amazon.SecurityToken.AmazonSecurityTokenServiceClient();

with lines:

var config = new AmazonSecurityTokenServiceConfig() { StsRegionalEndpoints = StsRegionalEndpointsValue.Regional };
var client = new AmazonSecurityTokenServiceClient(config);

and the solution started to work.

Thank you.

[ashovlin]

Glad it worked! Yeah, the defaulting to legacy for STS is an easy way to get tripped up with operations that require SigV4a, but we're waiting until the next version to change the default behavior.