.NET 8: S3.PutObjectAsync request throws exception "Request headers must contain only ASCII characters."

[indimini]

Describe the bug

In migrating to .NET 8, we've run into an issue when calling IAmazonS3.PutObjectAsync(). The call works when running on Windows (debug/dev environment), but when deployed to Linux servers (ECS/Fargate), any attempts to put an object results in the exception "Request headers must contain only ASCII characters".

The PutRequest object passed to the call contains no user-added headers, so it would appear that the offending header entry is being added within the call. The call stack from the exception thrown is as follows:

at System.Net.Http.HttpConnection.g__ThrowForInvalidCharEncoding|56_0()
at System.Net.Http.HttpConnection.WriteString(String s, Encoding encoding)
at System.Net.Http.HttpConnection.WriteHeaderCollection(HttpHeaders headers, String cookiesFromContainer)
at System.Net.Http.HttpConnection.WriteHeaders(HttpRequestMessage request, HttpMethod normalizedMethod)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)

Installed version of AWSSDK.S3 is 3.7.410.1, but I have tried the earlier version that worked with .NET 6 and still had the exception thrown.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Call to PutObjectAsync() should succeed

Current Behavior

Exception thrown "Request headers must contain only ASCII characters."

Reproduction Steps

Deploy .NET 8 based code to ECS/Fargate Linux instance; call fails

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.410.1

Targeted .NET Platform

.NET 8

Operating System and version

ECS/Fargate Linux instance

[ashishdhingra]

@indimini Good morning. Please share the following to troubleshoot the issue:

• Sample reproduction code solution and deployment steps.
• Locale of your execution environment (we have similar issue All cmdlets fail with "Request headers must contain only ASCII characters." for some locales aws-tools-for-powershell#188 for PowerShell for some locale)
• Are you logging or throwing an exception with text containing non-ASCII characters? (refer similar issue Unhandled exceptions which contain non ASCII characters in the exception message cause a Runtime.ExitError aws-lambda-dotnet#1231 which was fixed for Lambda environment)
• Does the key or bucket name used while invoking PutObjectAsync call contain any non-ASCII characters?
• Are you using S3Express?
• Are you able to turn on verbose logging using below statements and share the detailed logs:

  Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
  Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
  Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Thanks,
Ashish

[indimini]

@ashishdhingra - thanks for the quick response

• Sample Code (Simplified):

    PutObjectRequest putRequest = new PutObjectRequest
    {
      BucketName = s3bucket,
      Key = s3key, // Use an Object Id so we don't have to modify any models.
      ContentBody = Convert.ToBase64String(bytes),
    };
    putRequest.Metadata.Add("contenttype", contentType);
    putRequest.Metadata.Add("uploadtime", DateTime.Now.ToString());
    putRequest.Metadata.Add("filename", HttpUtility.UrlEncode(fileName, Encoding.ASCII));
  
    PutObjectResponse response = await client.PutObjectAsync(putRequest);
  

This code works w/o issue on Windows, fails when deployed. I've verified that the putRequest.Headers object we pass to the put method has no entries.

• Locale of Environment:
  Service ARN = arn:aws:ecs:us-east-1:834521116217:service-deployment/dev-cluster/jan2025-service/s-30HNRQCDVgxmBZwZlZr
  Operating System/Architecture = Linux/X86_64
  App Environment = Fargate
  Network mode = awsvpc

• Logging/Throwing: We log any exceptions to our log collection in our DB, then throw the exception. Contents of call stack provided in original post; note, there is no Inner Exception for the caught exception.

• Key or bucket names do not contain Non-ASCII characters. (Note, the code works without issue on Windows server, but fails only when deployed to Linux service in AWS/Fargate)

• We are not using S3Express

• I tried added the verbose logging but received no additional details. Again, since this doesn't fail in Development on Windows, there is nothing to capture. I did subclass the TraceListener and added code to capture any messages passed and store them in our log, but no messages were captured

[dscpinheiro]

You'll need to run this on your ECS cluster as well, but can you add AWSConfigs.LoggingConfig.LogMetrics = true; to the commands @ashishdhingra mentioned earlier? That'll include the request the SDK sent to S3 (including the headers):

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

using var client= new AmazonS3Client();
await client.ListBucketsAsync(...);

Example logs:

AmazonS3Client 31|2024-12-10T15:06:31.132Z|INFO|Request metrics: {"properties":{"AsyncCall":"True","CanonicalRequest":"GET\n/\n\nhost:s3.us-west-2.amazonaws.com\nuser-agent:aws-sdk-dotnet-coreclr/3.7.410.2 ua/2.0 os/windows#10.0.22631.0 md/ARCH#X64 lang/.NET_Core#8.0.11 md/aws-sdk-dotnet-core#3.7.400.60 api/S3#3.7.410.2 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1\nx-amz-content-sha256:...\nx-amz-date:20241210T200630Z\n\nhost;user-agent;x-amz-content-sha256;x-amz-date","StringToSign":"AWS4-HMAC-SHA256\n20241210T200630Z\n20241210/us-west-2/s3/aws4_request","ServiceName":"AmazonS3","ServiceEndpoint":"https://s3.us-west-2.amazonaws.com/","MethodName":"ListBucketsRequest","AmzId2":"...","StatusCode":"OK","BytesProcessed":"0","AWSRequestID":"..."},"timings":{"CredentialsRequestTime":0.8868,"RequestSigningTime":22.3808,"HttpRequestTime":582.7608,"ResponseUnmarshallTime":13.1556,"ResponseProcessingTime":29.586,"ClientExecuteTime":731.1859},"counters":{}}

GET
/

host:s3.us-west-2.amazonaws.com
user-agent:aws-sdk-dotnet-coreclr/3.7.410.2 ua/2.0 os/windows#10.0.22631.0 md/ARCH#X64 lang/.NET_Core#8.0.11 md/aws-sdk-dotnet-core#3.7.400.60 api/S3#3.7.410.2 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1
x-amz-content-sha256:...
x-amz-date:20241210T200630Z

The SDK tries to fetch some information from the environment to build the user-agent, so I'm wondering if something wasn't parsed correctly.

[indimini]

when I capture the logs in my dev environment, calling ListBucketsAsync(), I see the following:

Amazon Verbose: 0 : Received response (truncated to 1024 bytes): [
c761c404745ae5badbe4174a608b6f0aa44dedf93f48fd3dbc85233c6609eb1badminrootagent-installation-bucket-skillsmart2020-10-26T13:50:10.000Zaws-cloudtrail-logs-834521116217-1509af7d2021-03-24T12:38:44.000Zcdktoolkit-stagingbucket-1u82o2oum2n6l2021-12-06T19:23:46.000Zcodepipeline-us-east-1-9859142755782023-11-04T06:48:14.000Zconnect-e6e1ed35f3b82020-04-03T17:03:52.000Zelasticbeanstalk-us-east-1-8345211162172021-12-06T17:44:49.000Zinsight-amazon-root2022-07-16T14:48:46]
Amazon Information: 1 : Request metrics: {"properties":{"AsyncCall":"True","CanonicalRequest":"GET\n/\n\nhost:s3.amazonaws.com\nuser-agent:aws-sdk-dotnet-coreclr/3.7.410.1 ua/2.0 os/windows#10.0.26100.0 md/ARCH#X64 lang/.NET_Core#8.0.8 md/aws-sdk-dotnet-core#3.7.400.59 api/S3#3.7.410.1 cfg/retry-mode#legacy md/ClientAsync cfg/init-coll#1\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date:20241210T204720Z\n\nhost;user-agent;x-amz-content-sha256;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","StringToSign":"AWS4-HMAC-SHA256\n20241210T204720Z\n20241210/us-east-1/s3/aws4_request\n63e7d88314de300d448a0e3a83d8a3049e9b31ccc24e85477f0160e203b0b97d","ServiceName":"AmazonS3","ServiceEndpoint":"https://s3.amazonaws.com/","MethodName":"ListBucketsRequest","AmzId2":"ldgHVuZaIrdLG5uvKrVYaK09vnI0pN3b/L2gcDhdcwpdGOn9RPmdXcQtjk8+xSWR1u5pHmIrrcg=","StatusCode":"OK","BytesProcessed":"0","AWSRequestID":"FTRHRX67QS7ZMH3R"},"timings":{"CredentialsRequestTime":0.625,"RequestSigningTime":11.3467,"HttpRequestTime":241.7178,"ResponseUnmarshallTime":7.1743,"ResponseProcessingTime":14.2298,"ClientExecuteTime":320.0976},"counters":{}}

[ashishdhingra]

@indimini Thanks for your reply. As @dscpinheiro mentioned, please collect logs from your ECS cluster as well. May be there is some extra header which is parsed by SDK which contains some characters not well handled.

Also share the .NET 8 version used on both your dev environment and ECS cluster. There was a known issue in some .NET version, so just curious if it popped back up in .NET 8 in certain version.

Thanks,
Ashish