Creating an SQS queue results in invalid headers sent to X-Ray

[ctartamella]

Describe the bug

Using SQS as part of integration testing, we create a new FIFO queue. As part of this request, the error listed below occurs. It appears that the format of the string being sent in the X-Ray trace id header is incorrect based on AWS documentation.

As a work around, we provided our own HTTP message handler in order to strip these headers from the request before sending them, however it is unclear why this happens to begin with. This is code running on a local machine with credentials provided through a shared credential file. With the work around these tests run correctly. Similar bugs have been posted previously to the Python SDK. My uneducated guess is that the format of the X-Ray trace ID changed and the SDK somehow wasn't updated.

Work around below:

public class RemoveXrayMessageHandler : DelegatingHandler
{
    private const string XrayHeader = "X-Amzn-Trace-Id";
    
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.Remove(XrayHeader);
        return base.SendAsync(request, cancellationToken);
    }

    protected override HttpResponseMessage Send(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.Remove(XrayHeader);
        return base.Send(request, cancellationToken);
    }
}

        // Inject a message handler to remove AWS X-Ray headers from calls to SQS.
        _services.AddTransient<RemoveXrayMessageHandler>();
        _services.Configure<HttpClientFactoryOptions>(testConfig.LambdaConfig.EnqueuerQueue.HttpClientName, options =>
        {
            options.HttpMessageHandlerBuilderActions.Add(b =>
            {
                b.AdditionalHandlers.Add(b.Services.GetRequiredService<RemoveXrayMessageHandler>());
            });
        });

Expected Behavior

I would expect the queue to be created as desired.

Current Behavior

The following error occurs

   Amazon.SQS.AmazonSQSException : The request has a 'X-Amzn-Trace-Id' HTTP header which is reserved for AWS X-Ray trace header and has an invalid value 'dae04c7a-6a43-4d8c-9a86-6caac024711e'
---- Amazon.Runtime.Internal.HttpErrorResponseException : Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
  Stack Trace:
     at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)

Reproduction Steps

The following request documents what is being sent to SQS:

    {
        var fileText = await File.ReadAllTextAsync("sync-queue.json");
        var attributes = JsonConvert.DeserializeObject<Dictionary<string, string>>(fileText);
            
        var request = new CreateQueueRequest
        {
            QueueName = queueName,
            Attributes = attributes,
            Tags = new Dictionary<string, string>
            {
                { "PRODUCT", "VALUE" }
            }
        };

        var createResponse = await _sqsClient.CreateQueueAsync(request).ConfigureAwait(false);
    }

sync-queue.json:

{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "false"
}

Possible Solution

No known solution other than the work around (which would disable X-ray tracing).

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.Core 3.7.106.41
AWSSDK.SQS 3.7.102.9
AWSSDK.Extensions.NETCore.Setup 3.7.7

Targeted .NET Platform

.NET 6

Operating System and version

Windows 11

[ashishdhingra]

Hi @ctartamella,

Good afternoon.

Could you please share the below:

• Sample code solution which demonstrates enabling X-ray for your application?
• What is the execution environment for your code?

I found reference to SetRecursionDetectionHeader() in SDK which adds X-Amzn-Trace-Id header for recursion detection within Lambda workloads. The value is retrieved from Lambda _X_AMZN_TRACE_ID environment variable.

Thanks,
Ashish

[ctartamella]

So, this is running in some .NET integration tests on my local machine. Effectively a console application. We are not setting that environment variable at all nor are we trying to opt in to X-Ray. This header seems to be added automatically for reasons I can not tell.

[ashishdhingra]

So, this is running in some .NET integration tests on my local machine. Effectively a console application. We are not setting that environment variable at all nor are we trying to opt in to X-Ray. This header seems to be added automatically for reasons I can not tell.

@ctartamella Thanks for your response. Could you please share the sample code solution that reproduces the issue?

[ctartamella]

As I am working on putting together a minified code sample, there are a few context points I'd like to include:

1. These occur during integration tests only running under the XUnit runner.
2. (and maybe most important) These integration tests are hosting a lambda using a 3rd party package (MartinCostello.Testing.AwsLambdaTestServer). Looking at it's code, I do think it is the culprit. I see a place where he is setting Lambda-Runtime-Trace-Id to a random Guid. Is it possible this is the culprit? If so, my issue likely lies there and not with the AWS sdk.

It's not clear to me why this would be the case as I am using named HTTP clients, however it is the most likely.

[ashishdhingra]

As I am working on putting together a minified code sample, there are a few context points I'd like to include:

1. These occur during integration tests only running under the XUnit runner.
2. (and maybe most important) These integration tests are hosting a lambda using a 3rd party package (MartinCostello.Testing.AwsLambdaTestServer). Looking at it's code, I do think it is the culprit. I see a place where he is setting Lambda-Runtime-Trace-Id to a random Guid. Is it possible this is the culprit? If so, my issue likely lies there and not with the AWS sdk.

It's not clear to me why this would be the case as I am using named HTTP clients, however it is the most likely.

@ctartamella Thanks for the update. Also refer https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-traces, where it has information about the format of X-Amzn-Trace-Id header.

[ctartamella]

I've confirmed that the MartinCostello.Testing.AwsLambdaTestServer package is indeed the likely culprit here. I'm going to pursue this with him. Thanks for the attention!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[martincostello]

For anyone who may come across this issue in the future, this was fixed in v0.7.1 of Lambda Test Server.