Error messages have incorrect NServiceBus.FailedQueue header

[timbussmann]

Describe the bug

Description

Expected behavior

When a message is moved to the error queue from a regular endpoint using the SQS transport, the failed queue header (NServiceBus.FailedQ) contains the input queue name of the endpoint that tried to process the message.

This error queue is used for retries via ServicePulse / ServiceInsight.

Actual behavior

When a message is moved to the error queue from an Lambda endpoint, the failed queue header (NServiceBus.FailedQ) contains the queue URL instead of the queue name. When SQS transport is used in a regular endpoint, the failed queue header uses the queue name.

This is caused by the Lambda endpoint providing the queue URL to the ErrorContext. The SQS transport passes the queue name.

When trying to retry failed messages on an AWS Lambda endpoint using ServicePulse, the message can't be retried and the ServiceControl log files contains something like this:

ServiceControl.Recoverability.RetryProcessor|Staging batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54.
ServiceControl.Recoverability.RetryProcessor|Staging 1 messages for retry batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54 with staging attempt Id d589c030-a2e5-426c-a341-ed50de6050da.
ServiceControl.Recoverability.RetryProcessor|Retry batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54 staged with Staging Id d589c030-a2e5-426c-a341-ed50de6050da and 1 matching failure retries
ServiceControl.Recoverability.RetryProcessor|Batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54 with 1 messages staged and 0 skipped ready to be forwarded.
ServiceControl.Recoverability.RetryProcessor|Starting forwarder for batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54 with 1 messages in counting mode.
ServiceControl.Recoverability.ReturnToSenderDequeuer|Forwarder for batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54 started receiving messages from test-Particular-SC_SQS-staging.
NServiceBus.Transport.SQS.MessageDispatcher|Exception from Send.
System.Exception: Address https://sqs.eu-west-1.amazonaws.com/<redacted>/test-ServerlessEndpoint with configured prefix test- is longer than 80 characters and therefore cannot be used to create an SQS queue. Use a shorter queue name.
   at NServiceBus.Transport.SQS.QueueCache.GetSqsQueueName(String destination, TransportConfiguration transportConfiguration) in /_/src/NServiceBus.Transport.SQS/QueueCache.cs:line 78
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at NServiceBus.Transport.SQS.QueueCache.<GetQueueUrl>d__4.MoveNext() in /_/src/NServiceBus.Transport.SQS/QueueCache.cs:line 46
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<ApplyUnicastOperationMapping>d__18.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 455
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<PrepareMessage>d__10.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 279
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<DispatchBatched>d__4.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<Dispatch>d__1.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 61
2023-10-09 
ServiceControl.Recoverability.ReturnToSenderDequeuer+CaptureIfMessageSendingFails|Failed to send 'acb39abb-c46e-f004-bc0b-2869a1626306' message to 'https://sqs.eu-west-1.amazonaws.com/<redacted>/test-ServerlessEndpoint' for retry. Attempting to revert message status to unresolved so it can be tried again.
System.Exception: Address https://sqs.eu-west-1.amazonaws.com/<redacted>/test-ServerlessEndpoint with configured prefix test- is longer than 80 characters and therefore cannot be used to create an SQS queue. Use a shorter queue name.
   at NServiceBus.Transport.SQS.QueueCache.GetSqsQueueName(String destination, TransportConfiguration transportConfiguration) in /_/src/NServiceBus.Transport.SQS/QueueCache.cs:line 78
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at NServiceBus.Transport.SQS.QueueCache.<GetQueueUrl>d__4.MoveNext() in /_/src/NServiceBus.Transport.SQS/QueueCache.cs:line 46
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<ApplyUnicastOperationMapping>d__18.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 455
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<PrepareMessage>d__10.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 279
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<DispatchBatched>d__4.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 106
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.MessageDispatcher.<Dispatch>d__1.MoveNext() in /_/src/NServiceBus.Transport.SQS/MessageDispatcher.cs:line 66
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ServiceControl.Recoverability.ReturnToSender.<HandleMessage>d__1.MoveNext() in /_/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSender.cs:line 85
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ServiceControl.Recoverability.ReturnToSenderDequeuer.<Handle>d__8.MoveNext() in /_/src/ServiceControl/Recoverability/Retrying/Infrastructure/ReturnToSenderDequeuer.cs:line 59
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.Transport.SQS.InputQueuePump.<InnerProcessMessage>d__6.MoveNext() in /_/src/NServiceBus.Transport.SQS/InputQueuePump.cs:line 297
NServiceBus.Raw.RunningRawEndpointInstance|Stopping receiver.
NServiceBus.Raw.RunningRawEndpointInstance|Receiver stopped.
NServiceBus.Raw.StoppableRawEndpoint|Initiating shutdown.
NServiceBus.Raw.StoppableRawEndpoint|Shutdown complete.
ServiceControl.Recoverability.ReturnToSenderDequeuer|Forwarder for batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54 finished forwarding all messages.
ServiceControl.Recoverability.InMemoryRetry|Retry operation ade30380-9c89-f6a7-bf31-a0dba37c31b8 completed. 0 messages skipped, 1 forwarded. Total 1.
ServiceControl.Recoverability.RetryProcessor|Done forwarding batch RetryBatches/bf038349-ef96-4e3f-b824-b34b92e74a54.

Versions

tested with version 1.1 (and SC 4.32.2)

Steps to reproduce

1. Create an AWS Lambda endpoint with a handler that fails to process a message consistently
2. Send a message so that it will fail processing and be moved to the error queue
3. inspect the failed message in the error queue or ingest it into SC
4. (retry the message from ServicePulse)

Relevant log output

No response

Additional Information

• NServiceBus transport tests verify the expected behavior

Backported to

This bug has been backported to:

• version 1.0.1 via Error messages have incorrect NServiceBus.FailedQueue header #319