AmazonSQS Transport : Retries don't work #1433

AbdoHassanUMG · 2018-09-25T16:47:24Z

Message Replay doesn't work when using AmazonSQS transport because of inconsistency in the casing of Error Queue. It is called "{InstanceName}.Errors" when writing the error message and "{InstanceName}.errors" when forwarding it.
This should cause no problem if tested using MSMQ but causes trouble in SQS because queue names are case-sensitive.
I have created a fix for this and I want to send it as a pull request

AbdoHassanUMG · 2018-09-25T16:49:06Z

I think the problem is here

public class RawEndpointFactory
    {
        public RawEndpointFactory(Settings settings, TransportSettings transportSettings, TransportCustomization transportCustomization)
        {
            this.transportSettings = transportSettings;
            this.settings = settings;
            this.transportCustomization = transportCustomization;
        }

        public RawEndpointConfiguration CreateRawEndpointConfiguration(string name, Func<MessageContext, IDispatchMessages, Task> onMessage, TransportDefinition transportDefinition)
        {

            var config = RawEndpointConfiguration.Create(name, onMessage, $"{transportSettings.EndpointName}.errors");
            config.LimitMessageProcessingConcurrencyTo(settings.MaximumConcurrencyLevel);

            transportCustomization.CustomizeRawEndpoint(config, transportSettings);
            return config;
        }

        Settings settings;
        TransportCustomization transportCustomization;
        private TransportSettings transportSettings;
    }

danielmarbach · 2018-09-26T06:30:31Z

Correct AmazonSQS is case sensitive. Really weird that our acceptance tests didn't catch that one. We'll have to do a closer investigation, especially because we have tests that do retry and they are green in our CI environment. Otherwise, we wouldn't have released this functionality

Thanks for reporting

danielmarbach · 2018-09-26T07:16:44Z

@AbdoHassanUMG Can you explain why you think retries don't work? The queue we are talking about is Particular-ServiceControl-Errors which ServiceControl sends internally failed messages to. That queue is not used for message retrying. While we still want to fix the casing there and your PR is valid the retry functionality will use the staging queue Particular-ServiceControl-staging.

AbdoHassanUMG · 2018-09-26T12:16:45Z

Without handling the case-sensitivity I get the following error at SC console when attempting a "retry"
Looking at trafic to SQS using fiddler I discovered the cases sensitivity issue as one of SQS API calls returns a 400 Bad request.

2018-09-26 13:01:25.7133|93|Info|ServiceControl.Recoverability.RetryProcessor|Forwarding batch RetryBatches/935223a9-4c7a-4457-b88c-76fc19a965bb with 1 messages

2018-09-26 13:01:40.0814|148|Fatal|NServiceBus.Raw.StartableRawEndpoint|Receiver failed to initialize. Amazon.SQS.Model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.Runtime.Internal.HttpRequest.d__16.MoveNext()
--- End of inner exception stack trace ---
at Amazon.Runtime.Internal.HttpRequest.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.Internal.HttpHandler1.d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.Runtime.Internal.Unmarshaller.d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.SQS.Internal.ValidationResponseHandler.d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext() --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler1.Handle(IExecutionContext executionContext, Exception exception)
at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
at Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.Runtime.Internal.CallbackHandler.d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.Runtime.Internal.CredentialsRetriever.d__71.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.Runtime.Internal.RetryHandler.d__101.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at NServiceBus.AmazonSQS.QueueUrlCache.<GetQueueUrl>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at NServiceBus.Transports.SQS.MessagePump.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at NServiceBus.Raw.StartableRawEndpoint.d__14.MoveNext()`

danielmarbach · 2018-09-26T13:41:39Z

Ah ok! That makes sense. We recently changed the behavior of the raw factory to no longer auto-create queues and that's why it is now broken. Smells like a patch to me. Thoughts @Particular/servicecontrol-maintainers ?

Fix for AmazonSQS Transport : Retries don't work #1433

danielmarbach · 2018-09-27T06:48:12Z

Fixed as part of #1434. Thanks @AbdoHassanUMG for the contribution!

AbdoHassanUMG · 2018-09-27T10:01:45Z

Thanks @danielmarbach
When is release 3.2.0 scheduled for ?

danielmarbach · 2018-09-27T14:39:55Z

Soonish ;) Latest Monday I think

AbdoHassanUMG · 2018-09-27T15:49:09Z

Thanks Daniel !

danielmarbach · 2018-09-28T06:25:22Z

@AbdoHassanUMG it is out

https://github.com/Particular/ServiceControl/releases/tag/3.2.0
https://discuss.particular.net/t/servicecontrol-3-2-0-minor-release-available/788

Thank you so much for your patience and contributions!

AbdoHassanUMG · 2018-10-01T09:57:25Z

@danielmarbach Thanks a lot for turning this arranging this release quickly.
but the release link above doesn't show the executable as download ? And the Platform Download page still shows 3.1.2 as the latest release.

danielmarbach · 2018-10-01T12:56:27Z

Sorry we had a few hick ups with the build pipeline https://github.com/Particular/ServiceControl/releases

AbdoHassanUMG added a commit to AbdoHassanUMG/ServiceControl that referenced this issue Sep 25, 2018

Fix for AmazonSQS Transport : Retries don't work Particular#1433

1b586ec

danielmarbach mentioned this issue Sep 26, 2018

Fix for AmazonSQS Transport : Retries don't work #1433 #1434

Merged

danielmarbach added the Bug label Sep 27, 2018

danielmarbach added this to the 3.2.0 milestone Sep 27, 2018

danielmarbach added a commit that referenced this issue Sep 27, 2018

Merge pull request #1434 from AbdoHassanUMG/issue-1433

d77244d

Fix for AmazonSQS Transport : Retries don't work #1433

danielmarbach closed this as completed Sep 27, 2018

AbdoHassanUMG mentioned this issue Oct 4, 2018

SQS transport ignores Queue Name Prefix in Connection String #1432

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AmazonSQS Transport : Retries don't work #1433

AmazonSQS Transport : Retries don't work #1433

AbdoHassanUMG commented Sep 25, 2018

AbdoHassanUMG commented Sep 25, 2018 •

edited by mauroservienti

danielmarbach commented Sep 26, 2018

danielmarbach commented Sep 26, 2018

AbdoHassanUMG commented Sep 26, 2018

danielmarbach commented Sep 26, 2018

danielmarbach commented Sep 27, 2018

AbdoHassanUMG commented Sep 27, 2018

danielmarbach commented Sep 27, 2018

AbdoHassanUMG commented Sep 27, 2018

danielmarbach commented Sep 28, 2018

AbdoHassanUMG commented Oct 1, 2018

danielmarbach commented Oct 1, 2018 via email

AmazonSQS Transport : Retries don't work #1433

AmazonSQS Transport : Retries don't work #1433

Comments

AbdoHassanUMG commented Sep 25, 2018

AbdoHassanUMG commented Sep 25, 2018 • edited by mauroservienti

danielmarbach commented Sep 26, 2018

danielmarbach commented Sep 26, 2018

AbdoHassanUMG commented Sep 26, 2018

danielmarbach commented Sep 26, 2018

danielmarbach commented Sep 27, 2018

AbdoHassanUMG commented Sep 27, 2018

danielmarbach commented Sep 27, 2018

AbdoHassanUMG commented Sep 27, 2018

danielmarbach commented Sep 28, 2018

AbdoHassanUMG commented Oct 1, 2018

danielmarbach commented Oct 1, 2018 via email

AbdoHassanUMG commented Sep 25, 2018 •

edited by mauroservienti