[Messenger] [RFC] Correct way to handle random network failure when dispatching message

[tristanbes]

Symfony version(s) affected: 4.3

Description
When calling $this->messageBus->dispatch(); on a loop; if for some reason, there is a connectivity issue, preventing from publishing a message in the queue; then the code stops;

Aws\Sqs\Exception\SqsException
Error executing "SendMessage" on "https://sqs.eu-west-2.amazonaws.com/xyxyxyxyxyxy/my-aweome-queue-name"; AWS HTTP error: cURL error 6: Could not resolve host: sqs.eu-west-2.amazonaws.com (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

Screenshot of the Sentry trace

I think it is bad DX, because we don't except dispatch to stop code execution until the code on the production tells you it can happen :).
And after it's difficult to know which messages has been correctly sent to the queue and the one that didn't;

So I wrote this instead:

try {
    $this->messageBus->dispatch(Message::create());
} catch (\Exception $exception) {
    try {
        sleep(1);
        $this->messageBus->dispatch(KeywordMessage::create());
    } catch (\Exception) {
       // do nothing so the code still continue to iterate
    }
}

The code is not particularly beautiful; but allow at least 1 retry in case of some failure;

1. Should the message bus implement some logic to retry sending the message to the queue ?

2. should the developper implement and should be encouraged to wrap all calls to dispatch in a try/catch block ? (something similar to what has been done to the Symfony Http Client)

3. Or should the libraries/adapters underneath do that ?

I'm using:
sroze/messenger-enqueue-transport
enqueue/sqs
symfony/messenger (v4.3.4)

[dmaicher]

For me the current behavior is expected. I would like to have control myself over what happens when there is a serious issue with the transport layer.

Actually to me this is the same situation as when interacting with a database via Doctrine for example. I mean if the DB Server is down then Doctrine will also just let the exception bubble up the call stack and you can handle it or not.

Having a generic retry strategy seems tricky as it really depends on your exact needs.

I'm not using the messenger component myself but I guess if you really want to have a retry you could decorate the MessageBus and implement something like this yourself?

[tristanbes]

To me, the error it's not a "serious issue", since it's a random network/dns failure (it wasn't a real amazon SQS problem, the code just crashed in the middle of 8,000 over a 16,000 loop iteration);

So, If I get your point, you're in favor of the point 2 ? Because right now in the documentation (maybe i'm wrong); nothing tells you that dispatching a message could fail (fatal) and it's your responsability to deal with that;

I mean it's maybe common knowledge when you work with queues, but I never worked with it before, and I didn't know that; So adding this kind of warning maybe benefit for Messenger features newcomers ?

[dmaicher]

Yes I would go with option 2. What do you think @sroze ?

[xabbuh]

[...] if you really want to have a retry you could decorate the MessageBus and implement something like this yourself

this sounds like the solution to me

[tristanbes]

Ok thank you; I went with the decorator; I still think developpers should be warned against this type of failure and it's their responsability to implement this using a decorator for example to implement a retry logic :)

Thanks

[Nyholm]

I see it as the same problem if you cannot connect to your database.

Im happy you solved the problem.