fix Sqs fifo message group invisibility after triggering move to dlq #10859
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Users raised the issue that fifo queues become unusable after messages were moved to a DLQ.
After a fifo queue's message's receive count exceeds the maximum and is moved to a DLQ, its message group remains invisible. This PR addresses this issue.
fixes #10107
Changes
I am not happy with how this change is implemented, but I am unsure what the best way is.
The root cause is that we simply never addressed the message group visibility when a dlq move is triggered, so obviously we need to address that. There are two main ways to do this that I see:
isinstance(queue,FifoQueue)
check, and the provider shouldn't care what kind of message it handles, the point of the different classes is to abstract that away.queue._on_remove_message
, to get rid of the Fifo check. But this method seems to actually be "queue._on_delete_message" from a program-logic point of view and is expected to be called during deletion. Moving a message to a dlq is very similar, but not quite the same, and this approach would result in e.g. a re-heapifying for a standard queue every time (that's the reason why this solution was discarded).put
to the dlq. This options also has the advantage that we can add this message group visibility related change to the place in the code where the other checks of this kind are. However we would need the source queue object for that:put
for standard_queues too though, as well as passing it all the way down from the provider, and that doesn't seem that clean either.DeadLetterQueueSourceArn
with the message, and could get the queue from said arn. This way, we would have the change exactly where it belongs (in my opinion) without touching other code from a program-logic point of view. However, the queues are not context aware, and we would need that to access the proper queue from the store. This comes with some medium sized refactoring and is therefore quite the architectural change, which was the reason why this was discarded.Sorry for the wall of text, if something is unclear please ask ahead. The reason for it is that message group visibility has been quite the back and forth with patching so far, so I'd like to make extra sure that we do this the right way.