bug: Using DLQ for DLQ, polling/receiving from the first DLQ causes the message to go into the second DLQ immediately

[eyalrin]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Defined 3 SQS queues:

1. Queue1 (Set it's DLQ to Queue2 using REDRIVE_POLICY with maxReceiveCount=1 and VISIBILITY_TIMEOUT=3)
2. Queue2 (Set it's DLQ to Queue3 using REDRIVE_POLICYwith maxReceiveCount=1 and VISIBILITY_TIMEOUT=3)
3. Queue3

Therefore the messages flow is Queue1 -> Queue2 -> Queue3 (in case of no messages deletion and more then 1 receive operation on the queue).

Flow:

1. Send message to Queue1 (aws sqs send-message)
2. Poll message from Queue1 (aws sqs receive-message) and got message
3. Wait for VISIBILITY_TIMEOUT to expire
4. Poll message from Queue1 (aws sqs receive-message) and got no message -> message goes to Queue2
5. Poll message from Queue2 (aws sqs receive-message) - got no message -> message goes to Queue3

Expected Behavior

Flow (Same queues configuration):

1. Send message to Queue1 (aws sqs send-message)
2. Poll message from Queue1 (aws sqs receive-message) and got message
3. Wait for VISIBILITY_TIMEOUT to expire
4. Poll message from Queue1 (aws sqs receive-message) and got no message -> message goes to Queue2
5. Poll message from Queue2 (aws sqs receive-message) and got message
6. Poll message from Queue2 (aws sqs receive-message) - got no message -> message goes to Queue3

Look at 5 - This step does not happen actually in Localstack (tested same configuration on AWS and it works as expected)

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker compose up -d

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

awslocal sqs create-queue --queue-name queue3 --attributes file:///docker-entrypoint-initaws.d/sqs-config/queue3-attributes.json

awslocal sqs create-queue --queue-name queue2 --attributes file:///docker-entrypoint-initaws.d/sqs-config/queue2-attributes.json
 
awslocal sqs create-queue --queue-name queue1 --attributes file:///docker-entrypoint-initaws.d/sqs-config/queue1-attributes.json

(Remove the .txt from each file - added it just to be able to upload)
queue1-attributes.json.txt
queue2-attributes.json.txt
queue3-attributes.json.txt

Environment

- OS: MacOS
- LocalStack: localstack-full:0.13.0.10

Anything else?

No response

[whummer]

Hi @eyalrin , thanks for reporting. Indeed, there's currently a few issues with the default SQS provider.

We have recently added a new, improved SQS implementation, which can be selected by setting this config option:

PROVIDER_OVERRIDE_SQS=asf

Seems like with this option, your use case should be supported. Can you please give that a try and let us know? Thanks! (We'll look into making this the default SQS provider in the near future..)

[eyalrin]

Hi @whummer,

I Tried setting .withEnv("PROVIDER_OVERRIDE_SQS", "asf") in my code but it causes Localstack’s SQS to not load properly (I guess) because now I can’t create a queue…

[thrau]

hey @eyalrin, i see you are using 0.13.0.1, but the asf SQS provider was added in 0.13.3, would it be possible to upgrade localstack to that version?

[eyalrin]

Hi @thrau & @whummer,

I tried now with localstack:0.13.3, seems SQS loads properly now but when I create in code the queue and tries to get its URL like so:

SqsClient sqsClient = SqsClient.builder().region(Region.of(AWS_CONTAINER.getRegion())).endpointOverride(AWS_CONTAINER.getEndpointOverride(SQS)).build();
sqsClient.createQueue(CreateQueueRequest.builder().queueName("queueName").build()).queueUrl()

I get:

http://localhost:4566/000000000000/queueName

where before the change I used to get:

http://aws-localstack:55416/000000000000/queueName

Why is that? Seems like it doesn't respect testcontainers rules...I don't want to change all my settings now to a new URL structure...

[whummer]

Hi @eyalrin , thanks for the update. Looks like the new SQS provider didn't fully respect the HOSTNAME_EXTERNAL configuration yet. This should now be fixed via #5238 - can you please pull the latest Docker image one more time and give it a try? Thanks!

[eyalrin]

Hi @whummer,

Now I getting a mixture of the two:

http://aws-localstack:4566/000000000000/webhook-delivery-deadletter

i.e., aws-localstack instead of localhost but still 4566 instead of the textcontainer port (5 digits, start with 55)