SQS: receives duplicate messages when Default Visibility Timeout is 0

[neil-rubens]

I have 1 message in my SQS queue; however calling
ReceiveMessageRequest().withMaxNumberOfMessages(10)
returns 3 messages at once (which should be impossible); all of which are duplicates.
(expected return should be 1 message).

paginator: sqsMsgs.length: 3

Changing Default Visibility Timeout: 0 -> 10
fixes the problem.

// get messages
    val sqsReceiveRequest = new ReceiveMessageRequest()
        .withQueueUrl(Paginator.sqsURL)
        .withMaxNumberOfMessages(10)
    val sqsMsgs = sqs.receiveMessage(sqsReceiveRequest).getMessages

    println("paginator: sqsMsgs.length: " + sqsMsgs.length)

[manikandanrs]

After you had receive a message from the queue, do you delete the message using the receipt handle ? If not, the the message appears in the queue after the visibility time out period.

[neil-rubens]

I do delete the messages. But even the first batch receive returns 3 messages even through the queue contains only 1 (I revised the post to clarify that).

p.s. I also tested it with sqsMsgs.size() just to make sure it wasn't a scala.collection.JavaConversions._ issue

[neil-rubens]

I am running it locally (on a home machine); am only using aws-java-sdk-sqs lib current version 1.10.75.

Here are my queue settings:

[shorea]

This seems to be expected behavior when setting a visiblity timeout of 0. No visibility timeout means you aren't reserving the message so it can be picked up by multiple consumers.

See http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html#AboutVT-general-recommendations-for-visibility-timeout

[neil-rubens]

I am aware that the same message might be received by multiple recipients (to provide receive at least once guarantee).

The problem is that I get multiple copies of a message within a batch (see above); so please re-open the issue.

[shorea]

My apologies, I see now what you mean. SQS does not guarantee a message is delivered only once and it's required to have idempotent processing. That said it does seem a little surprising you are receiving duplicates in the same batch. I was unable to reproduce, I would only ever get 1 message (or 0 messages due to short polling). This doesn't appear to be an SDK issue so I'll pass this info back to the service team to comment. In the meantime could you provide details on how often you see this and any request ids of affected requests.

[neil-rubens]

I was unable to reproduce, I would only ever get 1 message (or 0 messages due to short polling).

Is your SQS Queue's Default Visibility Timeout set to 0?

(if it is set to any other value then there are not dups within the batch).

[shorea]

Yeah I used the same settings in your screenshot above.

[shorea]

Still waiting for service team response by the way, I'll ping them again.

[shorea]

@neil-rubens here's the response from the service team. Would setting a non-zero visibility timeout meet your application's needs? If not could you perhaps code around receiving multiple messages and filter out duplicates?

While this seems like quite an edge case, setting the visibility timeout to 0 is not a recommended value as it could cause unpredictable behaviour from the service.

It may seem counter intuitive at first but if you set the visibility to 0 (the message is always visible) and ask for 10 messages, then you can expect to receive that message up to 10 times as you have made that message always visible at the millisecond granularity - each time a new message is asked for from a backend cluster and the same message appended to the batch of messages to be sent back in the customer request.

We don't state in our docs not to set the value to 0, though it seems like odd use case. If the message(s) are always visible it seems more like they are using SQS as a key/value store and there are other AWS services e.g. elasticache, DDB for that use case.

[neil-rubens]

@shorea thank you very much getting the reply from the service team; it explains the duplicate behavior I've encountered.

My use case was as following; I am using SQS as a simple semi-persistent job queue for lambda (hence didn't use SNS). I've overcome the issue by simply increasing the wait time; but it took me a while to track down the problem. It could be beneficial to have some pop-up details in the queue settings for the 0 wait time; since the default behavior is atypical.

Thank you for your help.

[devxpy]

odd usecase

What about having a distributed system where you have multiple consumers that want to read from the queue at the same time.

To be more specific, I have my s3 configured to issue a message to SQS whenever a new file is uploaded.

Then I require a number of devices to read that notification and respond accordingly. Now there is no guarantee that these devices are alive when the message is recieved so the queue needs to be asynchronous.

Furthermore, the file itself is uploaded by one of these devices.

I tried sampling the s3 storage regularly from these devices, but that was too heavy on the CPU.

I chose SQS since it has polling, which has almost no CPU usage when the device is waiting for message.

[vposham]

odd usecase

What about having a distributed system where you have multiple consumers that want to read from the queue at the same time.

To be more specific, I have my s3 configured to issue a message to SQS whenever a new file is uploaded.

Then I require a number of devices to read that notification and respond accordingly. Now there is no guarantee that these devices are alive when the message is recieved so the queue needs to be asynchronous.

Furthermore, the file itself is uploaded by one of these devices.

I tried sampling the s3 storage regularly from these devices, but that was too heavy on the CPU.

I chose SQS since it has polling, which has almost no CPU usage when the device is waiting for message.

Isn't there any other way to accomplish this scenario? - Even Im looking to accomplish something similar. I was exploring this option and support saying "setting the visibility timeout to 0 is not a recommended value as it could cause unpredictable behaviour from the service" scares me to use it in production.