messages being processed multiple times by same worker name

[bokmann]

According to the readme, "subscribers that share the same application_name will act as multiple workers on the same queue. Only one will get to process each message", but this isn't true.

I have an app where processing each message is taking ~30 seconds of processor time.

Because of the code here:

propono/lib/propono/services/queue_listener.rb

Line 66 in 7b0dfe8

handle(sqs_message)

while the first worker is processing the message, a second worker has the opportunity to grab it.

Reversing those two lines kinds solves this issue, but creates a new one... a message can get lost if the worker fails during processing.

I don't think the semantics of sqs and sns alone can fix this issue one way or the other; sqs would need to be able to mark a message as 'processing'.

Perhaps deleting the message, calling the message handler, rescuing an exception, republishing the message, then re-raising the exception would be a potential strategy for handling this, but certainly raises the complexity.

[bokmann]

Diving into how the visibility_timeout parameter works with SQS

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

it seems like there is an answer, but that there might be a bug in propono.

I believe this line:

propono/lib/propono/services/queue_listener.rb

Line 42 in 7b0dfe8

messages = aws_client.read_from_sqs(queue, num_messages, long_poll: long_poll, visibility_timeout: visibility_timeout)

should reference "@visibility_timeout" rather than 'visibility_timeout'.

I'll investigate more during daylight.

[iHiD]

What have you set visibility_timeout to in your config? https://github.com/ihid/propono#visiblity-timeout

[iHiD]

should reference "@visibility_timeout" rather than 'visibility_timeout'.

That's just using the attr_reader not the instance variable directly. Shouldn't make a difference.

[bokmann]

Once I grokked the visibility timeout and resolved which client instance I needed to set it to, all is well.

I put it on my todo list to write some documentation for the project about this.