v4.4.0: Address race condition in dedupe handler
Full Changelog: v4.3.0...v4.4.0
SQS messages were observed being removed well before they should have been based on SQS queue re-drive and visibility policies. After some discussion and investigation, it was determined that a race condition in the deduplication logic was causing messages to be removed from the queue before expiry of the effective retention window. This race condition happens when:
- A message is consumed from the SQS queue
- A record is added to the dedupe cache with a TTL based on the SQS queue's visibility timeout
- this is the standard pattern
Between 1 and 2 some time passes, which was observed to be up to 2 seconds. Sometimes the message in the SQS queue is re-consumed before its record in the dedupe cache expires, causing the message to be removed from the queue early.
This issue is fixed by introducing a timeout_offset to the duplicate detection logic. The dedupe cache record TTL will be set with a timeout_offset-seconds shorter duration than the SQS queue's visibility timeout. While this introduces a surface area for duplicated work, the probability of such duplication is much higher in the first few seconds after a message is first consumed rather than the last few seconds before that message hits its visibility timeout.