When retrying error on Queue Throttled, client must use a new ReceiveRequestAttemptId.

[navinvasu]

Library version

2.0.3

SQS configuration

fifo_queue = true
content_based_deduplication = false
visibility_timeout_seconds = 3600
receive_wait_time_seconds = 20
fifo_throughput_limit = perMessageGroupId
deduplication_scope = messageGroup
enable_dlq = true
dlq_max_receive_count = 3
dlq_visibility_timeout_seconds = 30
dlq_message_retention_seconds = 1209600

Reproduction steps

This error occurs when the SQS is on load.
Had multiple interactions with the AWS support engineers. Here are the details provide on the insight based the SQS server logs.
Queue Throttled” responses while receiving messages, and on the follow-up retry, the response becomes “ReceiveRequestAttemptId is invalid”
Client to retry a throttled request, should use a different ReceiveRequestAttemptId for receiving further messages.
There are no network issue nor it is a response with generic error. Instead, the request responded with Queue Throttled and therefore the ReceiveMessage API call cannot be retried with the same ReceiveRequestAttemptId.

Reason for throttling -> SQS service team found out that SQS FIFO queue gets scaled down for the reason that no send message action was performed for the last few minutes. And then, the request arrived to SQS service side and gets throttled.

When retrying error on Queue Throttled, client must use a new ReceiveRequestAttemptId.

Stack Trace

Stack trace ERROR ->

software.amazon.awssdk.services.sqs.model.SqsException: Value 85d410a1-f61d-4aeb-a92f-a5bfa7e3e523 for parameter ReceiveRequestAttemptId is invalid. Reason: The receiveRequestAttemptId is not valid anymore since the receive call is not retryable. (Service: Sqs, Status Code: 400, Request ID: b254394b-cf1a-52d9-90ac-675c71ae8949) at s.a.a.c.i.h.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) at s.a.a.c.i.h.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82) at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:60) at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:41) at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:40) at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:30) at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 48b7f4e2-911b-5d1e-8d21-aaf52fa843d0) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 0b15c096-5d5f-50e7-a454-9e81e3ec23f7)

Followed by WARN, stack trace ->

software.amazon.awssdk.services.sqs.model.SqsException: Value 85d410a1-f61d-4aeb-a92f-a5bfa7e3e523 for parameter ReceiveRequestAttemptId is invalid. Reason: The receiveRequestAttemptId is not valid anymore since the receive call is not retryable. (Service: Sqs, Status Code: 400, Request ID: b254394b-cf1a-52d9-90ac-675c71ae8949) at s.a.a.c.i.h.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) at s.a.a.c.i.h.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82) at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:60) at s.a.a.c.i.h.CombinedResponseHandler.handle(CombinedResponseHandler.java:41) at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:40) at s.a.a.c.i.h.p.s.HandleResponseStage.execute(HandleResponseStage.java:30) at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 48b7f4e2-911b-5d1e-8d21-aaf52fa843d0) Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Queue Throttled (Service: Sqs, Status Code: 403, Request ID: 0b15c096-5d5f-50e7-a454-9e81e3ec23f7)

Related error is mentioned here as well - aws/aws-sdk-java#2613 (comment)