App stops consuming from SQS and becomes idle without any exception

[realradical]

We have a bizarre issue. Our app is hosted on ECS. What it does is pulling messages from a SQS queue and send them to our Kafka cluster. We usually have 20 tasks running. And from time to time, a container will become idle meaning CPU usage is close to 0 and SQS consume rate is 0, but the container is still running without any exception. We even enabled DEBUG logging option for AWS SDK. It is not printing any error. The last log stopped at the time as soon as CPU drops to 0.

[2021-01-18 07:16:44,558] [aws-java-sdk-NettyEventLoop-0-6] DEBUG software.amazon.awssdk.request - Received successful response: 200

You can check the below screenshot for the most recent issue. This container stopped consuming message at 7:16AM UTC.

Describe the bug

Our app is a "reactive streaming" app built with fs2 streaming framework (see doc https://fs2.io/). It uses "software.amazon.awssdk.services.sqs.SqsAsyncClient" to continuously poll messages from Sqs queue. The polled messages become the source of the stream. The stream processes each massage and sends them to a Kafka topic. Therefore, it is a non-stopping process.

Our issue is that there is no exception caught in our app. Our app can catch the exception and terminate the stream, which results in ECS task restarting. However, Aws SDK does not throw an exception in this case, not even useful debug logging.

Expected Behavior

In the case of SqsAsyncClient stops consuming messages, SDK should throw an exception with a detailed explanation.

Current Behavior

We even enabled DEBUG logging option for AWS SDK. It is not printing any error. The last log stopped at the time as soon as CPU drops to 0.
[2021-01-18 07:16:44,558] [aws-java-sdk-NettyEventLoop-0-6] DEBUG software.amazon.awssdk.request - Received successful response: 200

Your Environment

• AWS Java SDK version used: 2.15.59
• JDK version used: openjdk:11-jre-slim
• Operating System and version: linux

[debora-ito]

@realradical can you provide a thread dump? It may help to see the container threads activities after it stops consuming messages.

[realradical]

This is the thread dump from that container.
log-events-viewer-result (1).csv.zip

[debora-ito]

Hi @realradical thank you for the thread dump. I don't see anything unusual though.

Some questions that might help in the investigation:

• Can you share a reproducible code sample, including how you are creating the SQS client?
• Can you enable SDK client-side metrics? You can read more about it in the blog and in the Dev Guide.
• Did this start recently with a package or code update, or has it always happened?
• Do you have any other containers running the same code?

[realradical]

• Do you just need a code snippet where you can check how we create the SQS client? it is maybe against the company policy to share the entire project.
• Yes, I am enabling the client-side metrics together with a workaround. Basically we terminate the app by throwing a custom exception if we detect SQS client is idling. So ECS will replace it with a new task.
• It has always been there. I have updated the SDK version lately, the issue still exists.
• Yes we have 20 tasks(containers) running. The number of idling tasks will increase gradually. Usually a random task becomes unhealthy after a few days.

[debora-ito]

Just a code snippet would be ok, but ideally we'd like to have a reproducible case. Please don't provide confidential code. It's hard to investigate such cases without a repro code.

Let us know what you find in the client-side metrics.

[debora-ito]

@realradical if you're concerned about sharing the info here on github, you can provide it in the AWS Support ticket you've opened about this issue, just still be mindful about your company policy.