s3.getObject Unable to execute HTTP request: Timeout waiting for connection from pool #2663
Comments
umutyusuf
added
bug
|
@umutyusuf thank you for reaching out. Is the S3 client shared or is a new instance of the client created with every new request? We recommend reusing the same client instance whenever possible. The error If you enable the client-side metrics you can obtain more insights on the performance of your application over time and perhaps see why the error only occurs after 5h. Metrics like To enable the client-side metrics see the blog post - https://aws.amazon.com/blogs/developer/enabling-metrics-with-the-aws-sdk-for-java/ |
debora-ito
added
response-requested
|
Hey @debora-ito, thanks for the response. the |
github-actions
bot
removed
the
response-requested
|
The errors started to after 6 days without any changes. So I made a little adjustment in the code after the exceptions started. private List<SomeObject> parseFile(@NotNull S3Object object) {
final String fileKey = object.getKey();
try (object) {
try (final GZIPInputStream gzipInputStream = new GZIPInputStream(object.getObjectContent())) {
final byte[] bytes = IOUtils.toByteArray(gzipInputStream);
return objectMapper.readValue(bytes, new TypeReference<>() {
});
}
} catch (Throwable t) {
logger.error("Error while reading result file for key = {}", fileKey, t);
}
return Collections.emptyList();
}with this private List<SomeObject> parseFile(@NotNull S3Object object) {
final String fileKey = object.getKey();
try {
return objectMapper.readValue(new GZIPInputStream(object.getObjectContent()), new TypeReference<>() {
});
} catch (Throwable t) {
logger.error("Error while reading result file for key = {}", fileKey, t);
}
return Collections.emptyList();
}Here As I mentioned in the original entry, this application runs on AWS Lambda. Meaning the application does not run indefinitely and is restarted roughly every 20 minutes. So I don't expect that the thread pool issue should not persist across different applications. I've also tried to enable the metrics as @debora-ito mentioned by adding an environment variable to lambda function as Key = But still can not see the metrics in the cloudwatch under PS: I have 3 separate applications that use this flow, read from the s3 process, and terminate. All three applications use different s3 buckets, but I only face this exception in one of them. |
|
hey, @debora-ito I've collected the values, and the issue is resurfaced. However, I failed to share the whole graph so I'll try to follow your lead on how you want the see the metrics. The reason for sharing these separately is that the connection time values jump to thousands and the other values are not readable anymore. After nearly 4 hours, For the same time period, here is the values for
As you can see from the graph, the available thread count always caught up with leased one which I assume indicates that the threads are released properly. I don't know where the problem is, ready to provide more info about the issue if you require it. Thanks. |
|
I think I found the issue, once the file size exceeds the max thread count, the new requests are blocked. Sorry for the inconvinence. |
debora-ito
added
guidance
|
@umutyusuf We're having the same problem, what do you mean by "file size exceeds the max thread count"? I'm only seeing it on larger files. |
|
@highfestiva I was trying to read too many objects in parallel and relying on the pool to be available once the file read operation is completed, however after performing tests, once the max thread count of the client exceeded, all the threads got blocked and the client halts in whole. I know this is an issue but I didn't want to persist further. |
|
Is anyone else able to explain how to programmatically recover from this error? I'm closing everything except the GetObjectRequest itself but the documentation and example code doesn't imply I need to close request objects. Avoiding reading too many files in parallel means I have to reverse engineer the connection pool logic etc to match it to what I'm doing and opens a can of worms on correctness. Can the connection pool be restarted? |
My solution is to create the client per request, cause as others have mentioned, there's no way to get the connection pool size and throttle accordingly. |
|
Seems to be a dupe of #1405 which seems to have a better solution than creating new s3 clients (for people stumbling upon this one in the future) |
Describe the bug
I'm having a problem while trying to get the object over the S3 instance.
com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from poolI'm trying to reach the s3 over lambda with EventBridge every 3 minutes with a max of 1 instance. After 5 hours, the exceptions start. This code runs in a single thread.
Here are the code pieces to that produces the exception.
The pieces that produces the exception
Expected behavior
As you can see from the code pieces, I'm closing every object that's opened, even though some exceptions might occur, every item in
objectsshould be properly closed. The expected behaviour is that the connections must be properly released, and new connections can be successfully established.Current behavior
Exception is thrown after a while (observed to be nearly 5 hours - triggered every 3 mins). Here are the logs
Steps to Reproduce
You can only reproduce this by running it for a long time with the same s3 instance.
Possible Solution
It's either in my code, or the connections are not released properly even though they are released properly. I've also checked the previously created issue but it got me nowhere.
Context
No response
AWS Java SDK version used
1.11.973
JDK version used
Java 11 (Corretto)
Operating System and version
Labmda - Java 11 (Corretto)
The text was updated successfully, but these errors were encountered: