(short issue description) #3072

MichaelJHaywood · 2023-12-20T16:44:17Z

Describe the bug

I'm getting a Connection pool shut down exception when using a LambdaClient. The LambdaClient is created each time I need to call the lambda as the config can change.

After what appears to be a random number of calls it starts failing and throwing a Connection pool shut down exception. It will then fail with every subsequent call and is only resolved by restarting the service.

It's strange as we're creating the LambdaClient each time and it should call .close() as it's using a try-with-resource statement. It's as though there is something stored globally that is being shut down which means any new LambdaClients will fail to make a connection.

I've also noticed this error seems to occur after the service has been running for 60 minutes (I can't be certain). It works fine for many requests and then starts failing every time.

try (LambdaClient client = LambdaClient.builder()
        .region(region)
        .credentialsProvider(DefaultCredentialsProvider.create())
        .overrideConfiguration(
                builder -> builder.apiCallTimeout(Duration.ofSeconds(118))
                        .apiCallAttemptTimeout(Duration.ofSeconds(118))
                        .retryPolicy(RetryPolicy.none()))
        .httpClientBuilder(ApacheHttpClient.builder()
                .maxConnections(10)
                .socketTimeout(Duration.ofSeconds(130))
                .connectionTimeout(Duration.ofSeconds(130))
        )
        .build()) {
    res = client.invoke(request);
    lambdaResponse = res.payload().asUtf8String();
}

Exception

2023-12-20T16:23:27.765Z ERROR 1 --- [undedElastic-23] p.h.GlobalResponseEntityExceptionHandler : Run Time Exception: Connection pool shut down

java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.16.jar:4.4.16]
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.Mono.block(Mono.java:1712) ~[reactor-core-3.5.11.jar:3.5.11]
		at 
	4) ~[app/:na]
		at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:73) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.MonoRunnable.call(MonoRunnable.java:32) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:227) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.5.11.jar:3.5.11]
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
		at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
		at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Handler 
Original Stack Trace:
		at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.16.jar:4.4.16]
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[httpclient-4.5.14.jar:4.5.14]
		at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) ~[apache-client-2.21.11.jar:na]
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.14.jar:4.5.14]
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.14.jar:4.5.14]
		at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) ~[apache-client-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:89) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:52) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:37) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.21.11.jar:na]
		at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.21.11.jar:na]

Why is the connection pool closed for a new LambdaClient?

Expected Behavior

A LambdaClient should be able to be created and disposed without affecting other LambdaClients.

Current Behavior

A newly created LambdaClient throws an exception due to the connection pool being shutdown.

Reproduction Steps

This isn't easily reproduced. We only have this piece of code that creates a LambdaClient, makes a request and closes and this starts failing after the service has been running for some time.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.21.11

JDK version used

Amazon Corretto: 17

Operating System and version

Amazon Linux: 2

The text was updated successfully, but these errors were encountered:

debora-ito · 2024-01-03T22:13:08Z

It's strange as we're creating the LambdaClient each time and it should call .close() as it's using a try-with-resource statement.

Is it possible that some part of the system still tries to use the client after it got closed?

For context, the SDK does not close connection pools when the client is in use, the SDK only closes the connection pool when close() is invoked directly or indirectly (by something outside the SDK, like Spring @Bean destroyMethod() for example).

Have you look into this error?

	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103) ~[reactor-core-3.5.11.jar:3.5.11]
		at reactor.core.publisher.Mono.block(Mono.java:1712) ~[reactor-core-3.5.11.jar:3.5.11]

github-actions · 2024-01-09T00:05:45Z

It looks like this issue has not been active for more than five days. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.

MichaelJHaywood added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 20, 2023

debora-ito added p3 This is a minor priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Jan 3, 2024

debora-ito self-assigned this Jan 3, 2024

debora-ito added the response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days. label Jan 3, 2024

github-actions bot added closing-soon This issue will close in 2 days unless further comments are made. closed-for-staleness and removed closing-soon This issue will close in 2 days unless further comments are made. labels Jan 9, 2024

github-actions bot closed this as completed Jan 11, 2024

codespearhead mentioned this issue Jul 17, 2024

Bedrock API read timeout in 2 minutes, even I set it as 10 spring-projects/spring-ai#1061

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(short issue description) #3072

(short issue description) #3072

MichaelJHaywood commented Dec 20, 2023

debora-ito commented Jan 3, 2024

github-actions bot commented Jan 9, 2024

(short issue description) #3072

(short issue description) #3072

Comments

MichaelJHaywood commented Dec 20, 2023

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Java SDK version used

JDK version used

Operating System and version

debora-ito commented Jan 3, 2024

github-actions bot commented Jan 9, 2024