Skip to content

Commit

Permalink
HTTPCLIENT-1924: HttpClient to shut down the connection manager if a …
Browse files Browse the repository at this point in the history 
…fatal error occurs in the course of a request execution
  • Loading branch information
@ok2c
ok2c committed Jun 16, 2018
1 parent 7c193f6 commit ca98ad6
Show file tree
Hide file tree
Showing 2 changed files with 6 additions and 0 deletions.
3 changes: 3 additions & 0 deletions httpclient/src/main/java/org/apache/http/impl/execchain/MainClientExec.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,9 @@ public CloseableHttpResponse execute(
} catch (final RuntimeException ex) {
connHolder.abortConnection();
throw ex;
} catch (final Error error) {

This comment has been minimized.

Sign in to view

Copy link
@nikhil-bidgely

nikhil-bidgely Jun 14, 2019 •
edited
Loading

@ok2c @sebbASF @garydgregory Shouldn't we just abort the connection instead of shutting down? Now when an error like: http://mail-archives.apache.org/mod_mbox/hc-dev/201806.mbox/%3CJIRA.13165247.1528697329000.142310.1528697340016@Atlassian.JIRA%3E comes. That is when JVM throws out of memory error we'll shut down the POOL and future request will all throw "Connection pool shut down".
This is what I've observed when the system was in load (and java heap space was low), but "Connection pool shut down" persist even after system recovered.

This comment has been minimized.

Sign in to view

Copy link
@ok2c

ok2c Jun 14, 2019

Author Member

@nikhil-bidgely Error thrown in the middle of a lease / release operation may leave the pool in an inconsistent state. Shutting down the entire pool is the only safe option. Generally I have never seen a complex Java application capable of automatic recovery from OutOfMemory condition without some loss of state.

This comment has been minimized.

Sign in to view

Copy link
@nikhil-bidgely

nikhil-bidgely Jun 14, 2019 •
edited
Loading

In the new version 5-beta I see that you're not shutting down the pool instead it's been closed, what the reason?
Server applications can usually recover from such condition, that is under heavy load they might throw the error but once the load is reduced they can continue processing.
We're using aws-java-sdk and with that after the above error the process is never able to run any method of the aws client and always throws "Connection pool shut down".
Also, regarding (aws/aws-sdk-java#1988) are we handling special characters like 'información de la energía.txt' in URLs? aws-sdk is again throwing signature mismatch.

This comment has been minimized.

Sign in to view

Copy link
@mmichna

mmichna Jan 25, 2020

@ok2c: we have service we process large documents (GB RAM) and we correctly recover from OutOfMemory error. We use several AWS services in our system but we have problem with AWS SQS when OutOfMemory occur and we then see only "Connection pool shut down" in our logs and no new messages are process with problematic ECS instance.

Is possible make this behavior configurable via some flag? I think this is very useful feature which avoid problem on lot of projects. thx

This comment has been minimized.

Sign in to view

Copy link
@michael-o

michael-o Jan 25, 2020 •
edited
Loading

Member

@mmichna Please raise a JIRA issue with your case.

This comment has been minimized.

Sign in to view

Copy link
@ok2c

ok2c Jan 26, 2020

Author Member

@mmichna Your service might be able to recover gracefully from OOM condition but the pooling connection manager cannot. It is way too complex. HttpClient cannot leave it in a potentially inconsistent state so it shuts it down. If you do not like the default behavior you can subclass PoolingHttpClientConnectionManager, disable its #shutdown method and provide a different means of shutting it down.

This comment has been minimized.

Sign in to view

Copy link
@Dishant18

Dishant18 Sep 14, 2022

@mmichna were you able to solve this issue with AWS SQS / S3? If yes, can you pls share how?

This comment has been minimized.

Sign in to view

Copy link
@SmallerCoder

SmallerCoder via email Sep 14, 2022

This comment has been minimized.

Sign in to view

Copy link
@sportymsk

sportymsk Feb 20, 2024

@mmichna - were you able to solve the connection pool shut down issue?

This comment has been minimized.

Sign in to view

Copy link
@mmichna

mmichna Feb 23, 2024

I provided OnOutOfMemoryErrorHealthIndicator spring-boot actuator component which only check existence of @Value("${management.endpoint.on-out-of-memory-error.pid-file:#{T(java.lang.String).format('%s/%s-oom.pid', '${java.io.tmpdir}', '${spring.application.name}')}}") file and in you application only configure JAVA_OPTS="${JAVA_OPTS} -XX:OnOutOfMemoryError='echo %p > /tmp/${APPLICATION_NAME}-oom.pid'"

With this approach you can make un-healthy any instance which produce OOM error
connManager.shutdown();
throw error;
}
}

Expand Down
3 changes: 3 additions & 0 deletions httpclient/src/main/java/org/apache/http/impl/execchain/MinimalClientExec.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,9 @@ public CloseableHttpResponse execute(
} catch (final RuntimeException ex) {
releaseTrigger.abortConnection();
throw ex;
} catch (final Error error) {
connManager.shutdown();
throw error;
}
}

Expand Down

0 comments on commit ca98ad6

Please sign in to comment.