What happens when the response is not closed with Apache Connector in Jersey Client?

[DashwoodIce9]

I run a Java web service at my workplace which uses JerseyClient to make HTTP requests. A few weeks ago we switched the default connector implementation with ApacheConnector and our server stopped handling requests after some time due to threads getting stuck. The issue was exactly same as described here - #3772. We were running a pretty outdated version of Jersey - 2.19. We fixed the issue by using a higher version of Jersey where this bug was fixed. However, I am still curious as to why things broke and want to understand them better.

According to the conversation on the GitHub issue, the problem is due to a bug in Jersey code wherein response.close() is not called.

This comment on the commit which fixes the issue says -

2 years down the line and because of a missing response.close() in previous versions of Jersey in a finally block, the pooled http connection was never released.
Our leased http connections never were returned to the pool because response wasn't being explicitly read into entities in some cases.

I don't understand what this means. I want to understand how does this bug work. Why do threads get stuck? Why is it important to call response.close(). What does response.close() do? What scenarios does the bug surface in? (because my server did not die outright, it was fine for a few days then broke).

I was able to reproduce this issue when I artificially injected high latency in the calls made by Jersey (we have a tool at work which can intercept packets and hold them for a set amount of time, so the application perceives that as latency in the response of the call). Which led to a javax.ws.rs.ProcessingException with "Read timed out". Why did this experiment reproduce the issue?

Thanks!

P.S. - I have asked the same question on StackOverflow as well - https://stackoverflow.com/questions/78114533

[jansupol]

See Response#close() javadoc:

Close the underlying message entity input stream (if available and open) as well as releases any other resources associated with the response (e.g. buffered message entity data).

The close() method should be invoked on all instances that contain an un-consumed entity input stream to ensure the resources associated with the instance are properly cleaned-up and prevent potential memory leaks. This is typical for client-side scenarios where application layer code processes only the response headers and ignores the response entity.

The Response object holds resources (streams, buffered data, even threads - Apache provides a connection pool associated with a thread pool - Apache connections can be reused). See a blog article related to the Apache Connector, stuck threads, and not-closed Responses and other Closables.

[DashwoodIce9]

Hi Jan, thanks for the info. I checked out the blog post you shared but more questions popped up in my head after that -

1. I understand resources are not cleaned up if close() is not called. But what I wanted to understand was that what exactly is getting exhausted which causes the request threads to get stuck? Is it the number of connections in the Apache connection pool? If yes, why don't connections auto expire and automatically reclaimed after some time?
2. It doesn't look like that Jersey Apache Connector v2.19 called close() on Apache's CloseableHttpResponse. So, from the above explanation this issue should appear in every case not just in an exceptional one. Yet, the threads don't get stuck in normal requests for me. They got stuck when I injected artificial latency, packet loss, things like that. Why would that happen? I am assuming that whatever functions CloseableHttpResponse.close() would have called are being called manually. So, in the happy flow, those functions are called and the responsibility of CloseableHttpResponse.close() is fulfilled without calling it. And in the unhappy case, an exception is raised earlier than these functions are called so the responsibility of CloseableHttpResponse.close() is not fulfilled. Is this correct?
3. I was able to reproduce the issue with high traffic to my webserver. I wasn't able to get the threads to get stuck with low traffic (even after running it for several hours whereas with high traffic server crashed in a few mins). So, is there some threshold of concurrent requests/threads that need to fail for this issue to surface? How does that work? Can you point me to some documentation for the same?

[jansupol]

While you may have good questions, you are discussing a piece of software from 2015 (Jersey 2.19) that worked somehow, but has been many times reworked since and you also discuss a 3rd party product (Apache) of an unknown version and possibly also very old (the Apache 4 is more than a year obsolete). Possibly, you will get better answers for Apache HTTP client internal details on their support page.

From the information you provided, it sounds like you need to set some additional timeouts for your connections.

[DashwoodIce9]

I'll try checking with the Apache team. Thank you for your help!