Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue4810 #4823

Merged
merged 3 commits into from
Jul 26, 2021
Merged

Issue4810 #4823

merged 3 commits into from
Jul 26, 2021

Conversation

jbescos
Copy link
Member

@jbescos jbescos commented Jul 8, 2021

This refers to the issue #4810

When there are many requests, some threads are blocked forever waiting for a CompletableFuture that is never completed. This is reproducible in the test. Usually this means one of these 2 possibilities:

  1. There is an unexpected workflow that is not completing the requests.
  2. There is one issue with concurrence.

In the original issue there are some errors related to:

Exception in thread "jdk-connector-9566" java.lang.IllegalStateException: Request not found
        at org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool.getRequest

Right after that method, the request should be completed, but this does not happen when there is that error.

With the test I saw that when the thread blocks forever, it is possible to see some leftovers in 'requestsInProgress':

idleConnections: []
connections: []
pendingRequests: []
requestsInProgress: {org.glassfish.jersey.jdk.connector.internal.HttpConnection@24063407=org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool$RequestRecord@71cf2f80, org.glassfish.jersey.jdk.connector.internal.HttpConnection@739f3f9b=org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool$RequestRecord@3ca2389c, org.glassfish.jersey.jdk.connector.internal.HttpConnection@5788db8d=org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool$RequestRecord@759c7d70, org.glassfish.jersey.jdk.connector.internal.HttpConnection@4ba397a6=org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool$RequestRecord@5b983bdf, org.glassfish.jersey.jdk.connector.internal.HttpConnection@50dc70da=org.glassfish.jersey.jdk.connector.internal.DestinationConnectionPool$RequestRecord@5b339b0, ...

These requestsInProgress are stored in a HashMap, and multiple threads can access it (for example in DestinationConnectionPool#getRequest and DestinationConnectionPool#processPendingRequests). I just changed it by a ConcurrentHashMap and I am not able to get the issue anymore.

@jbescos
Copy link
Member Author

jbescos commented Jul 8, 2021

I set this as a draft because I am still investigating more.

@jbescos jbescos force-pushed the issue4810 branch 4 times, most recently from 740977f to 85288ba Compare July 9, 2021 06:24
Signed-off-by: Jorge Bescos Gascon <jorge.bescos.gascon@oracle.com>
Signed-off-by: Jorge Bescos Gascon <jorge.bescos.gascon@oracle.com>
@jbescos jbescos force-pushed the issue4810 branch 5 times, most recently from 7b51498 to d9b1098 Compare July 9, 2021 14:10
Signed-off-by: Jorge Bescos Gascon <jorge.bescos.gascon@oracle.com>
@jbescos jbescos marked this pull request as ready for review July 12, 2021 09:24
@jbescos
Copy link
Member Author

jbescos commented Jul 12, 2021

I am not able to reproduce any issue more but it is possible that there are more.

@jbescos jbescos requested review from jansupol and senivam July 12, 2021 10:27
Copy link
Contributor

@senivam senivam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants