Executing request in a reentrant handler causes connection timeouts

[stevenschlansker]

It appears that invoking a blocking http request from within a handler is not reentrant safe. The following code:

public class ReentrancyTest {
    private AsyncHttpClient http;

    @Before
    public void setUp() {
        System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "TRACE");
        http = new DefaultAsyncHttpClient(new DefaultAsyncHttpClientConfig.Builder()
                .setConnectTimeout(5000)
                .build());
    }

    @After
    public void tearDown() throws Exception {
        http.close();
    }

    @Test
    public void testReentrancy() throws Exception {
        dispatch()
            .thenApply(r -> {
                try {
                    dispatch().handle(this::handle).get();
                } catch (Exception e) {
                    e.printStackTrace();
                }
                return r;
            })
            .handle(this::handle)
            .get();
    }

    private CompletableFuture<Response> dispatch() {
        return http.prepareGet("http://www.google.com")
            .execute()
            .toCompletableFuture();
    }

    private Response handle(Response r, Throwable t) {
        if (t != null) {
            t.printStackTrace();
            return r;
        }
        System.out.println(r.getResponseBody());
        return r;
    }
}

hangs:

https://gist.githubusercontent.com/stevenschlansker/98ff589533f9c37ec2b7763a2e41dd25/raw/a013fa12170ec770900a86570233a1f96ba3bf45/hang.log

It looks like it tries to re-use the in flight connection that is still servicing the original transaction.

If this isn't supported, it should be explicitly documented. But it feels like a bug?

[slandelle]

But it feels like a bug?

Not a bug. You must NOT perform any blocking operation in the IO thread, where the CompletableFuture callback is executed, as you use thenApply and not thenApplyAsync.

The reason is that Netty performs thread affinity with connections. Hence, when performing the second blocking request, there's a chance the associated connection gets/is already pinned to the same thread, resulting in dead lock.

In your case, you always get a dead lock as we try to minimize the number of open connections, and as you hit the same host, it tries to reuse the same connection.

[slandelle]

I added a warning in the ÀsyncHandler javadoc. Not sure where to document this for the Future. Contribs welcome.

[stevenschlansker]

Okay, thanks for the explanation.