blocking netty thread causes a calling http client to hang

[pkonyves]

I have some REST endpoints implemented in micronaut (service A). One of the endpoint calls another service (service X) using a java.net.http.HttpRequest. Service X may have a long response time, e.g. minutes. Once this call is in progress, I am regularly calling service A (from curl) and every once in a while it just hangs.

For me it looks like that once service A is in the calling to service X, it's holding one of the nioEventLoopGroup-1-x threads waiting for the blocking operation to finish. Subsequent calls to service A endpoints will be handled by different nioEventLoopGroup-1-# threads in a round-robin fashion. But once the blocking nioEventLoopGroup-1-x would have the turn, the request just hangs. This behavior is deterministic, seeing from the logs which nio thread is handling the request, I see when the blocked thread has the turn, the call to service A just hangs. Then I make a new request to service A, which will respond perfectly. If given the netty thread pool size of 5 and 1 netty handler thread is blocked, then I will have every 5th request to service A hanging.

In my opinion Micronaut should never try to assign an http request handling to a thread that is blocked but apparently this is happening.

Steps to Reproduce

Create two REST endpoints.

• one endpoint put Thread.sleep(10*60*1000), call it /blocking
• second endpoint return immediately, call it /ok
• set micronaut.server.netty.worker.threads: 3
• start application

make some calls to above endpoints from curl

Expected Behaviour

• call curl http://localhost:8080/blocking
  • expected: hangs, hit ctrl + C
• 1st call curl http://localhost:8080/ok ->
  • expected: return immediately
• 2nd call curl http://localhost:8080/ok ->
  • expected: return immediately
• 3rd call curl http://localhost:8080/ok ->
  • expected: return immediately

Actual Behaviour

• call curl http://localhost:8080/blocking
  • actual: hangs
• 1st call curl http://localhost:8080/ok ->
  • actual: return immediately
• 2nd call curl http://localhost:8080/ok ->
  • actual: return immediately
• 3rd call curl http://localhost:8080/ok ->
  • actual: Hangs

Environment Information

• Operating System: MacOS X, Docker openjdk 11, hotspotvm
• Micronaut Version: 1.2.8
• JDK Version: 11

[jameskleeh]

Please upload sample application(s) to reproduce the issue. Without knowing more specifics I don't know where I would start to investigate your issue

[pkonyves]

I will provide an example tomorrow, I cannot reproduce it on my windows machine.

[pkonyves]

Hi @jameskleeh i could finally reproduce the issue in a clean project based on the 'creating-your-first-micronaut-app' tutorial project.

Tested on Mac OS

Please:

git clone https://github.com/pkonyves/creating-your-first-micronaut-app.git
git checkout test
cd complete
./gradlew clean test

Test setup:

• There is an Http Filter MockAuthorizationFilter that mimics to be a 5 seconds long blocking task
• micronaut.server.netty.worker.threads: 5
• There is a test HelloControllerTest.failing_test_blocking_http_filter_blocks_subsequent_calls
  • This test will first make a call to the service that triggers the MockAuthorizationFilter therefore suspending a request handler thread for 5 seconds
  • Then will make several (10) calls to another endpoint that do not have http filter
  • The test will fail exactly at the 5th request because we had 5 netty worker threads and the 1st one is now suspended

I found that suspending the request handler thread must be within the Http Filter. Suspend within the @controller will not trigger this behavior.

[jameskleeh]

I don't see anything here that is a bug. You should never block the event loop. You can move blocking operations onto a separate thread pool.

Suspend within the @controller will not trigger this behavior.

Likely because the controller isn't being executed on the event loop

[pkonyves]

Hi @jameskleeh, thanks for the quick response. Now carefully re-reading the documentation it does tell that I should use the filter in a non-blocking way (https://docs.micronaut.io/latest/guide/index.html#_writing_a_filter).

Problem was that I expected that the io.micronaut.http.filter.HttpServerFilter works as in every normal framework (e.g. javax.servlet.Filter, javax.ws.rs.container.ContainerRequestFilter), just executes in the http request handler thread, therefore I don't need to do any other precautionary actions.

Wouldn't it be possible to execute the io.micronaut.http.filter.HttpServletFilter not in the event loop on a framework level?

If I may suggest it would be great to highlight in the documentation that any blocking operation within a HttpServerFilter will cause hard-to-track problems of unresponsive service.

[jameskleeh]

@pkonyves If you think the documentation can be made more clear, a PR would be welcome

It would not be desirable to offload the filter execution. Given filters are reactive in nature it should be assumed that blocking operations should be offloaded.

[pkonyves]

@jameskleeh thx, I will!