Tomcat is not able to stop because of "OkHttp ConnectionPool" and "Okio Watchdog" threads

[dani-e-l]

There is no possibility to stop/terminate OkHttp ConnectionPool and Okio Watchdog threads.
When there is long poling connections e.g. 10 minutes there is no way to terminate it and stop threads.
Method evictAll doesn't work in this situation.
Threads will be stopped after 10 minutes but it is to late and Tomcat will never stop.
Many API'a like Consul use long polling

apache-tomcat-9.0.26

Classes RealConnectionPool and ConnectionPool are final and there is no possibility to do anything with sockets

08-Oct-2019 11:29:38.518 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [okapi_demo_app_jaxrs_war] appears to have started a thread named [OkHttp ConnectionPool] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:460)
 okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)

08-Oct-2019 13:42:21.959 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [okapi_demo_app_jaxrs_war] appears to have started a thread named [Okio Watchdog] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:460)
 okio.AsyncTimeout$Companion.awaitTimeout$jvm(AsyncTimeout.kt:358)
 okio.AsyncTimeout$Watchdog.run(AsyncTimeout.kt:229)

[swankjesse]

If you have long polling calls that are still active, you need to cancel those yourself.

[dani-e-l]

Long polling is not the problem.

The problem is okhttp creates threads and do not allow to cleanup.
In my opinion every library that creates threads should allow to terminate them.
Okhttp doesn't and that is the problem.
Everyone who will use this lib in Tomcat will not be able to stop this container because of working/idle threads.
There is no possibility to stop "Okio Watchdog" and "OkHttp ConnectionPool" threads using API

I agree that those threads will stop after defined timeouts but there is need to stop them NOW.

My suggestion is to enrich an API that allows publicly shutdown/stop "Okio Watchdog" thread and shutdownNow ThreadPoolExecutor

In okio.AsyncTimeout

new Watchdog().start()

No way to stop it.

In okhttp3.internal.connection.RealConnectionPool

    private val executor = ThreadPoolExecutor(
        0, // corePoolSize.
        Int.MAX_VALUE, // maximumPoolSize.
        60L, TimeUnit.SECONDS, // keepAliveTime.
        SynchronousQueue(),
        threadFactory("OkHttp ConnectionPool", true)

No way to shutdownNow

[dani-e-l]

This is how it works.
ThreadPools have keepAliveTime = 60s

    private val executor = ThreadPoolExecutor(
        0, // corePoolSize.
        Int.MAX_VALUE, // maximumPoolSize.
        60L, TimeUnit.SECONDS, // keepAliveTime.
        SynchronousQueue(),
        threadFactory("OkHttp ConnectionPool", true)
    )

and RealConnectionPool is waiting additional 5min(by default) in this@RealConnectionPool.lockAndWaitNanos(waitNanos)

And there is no API to shutdownNow those ThreadPoolExecutors/Threads

As expected all threads dies after 6 minutes

Demo app

package okhttp;

import okhttp3.*;
import org.jetbrains.annotations.NotNull;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.mockserver.integration.ClientAndServer;

import java.io.IOException;

import static org.mockserver.integration.ClientAndServer.startClientAndServer;
import static org.mockserver.model.HttpRequest.request;
import static org.mockserver.model.HttpResponse.response;

public class OkhttpThreads {

  private static ClientAndServer mockServer;


  @BeforeAll
  public static void startMockServer() {
    mockServer = startClientAndServer(12345);
    mockServer.when(request().withMethod("GET").withPath("/test"))
        .respond(response().withBody("Hello there!"));
  }

  @AfterAll
  public static void stopMockServer() {
    mockServer.stop();
  }

  @Test
  public void test() throws IOException, InterruptedException {

    OkHttpClient.Builder builder = new OkHttpClient.Builder();
    ConnectionPool cp = new ConnectionPool();
    builder.connectionPool(cp);
    OkHttpClient client = builder.build();


    Request req = new Request.Builder()
            .url("http://127.0.0.1:12345/test")
            .build();
    client.newCall(req).enqueue(new Callback() {
      @Override
      public void onFailure(@NotNull Call call, @NotNull IOException e) {
        e.printStackTrace();
      }

      @Override
      public void onResponse(@NotNull Call call, @NotNull Response response) throws IOException {
        System.out.println("Async Response: " + response.body().string());
      }
    });

    Thread.sleep(1000);

    cp.evictAll();
    client.dispatcher().cancelAll();

    int counter = 0;
    int sleepSec = 10;
    while (true) {
      System.out.println("Print OK threads " + (counter++ * sleepSec) + "s.");
      Thread.getAllStackTraces().keySet().stream().filter(t -> t.getName().contains("Ok"))
              .forEach(t -> System.out.println("OkHttp thread \"" + t.getName() + "\" is in state " + t.getState()));
      Thread.sleep(sleepSec*1000);
    }
  }

}

System out

Async Response: Hello there!
Print OK threads 0s.
OkHttp thread "Okio Watchdog" is in state TIMED_WAITING
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
OkHttp thread "OkHttp Dispatcher" is in state TIMED_WAITING
Print OK threads 10s.
OkHttp thread "Okio Watchdog" is in state TIMED_WAITING
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
OkHttp thread "OkHttp Dispatcher" is in state TIMED_WAITING
.
.
.
Print OK threads 50s.
OkHttp thread "Okio Watchdog" is in state TIMED_WAITING
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
OkHttp thread "OkHttp Dispatcher" is in state TIMED_WAITING
Print OK threads 60s.
OkHttp thread "Okio Watchdog" is in state TIMED_WAITING
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
Print OK threads 70s.
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
Print OK threads 80s.
.
.
.
Print OK threads 340s.
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
Print OK threads 350s.
OkHttp thread "OkHttp ConnectionPool" is in state TIMED_WAITING
Print OK threads 360s.
Print OK threads 370s.
Print OK threads 380s.

[dani-e-l]

Can we help you to solve this problem?

[swankjesse]

We're making threading changes for the next release.
#5512

[swankjesse]

Follow the bug above for tracking.