Memory leaks reported when using TestContainers as Spring beans.

[vpavic]

This was originally reported as spring-projects/spring-session#869.

We're using TestContainers as Spring beans in some integration testing scenarios, as explained in #369. The support for this use case was improved with the latest 1.7.1 release, however we're still experiencing some memory leak log warnings:

Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [ducttape-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [threadDeathWatcher-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.testcontainers.shaded.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-ryuk] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:127)
 org.testcontainers.utility.ResourceReaper$$Lambda$99/254873768.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)

[vpavic]

Hey @bsideup, I gave 1.8.0 and OkHttp transport a spin, the leak is still reported:

Jun 15, 2018 6:10:59 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [ducttape-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
Jun 15, 2018 6:10:59 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [OkHttp ConnectionPool] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:460)
 org.testcontainers.shaded.okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
Jun 15, 2018 6:10:59 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [Okio Watchdog] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 org.testcontainers.shaded.okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:347)
 org.testcontainers.shaded.okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:312)
Jun 15, 2018 6:10:59 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-ryuk] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:127)
 org.testcontainers.utility.ResourceReaper$$Lambda$109/571987491.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)

[vpavic]

There are still have leaks reported with 1.9.0-rc2. See here.

[kiview]

Okay, I see it is getting better and we might have more control about the remaining leaks with okhttp.
It is basically:

• ducttape-0
• OkHttp ConnectionPool
• Okio Watchdog
• testcontainers-ryuk

[bsideup]

@kiview that's weird because all listed threads are daemon threads

[kiview]

I digged a litte bit into this and it seems it's not related to the threads being daemons or not.
This Tomcat feature is documented here: MemoryLeakDetection

The interesting example and explanation is in section Threads spawned by webapps:

If a webapp creates a thread, by default its context classloader is set to the one of the parent thread (the thread that created the new thread). In a webapp, this parent thread is one of tomcat worker threads, whose context classloader is set to the webapp classloader when it executes webapp code.

Furthermore, the spawned thread may be executing (or blocked in) some code that involves classes loaded by the webapp, thus preventing the webapp classloader from being collected.

So, if the spawned thread is not properly terminated when the application is stopped, the webapp classloader will leak because of the strong reference held by the spawned thread.

The proposed hack/solution is setting leakingThread.setContextClassLoader(null).

Since this feature is basically about undeployment of webapps in a Tomcat server, I reallly wonder if it's worse digging deeper.

I checked if it's possible to disable leak detection in Tomcat and it seems, leak detection on JVM shutdown will be disabled starting with Tomcat 9.0.9 (see https://bz.apache.org/bugzilla/show_bug.cgi?id=50175).

@vpavic @bsideup WDYT, should we ignore this issue, or hack around the warning?

[kiview]

It's actually not enabled by default, it can be configured as a context attribute:
https://tomcat.apache.org/tomcat-9.0-doc/config/context.html

skipMemoryLeakChecksOnJvmShutdown

If true, Tomcat will not perform the usual memory leak checks when the web application is stopped if that web application is stopped as part of a JVM shutdown. If not specified, the default value of false will be used.

[kiview]

After some more discussions on spring-session side it seems, we should try to terminate those threads if possible.

Here is the method responsible for stopping the threads (and so detecting the memory leak):
https://apache.googlesource.com/tomcat70/+/trunk/java/org/apache/catalina/loader/WebappClassLoaderBase.java#2458

(not the most recent version of the code)