Possible memory leak in ReferenceCountedOpenSslClientContext#ExtendedTrustManagerVerifyCallback

[floating-cat]

I have a netty app, and I saw memory increased slowly when running. Then I dumped that app and found there is 2700+ ReferenceCountedOpenSslClientContext#ExtendedTrustManagerVerifyCallback instances in my app. The instances of this class increase gradually.

This line

netty/handler/src/main/java/io/netty/handler/ssl/ReferenceCountedOpenSslServerContext.java

Line 203 in 69db5bf

SSLContext.setCertVerifyCallback(ctx, new ExtendedTrustManagerVerifyCallback(

seems suspicious to me.
The setCertVerifyCallback is a native method here, and I guess there might be some leak here.

Minimal yet complete reproducer code (or URL to code)

I don't have an easy sample to reproduce it. If you need one, I could try to see whether I could create one for reproducing.

I use the following code in my app:

private SslHandler getHandler(Channel ch, ServerInfo serverInfo) {
  SslContextBuilder.forClient()
    .sslProvider(SslProvider.OPENSSL_REFCNT)
    .protocols("TLSv1.2", "TLSv1.3")
    .build()
    .newHandler(ch.alloc(), serverInfo.hostname.getHost, serverInfo.hostname.getPort)
}

  // add SSL handler to pipeline somewhere
  outChannel.pipeline().addLast(
    getHandler(outChannel, serverInfo))

From my understanding, I use a handler to warp the OPENSSL_REFCNT and put it in the pipeline. So the leak should not occur generally.
But sometimes I still can see this:

22:12:32.349 [epollEventLoopGroup-2-1] ERROR io.netty.util.ResourceLeakDetector - LEAK: ReferenceCountedOpenSslContext.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:205)
	io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:185)
	io.netty.handler.ssl.ReferenceCountedOpenSslClientContext.<init>(ReferenceCountedOpenSslClientContext.java:67)
	io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:833)
	io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:576)
	cl.monsoon.star.client.SslUtil$.handler(SslUtil.scala:17)
	cl.monsoon.star.client.ClientConnectionProxyHandler.onConnected(ClientConnectionHandler.scala:134)
	cl.monsoon.star.client.ClientConnectionHandler.$anonfun$channelRead0$1(ClientConnectionHandler.scala:30)
	io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
	io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96)
	cl.monsoon.star.DirectClientHandler.channelActive(DirectClientHandler.scala:10)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216)
	io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:620)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:653)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	java.base/java.lang.Thread.run(Thread.java:834)

I am not sure whether I do something somewhere.
Thanks in advance.

Netty version

netty 4.1.50
netty-transport-native-epoll linux-x86_64 4.1.50
netty-tcnative-boringssl-static 2.0.31

JVM version (e.g. java -version)

openjdk version "11.0.7" 2020-04-14

OS version (e.g. uname -a)

Arch Linux
Kernel: 5.7.7

edited:
Seems the ReferenceCountedOpenSslContext#DefaultOpenSslEngineMap is also leaked.

[normanmaurer]

That is expected... You are using SslProvider.OPENSSL_REFCNT which means you are responsible for calling release() once you don't need the ReferenceCountedOpenSslContext anymore. Most of the times people want to use SslProvider.OPENSSL and let the finaliser clean up stuff later on.

That said there is also another thing you should fix. Usually you should build one SslContext object and re-use it for all clients / servers.

[floating-cat]

I saw the below documentation in the SslContext source code so I misunderstood that I don't need to release anything manually:

     * Creates a new {@link SslHandler}.
     * <p>If {@link SslProvider#OPENSSL_REFCNT} is used then the returned {@link SslHandler} will release the engine
     * that is wrapped. If the returned {@link SslHandler} is not inserted into a pipeline then you may leak native
     * memory!

Really thanks for the reply and that SslContext reuse tip.