Netty Epoll Bug occurs again (WorkAround Not working)

[prakash-premkumar]

I'm using Netty 4.1.29.Final.
Java Version 1.8.0_181-b13
Amazon AMI Linux 4.9.119-44.140.amzn1.x86_64 (2017.09)

I' using lafaspot imapnio client which uses netty. [(https://github.com/lafaspot/imapnio)]
I'm running my Java App in Tomcat 8.5.29.

I get the following exception while shutting down the tomcat server.
The NIO threads are not being killed, causing a memory leak.

20-Sep-2018 13:07:36.516 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [IMAPNIO-THREAD-POOL-CUSTOM-1-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:737)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

20-Sep-2018 13:07:36.517 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [IMAPNIO-THREAD-POOL-CUSTOM-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:737)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:392)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

This is the code I'm using to create and shutdown threads:

`public class IMAPClient {

/** instance id used for debug. */
private final String instanceId = Integer.toString(new Random(System.nanoTime()).nextInt());

/** counter for session. */
private AtomicInteger sessionCounter = new AtomicInteger(1);

/** The netty bootstrap. */
private final Bootstrap bootstrap;

/** Event loop group that will serve all channels for IMAP client. */
private final EventLoopGroup group;

/**
 * Constructs a NIO based IMAP client.
 *
 * @param threads number of threads to be used by IMAP client
 */
public IMAPClient(final int threads) {
    this.bootstrap = new Bootstrap();
    this.group = new NioEventLoopGroup(threads, new DefaultThreadFactory("IMAPNIO-THREAD-POOL-CUSTOM"));
    bootstrap.channel(NioSocketChannel.class);
    bootstrap.group(group);
}


/**
 * Close all of the sessions within a client, and shutdown the event group.
 */
public void shutdown() {
	Future<?> future = this.group.shutdownGracefully();
	try {
		
		if(future != null) {
			future.await();
		}
		boolean isTerminated = this.group.awaitTermination(300, TimeUnit.SECONDS);
		
	} catch (InterruptedException e) {
		
	}
   }

}`

I tried the workaround mentioned here https://netty.io/news/2012/09/13/4-0-0-alpha4.html

I added the -Dio.netty.epollBugWorkaround=true to JAVA_OPTS. But still I get this error.

Can you please tell me how to fix this error ?

[normanmaurer]

Are you sure the shutdown method of IMAPClient is really executed in this case ? It does not seems so.

[prakash-premkumar]

Thank you for the reply normanmaurer.

The shutdown method is executed. The isTerminated flag which the awaitTermination() function returns is true .
But the threads are not dying.
Is there a way to solve this problem?

[normanmaurer]

That’s strange... can you share a reproducer with me that I can run ?

…

Am 22.09.2018 um 10:23 schrieb Prakash Premkumar ***@***.***>: Thank you for the reply normanmaurer. The shutdown method is executed. The isTerminated flag which the awaitTermination() function returns true . But the threads are not dying. Is there a way to solve this problem? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[prakash-premkumar]

It's actually a web app where netty runs inside tomcat.
Which snippet of the code do you want to see ?

[normanmaurer]

I want you to create a small app that reproduces it that I can run myself

…

Am 22.09.2018 um 10:30 schrieb Prakash Premkumar ***@***.***>: May I know what you mean by reproducer ? It's actually a web app where netty runs inside tomcat. Which snippet of the code do you want to see ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[prakash-premkumar]

Okay. I will write it up and post it here !

[prakash-premkumar]

@normanmaurer , Unfortunately, I will not be able to share the reproducer with you, since it involves some proprietary code.

Tomcat is being shutdown (the process is being killed). Although, it complains of failure to stop threads. Will there be any memory leak in this case ?

May I know, what was the usual fix, when you encountered this error ? Is this a jdk bug ?
Is there a way to fix this problem ?

[prakash-premkumar]

@normanmaurer , Any updates on this ?

[normanmaurer]

@prakash-premkumar sorry without a way to reproduce here I can not really help. For me it looks like you not correctly shutdown all the EventLoopGroups.

[johnou]

-Dio.netty.epollBugWorkaround=true also has nothing to do with the stacktrace you pasted. How are you invoking shutdown()?

[prakash-premkumar]

@johnou, Thanks a lot for your reply.

I'm invoking shutdown in the contextDestroyed() of tomcat (which will be called when tomcat shuts down)

This is the shutdown method of the IMAPClient.

public void shutdown() {
		try {
			Future<?> future = this.group.shutdownGracefully().sync();
			if(future != null) {
				future.await();
			}
			boolean isTerminated = this.group.awaitTermination(60, TimeUnit.SECONDS);	
		} catch (InterruptedException e) {
			
		}
}

This is how I initialize the EventLoopGroup

public IMAPClient(final int threads) {
       this.bootstrap = new Bootstrap();
       this.group = new NioEventLoopGroup(threads, new DefaultThreadFactory("IMAPNIO-THREAD- 
        POOL-CUSTOM"));
       bootstrap.channel(NioSocketChannel.class);
       bootstrap.group(group);
   }

[johnou]

Are you sure that you are tracking all IMAPClient instances and invoking shutdown on all of them in contextDestroyed()?

[johnou]

It's important to note that if you are terminating the JVM there won't be a memory leak. The message you are seeing is only a problem if you are redeploying webapps without shutting down Tomcat.

[prakash-premkumar]

@johnou Yes, I'm invoking shutdown on all IMAPClient instances in contextDestroyed(). (And I'm using only one IMAPClient object in one app)

When I redeploy the webapp, I shutdown and restart tomcat, So you are saying that there wont be any memory leak in that case ? (My tomcat process is also getting killed when I shutdown ) Is that right ?

[normanmaurer]

Sorry but without a reproducer I can run I will not be able to help any further. As explained before I think you miss to call shutdown in some cases. Please re-open with more details if you still think there is a bug.

And yes if you restart tomcat there will be no leak