Netty Client concurrent issue when close FixedChannelPool

[caorong]

I use Netty as A tcp Client, use FixedChannelPool as channelPool.

when Close client，some time will cause follow exception

Expected behavior

no exception

Actual behavior

[xxxxxx]2018-04-03 14:34:56 04 [io.netty.util.concurrent.DefaultPromise.rejectedExecution]-[ERROR] Failed to submit a listener notification task. Event loop shut down?
 java.util.concurrent.RejectedExecutionException: event executor terminated

        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
        at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113)
        at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
        at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:1010)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:610)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:465)
        at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:972)
        at io.netty.channel.AbstractChannel.close(AbstractChannel.java:234)
        at io.netty.channel.pool.SimpleChannelPool.close(SimpleChannelPool.java:397)
        at io.netty.channel.pool.FixedChannelPool.access$1201(FixedChannelPool.java:38)
        at io.netty.channel.pool.FixedChannelPool$5.run(FixedChannelPool.java:458)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:745)

Steps to reproduce

my client code

public void init() {
    ioGroup = new NioEventLoopGroup(ioThreadCnt,
            new DefaultThreadFactory("mainstay-ioNioEventLoopGroup" + "-" + getUrl().getGroup()));
    
    bootstrap.group(ioGroup).channel(SocketChannel);
    bootstrap.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);
    bootstrap.option(ChannelOption.TCP_NODELAY, true);
    bootstrap.option(ChannelOption.SO_KEEPALIVE, true);
    bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, reqConnTimeout);
    bootstrap.option(ChannelOption.WRITE_BUFFER_WATER_MARK, new WriteBufferWaterMark(8 * 
        1024, 32 * 1024));
    bootstrap.remoteAddress(new InetSocketAddress(remoteIp, port));

    fixedChannelPool = new FixedChannelPool(bootstrap, nettyClientChannelPoolHandler, channelHealthChecker,
        FixedChannelPool.AcquireTimeoutAction.FAIL, 2000, 1000, 100,
        false,
        false);

}

public void close() {
    fixedChannelPool.close();
    if (ioGroup != null) {
        io.netty.util.concurrent.Future shutownfuture = ioGroup.shutdownGracefully(4, 32, TimeUnit.SECONDS);
        try {
          shutownfuture.get(64L, TimeUnit.SECONDS);
        } catch (Throwable e) {
          LOG.error("wait shutdown gracefully more than 64 second!", e);
        }
      }
}

my netty client runs well after init(), but when call close()，maybe cause rejectedExecution Exception， and this exception is not appear every time when call close().

This Exception happend probability is low ,may be lower than 5%

my solution

This is because, fixedChannelPool.close actually submit close Task to Threadpool， is
async.

so maybe ioGroup will be closed before.

If i add some sleep after fixedChannelPool.close will also reduce exception happend probability. but may still happened.

fixedChannelPool.close();
Thread.sleep(1500);
// ioGroup.close ...

how about fixedChannelPool.close return a Future， or make do not make it run in executor, make it run synchronized.

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.14.Final

JVM version (e.g. java -version)

java 1.6-1.8

OS version (e.g. uname -a)

Linux node2.live 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux