Connection stay in CLOSE_WAIT

[wahisoufiane]

Hi,

I'm using netty 3.10.1, to communicate with devices (GPS), and in a certain time, connection are not closed (netty didn't call close connection) and leave CLOSE_WAIT status. Here is a wireshark capture for a case of it:

As you can see, the server didn't send FIN to device.

[windie]

Did you call close in your codes?

[wahisoufiane]

I did this code :

    @Override
    public void channelDisconnected(ChannelHandlerContext ctx, ChannelStateEvent e) {
        Log.info("Closing connection by disconnect");
        e.getChannel().close();
    }
    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) {
        Log.info("Closing connection by exception");
        e.getChannel().close();
    }
    @Override
    public void channelIdle(ChannelHandlerContext ctx, IdleStateEvent e) {
        Log.info("Closing connection by timeout");
        e.getChannel().close();
    }

[johnou]

@wahisoufiane how have you configured your IdleStateHandler?

[wahisoufiane]

of course I did it, I add in my pipeline :
pipeline.addLast("idleHandler", new IdleStateHandler(GlobalTimer.getTimer(), resetDelay, 0, 0));

and also, in the last pipe, I add a class who extends from IdleStateAwareChannelHandler it is where I call channelIdle(... like I said before.

[wahisoufiane]

Hi again,

I did a Log in netty to find out why it didn't close connection, and I found that timeout call fireChannelIdlebut it is like the runnable in pipeline are not called. (wich means channelIdleis not called)

[normanmaurer]

Not sure I understand correctly. Can you give more details?

[wahisoufiane]

    @Override
    public void channelClosed(ChannelHandlerContext ctx, ChannelStateEvent e)
            throws Exception {
        System.out.println("DESTROY || CTX : " + String.format("%08X", ctx.getChannel().getId()));
        destroy(ctx);
        ctx.sendUpstream(e);
    }

    private void fireChannelIdle(
            final ChannelHandlerContext ctx, final IdleState state, final long lastActivityTimeMillis) {
       ctx.getPipeline().execute(new Runnable() {

            public void run() {
                try {
                    channelIdle(ctx, state, lastActivityTimeMillis);
                } catch (Throwable t) {
                    fireExceptionCaught(ctx, t);
                }
            }
        });
    }

    protected void channelIdle(
            ChannelHandlerContext ctx, IdleState state, long lastActivityTimeMillis) throws Exception {
        System.out.println("IDLE FROM ISH || CTX : " + String.format("%08X", ctx.getChannel().getId()));
        ctx.sendUpstream(new DefaultIdleStateEvent(ctx.getChannel(), state, lastActivityTimeMillis));
    }

    private final class ReaderIdleTimeoutTask implements TimerTask {

        private final ChannelHandlerContext ctx;

        ReaderIdleTimeoutTask(ChannelHandlerContext ctx) {
            this.ctx = ctx;
        }

        public void run(Timeout timeout) throws Exception {
            if (timeout.isCancelled() || !ctx.getChannel().isOpen()) {
                return;
            }
            State state = (State) ctx.getAttachment();
            long currentTime = System.currentTimeMillis();
            long lastReadTime = state.lastReadTime;
            String chId = String.format("%08X", ctx.getChannel().getId());
            System.out.println("COUNTER || CTX : " + chId + " TMO : " + (currentTime - lastReadTime));
            long nextDelay = readerIdleTimeMillis - (currentTime - lastReadTime);
            if (nextDelay <= 0) {
                // Reader is idle - set a new timeout and notify the callback.
                System.out.println("TIMEOUT TRIGGER || CTX : " + chId);
                state.readerIdleTimeout =
                    timer.newTimeout(this, readerIdleTimeMillis, TimeUnit.MILLISECONDS);
                fireChannelIdle(ctx, IdleState.READER_IDLE, lastReadTime);
            } else {
                // Read occurred before the timeout - set a new timeout with shorter delay.
                state.readerIdleTimeout =
                    timer.newTimeout(this, nextDelay, TimeUnit.MILLISECONDS);
            }
        }
    }

and the log I get is:

COUNTER || CTX : 6CABCAE7 TMO : 60037
TIMEOUT TRIGGER || CTX : 6CABCAE7
COUNTER || CTX : 6CABCAE7 TMO : 120137
TIMEOUT TRIGGER || CTX : 6CABCAE7
COUNTER || CTX : 6CABCAE7 TMO : 180237
TIMEOUT TRIGGER || CTX : 6CABCAE7
COUNTER || CTX : 6CABCAE7 TMO : 240337
TIMEOUT TRIGGER || CTX : 6CABCAE7
COUNTER || CTX : 6CABCAE7 TMO : 300436
TIMEOUT TRIGGER || CTX : 6CABCAE7
COUNTER || CTX : 6CABCAE7 TMO : 360536
TIMEOUT TRIGGER || CTX : 6CABCAE7

knowing that the timeout I give is 60 second, and after that log I have CLOSE_WAIT status, till i stop my app. but in the normal case I have:

COUNTER || CTX : 6CABCAE7 TMO : 60037
TIMEOUT TRIGGER || CTX : 6CABCAE7
IDLE FROM ISH || CTX : 6CABCAE7

[wahisoufiane]

I went little deeper in the code, and found something that helped me but not sure that is the best solution, or if it would generate some problems somewhere.

I changed the AbstractNioChannelSink.execute(Runnable task) function from:

 @Override
    public ChannelFuture execute(ChannelPipeline pipeline, final Runnable task) {
        Channel ch = pipeline.getChannel();
        if (ch instanceof AbstractNioChannel<?>) {
            AbstractNioChannel<?> channel = (AbstractNioChannel<?>) ch;
            ChannelRunnableWrapper wrapper = new ChannelRunnableWrapper(pipeline.getChannel(), task);
            channel.worker.executeInIoThread(wrapper);
            return wrapper;
        }
        return super.execute(pipeline, task);
    }

To:

 @Override
    public ChannelFuture execute(ChannelPipeline pipeline, final Runnable task) {
        return super.execute(pipeline, task);
    }

Like it is not override, so it will execute the channelIdle immediatly.

I don't know why the IOThread is full and get the channelIdle thread waiting for some other things to be done, and didn't know what this things could be.

Another thing I found is that in the newer version of Netty, you don't create a thread in the IdleStateHandler on timeout, you call immediatly the channelIdle instead. Was that creating thread before created you a problem like in my case, that is why you change it now?

[normanmaurer]

Netty 3 is EOL. Please upgrade to 4.x