Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DefaultHttp2FrameWriter.writeGoAway leak #5850

Closed
Scottmitch opened this issue Sep 22, 2016 · 5 comments
Closed

DefaultHttp2FrameWriter.writeGoAway leak #5850

Scottmitch opened this issue Sep 22, 2016 · 5 comments
Labels

Comments

@Scottmitch
Copy link
Member

[21:12:40] :             [io.netty:netty-codec-http2] WARNING: 7 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
[21:12:40] :             [io.netty:netty-codec-http2] Recent access records: 5
[21:12:40] :             [io.netty:netty-codec-http2] #5:
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:955)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:109)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:261)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.local.LocalChannel.doWrite(LocalChannel.java:359)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:787)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:779)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:760)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:429)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[21:12:40] :             [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[21:12:40] :             [io.netty:netty-codec-http2] #4:
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:927)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:35)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.local.LocalChannel.doWrite(LocalChannel.java:358)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:787)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:779)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:760)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:429)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[21:12:40] :             [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[21:12:40] :             [io.netty:netty-codec-http2] #3:
[21:12:40] :             [io.netty:netty-codec-http2]   Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:821)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:734)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:357)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:694)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:741)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[21:12:40] :             [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[21:12:40] :             [io.netty:netty-codec-http2] #2:
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:179)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:356)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:694)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:741)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[21:12:40] :             [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[21:12:40] :             [io.netty:netty-codec-http2] #1:
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:178)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:356)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:694)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:741)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[21:12:40] :             [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[21:12:40] :             [io.netty:netty-codec-http2] Created at:
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:107)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:353)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:694)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:741)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[21:12:40] :             [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[21:12:40] :             [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
@Scottmitch
Copy link
Member Author

number of leak detector records increased in #5849 so the leak report will be more helpful in the future

@Scottmitch
Copy link
Member Author

Seen again while building PR #6009 on build https://garage.netty.io/teamcity/viewLog.html?buildId=13740&buildTypeId=pulls_netty&tab=buildLog

[18:08:23] :         [io.netty:netty-codec-http2] 18:08:23.572 [defaultEventLoopGroup-32-1] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
[18:08:23] :         [io.netty:netty-codec-http2] Recent access records: 12
[18:08:23] :         [io.netty:netty-codec-http2] #12:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:955)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:109)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:271)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.local.LocalChannel.doWrite(LocalChannel.java:361)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:787)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:779)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:760)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:429)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #11:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:927)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:35)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.local.LocalChannel.doWrite(LocalChannel.java:360)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:787)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:779)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:760)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:429)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #10:
[18:08:23] :         [io.netty:netty-codec-http2]   Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:821)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:734)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:362)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #9:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:199)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:361)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #8:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:198)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:361)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #7:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:197)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:361)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #6:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:196)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:361)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #5:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeInt(AdvancedLeakAwareByteBuf.java:561)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:360)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #4:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeInt(AdvancedLeakAwareByteBuf.java:561)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:235)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:359)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #3:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:234)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:359)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #2:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:543)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:233)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:359)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] #1:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AdvancedLeakAwareByteBuf.writeMedium(AdvancedLeakAwareByteBuf.java:555)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:232)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:359)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] Created at:
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:252)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:160)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:151)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.local.PreferHeapByteBufAllocator.buffer(PreferHeapByteBufAllocator.java:40)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:358)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:696)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:743)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:428)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:635)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:39)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:624)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   java.lang.Thread.run(Thread.java:745)
[18:08:23] :         [io.netty:netty-codec-http2] 18:08:23.599 [defaultEventLoopGroup-31-2] DEBUG i.n.h.c.http2.Http2ConnectionHandler - [id: 0x24f29b70, L:local:http2connectionroundtriptest - R:local:E:ea93fad5] Sent GOAWAY: lastStreamId '3', errorCode '2', debugData 'Fake Exception'. Forcing shutdown of the connection.
[18:08:23] :         [io.netty:netty-codec-http2] 18:08:23.606 [defaultEventLoopGroup-31-2] DEBUG io.netty.buffer.PoolThreadCache - Freed 5 thread-local buffer(s) from thread: defaultEventLoopGroup-31-2
[18:08:23] :         [io.netty:netty-codec-http2] 18:08:23.634 [defaultEventLoopGroup-32-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 4 thread-local buffer(s) from thread: defaultEventLoopGroup-32-1
[18:08:23] :         [io.netty:netty-codec-http2] 18:08:23.705 [defaultEventLoopGroup-34-1] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
[18:08:23] :         [io.netty:netty-codec-http2] io.netty.channel.ChannelPipelineException: io.netty.handler.codec.http2.Http2ConnectionRoundtripTest$13.handlerAdded() has thrown an exception; removed.
[18:08:23] :         [io.netty:netty-codec-http2]   at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:617)
[18:08:23] :         [io.netty:netty-codec-http2]   at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:44)
[18:08:23] :         [io.netty:netty-codec-http2]   at io.netty.channel.DefaultChannelPipeline$2.run(DefaultChannelPipeline.java:172)
[18:08:23] :         [io.netty:netty-codec-http2]   at io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[18:08:23] :         [io.netty:netty-codec-http2]   at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
[18:08:23] :         [io.netty:netty-codec-http2]   at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[18:08:23] :         [io.netty:netty-codec-http2]   at java.lang.Thread.run(Thread.java:745)

@Scottmitch
Copy link
Member Author

Scottmitch commented Nov 16, 2016

The above leak trace confirms that release responsibility is on LocalChannel and it actually made it into the peer's inboundQueue

@Scottmitch
Copy link
Member Author

Scottmitch commented Feb 20, 2017

Observed while building PR #6417 more details at https://garage.netty.io/teamcity/viewLog.html?buildId=17155&buildTypeId=pulls_netty&guest=1

[14:44:42] :		 [io.netty:netty-codec-http2] 14:44:42.581 [main] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
[14:44:42] :		 [io.netty:netty-codec-http2] Recent access records: 11
[14:44:42] :		 [io.netty:netty-codec-http2] #11:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:924)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:35)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.local.LocalChannel.doWrite(LocalChannel.java:361)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:856)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:823)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1296)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:447)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #10:
[14:44:42] :		 [io.netty:netty-codec-http2] 	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:810)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:367)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #9:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:216)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:366)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #8:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:215)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:366)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #7:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:214)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:366)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #6:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeUnsignedInt(Http2CodecUtil.java:213)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:366)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #5:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeInt(AdvancedLeakAwareByteBuf.java:558)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:365)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #4:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeInt(AdvancedLeakAwareByteBuf.java:558)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:274)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:364)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #3:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:273)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:364)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #2:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeByte(AdvancedLeakAwareByteBuf.java:540)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:272)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:364)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] #1:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AdvancedLeakAwareByteBuf.writeMedium(AdvancedLeakAwareByteBuf.java:552)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:271)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:364)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)
[14:44:42] :		 [io.netty:netty-codec-http2] Created at:
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:237)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:308)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:162)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:153)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.local.PreferHeapByteBufAllocator.buffer(PreferHeapByteBufAllocator.java:40)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeGoAway(DefaultHttp2FrameWriter.java:363)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:770)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.goAway(Http2ConnectionHandler.java:817)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.handler.codec.http2.Http2ConnectionHandler.close(Http2ConnectionHandler.java:446)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:624)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:38)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:613)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.channel.DefaultEventLoop.run(DefaultEventLoop.java:54)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
[14:44:42] :		 [io.netty:netty-codec-http2] 	io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[14:44:42] :		 [io.netty:netty-codec-http2] 	java.lang.Thread.run(Thread.java:745)

Scottmitch added a commit to Scottmitch/netty that referenced this issue Feb 20, 2017
Motivation:
Some unit HTTP/2 unit tests use LocalChannel. LocalChannel's doClose method will ensure any pending items in the queue will be released, but it may execute a Runnable on the peer's EventLoop to ensure the peer's queue is also cleaned up. The HTTP/2 unit tests close the event loop groups with no wait time so that unit tests will execute quickly, but if the doClose Runnable is in the EventLoop's queue it will not run and thus the items in the queue will not be released.

Modifications:
- Ensure all HTTP/2 unit tests which use LocalChannel wait for both client and server channels to be closed before closing the EventLoop.

Result:
Related to netty#5850.
Scottmitch added a commit that referenced this issue Feb 20, 2017
Motivation:
Some unit HTTP/2 unit tests use LocalChannel. LocalChannel's doClose method will ensure any pending items in the queue will be released, but it may execute a Runnable on the peer's EventLoop to ensure the peer's queue is also cleaned up. The HTTP/2 unit tests close the event loop groups with no wait time so that unit tests will execute quickly, but if the doClose Runnable is in the EventLoop's queue it will not run and thus the items in the queue will not be released.

Modifications:
- Ensure all HTTP/2 unit tests which use LocalChannel wait for both client and server channels to be closed before closing the EventLoop.

Result:
Related to #5850.
@Scottmitch
Copy link
Member Author

PR #6422 hopefully addressed this. We can re-open later if not. We will have to be careful in other unit tests which have the same behavior (closing the EventLoop immediately without waiting to drain tasks).

liuzhengyang pushed a commit to liuzhengyang/netty that referenced this issue Sep 10, 2017
Motivation:
Some unit HTTP/2 unit tests use LocalChannel. LocalChannel's doClose method will ensure any pending items in the queue will be released, but it may execute a Runnable on the peer's EventLoop to ensure the peer's queue is also cleaned up. The HTTP/2 unit tests close the event loop groups with no wait time so that unit tests will execute quickly, but if the doClose Runnable is in the EventLoop's queue it will not run and thus the items in the queue will not be released.

Modifications:
- Ensure all HTTP/2 unit tests which use LocalChannel wait for both client and server channels to be closed before closing the EventLoop.

Result:
Related to netty#5850.
kiril-me pushed a commit to kiril-me/netty that referenced this issue Feb 28, 2018
Motivation:
Some unit HTTP/2 unit tests use LocalChannel. LocalChannel's doClose method will ensure any pending items in the queue will be released, but it may execute a Runnable on the peer's EventLoop to ensure the peer's queue is also cleaned up. The HTTP/2 unit tests close the event loop groups with no wait time so that unit tests will execute quickly, but if the doClose Runnable is in the EventLoop's queue it will not run and thus the items in the queue will not be released.

Modifications:
- Ensure all HTTP/2 unit tests which use LocalChannel wait for both client and server channels to be closed before closing the EventLoop.

Result:
Related to netty#5850.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant