Reference counting bug in GO_AWAY frame processing

[rschmitt]

Expected behavior

The server should be able to process a GO_AWAY frame from the client without throwing an IllegalReferenceCountException.

Actual behavior

I get an IllegalReferenceCountException, as described below.

Steps to reproduce

Here's what happens when the client (which in my case is nghttp) sends a GOAWAY frame, along with the refCnt of the ByteBuf instance at each step:

1. The ByteBuf comes in here containing a seventeen byte GO_AWAY frame; it is assigned to the cumulator field, which is initially null (as we are not currently in the middle of processing a frame). refCnt: 1.
2. The FrameListener creates an intermediate DefaultHttp2GoAwayFrame object and retains it. refCnt: 2.
3. The Http2MultiplexCodec calls onHttp2GoAwayFrame, which broadcasts the frame to all its child streams, followed by fireChannelRead to propagate the GO_AWAY through the rest of the parent channel.
   1. The onHttp2GoAwayFrame method calls release on the supplied frame. refCnt: 1.
   2. The fireChannelRead call will typically end up in the TailContext, which will release the object again in the onUnhandledInboundMessage method. refCnt: 0.
4. Finally, the ByteToMessageDecoder tries to call release on the cumulator field here, whose refCnt is already zero.

Minimal yet complete reproducer code (or URL to code)

I can attempt to write a repro case if desired. I hope that the above play-by-play will make the bug clear. My guess is that the onHttp2GoAwayFrame method should not be releasing the frame; @normanmaurer mentioned in another context that "all on* methods of this class will 'NOT' take ownership of the buffer."

Netty version

4.1.24

JVM version (e.g. java -version)

1.8.0_161

OS version (e.g. uname -a)

Darwin f40f243a5fab.ant.amazon.com 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64 x86_64

[normanmaurer]

@rschmitt you are right... I will come up with a PR shortly.

[normanmaurer]

@rschmitt PTAL #7894