Memory Wasting in Netty chanel read byeBuf

[lshmouse]

When testing the memory usage of spark shuffle which uses Netty for data communication, we found that the memory used by Netty usually are more than 2x of the size of data transferred. The reason is that the ByteBuf fired by the channel is no fully used. For an example, the capacity of the ByteBuf is 16K, but the size of readable bytes is 2K sometimes, 14K memory is wasted.

If the handler reuse the ByteBuf, the memory used by rpc can not be controlled accurately.

See: https://github.com/eBay/Spark/blob/master/network/common/src/main/java/org/apache/spark/network/util/TransportFrameDecoder.java#L61

As an newbie for Netty, I don't know if this behavior is expected and we need to handle this situation in the handler? Or any option to avoid this waste of memory.

[normanmaurer]

@lshmouse what netty version you are using ?

[radimch]

As far as I can see, they're using
Netty 4.0.29.Final
https://github.com/eBay/Spark/blob/master/pom.xml#L657

[normanmaurer]

Can you guys point me to the code where they bootstrap the server ?

[lshmouse]

@normanmaurer

The server is bootstrap at
https://github.com/eBay/Spark/blob/master/network/common/src/main/java/org/apache/spark/network/server/TransportServer.java#L88

and the pipeline is built at
https://github.com/eBay/Spark/blob/master/network/common/src/main/java/org/apache/spark/network/TransportContext.java#L131

[normanmaurer]

@lshmouse usually Netty uses a AdaptativeRecvByteBufAllocator which will adjust the used ByteBuf to match the ByteBuf to the data that is received. So do you always see 16k buffers or does it sometimes user different sizes ?

[lshmouse]

@normanmaurer
Yes, i saw the buffer size changed.

But according the code of AdaptiveRecvByteBufAllocator, the buffer size is adapted when the actual read bytes is less than a quarter of the current buffer size for two times continually. In the worst case, the memory used may be 8 times of the memory actually needed.

https://github.com/netty/netty/blob/83c349ffa94d3992c4ee511d3625afc0c97c12bb/transport/src/main/java/io/netty/channel/AdaptiveRecvByteBufAllocator.java

       @Override
        public void record(int actualReadBytes) {
            if (actualReadBytes <= SIZE_TABLE[Math.max(0, index - INDEX_DECREMENT - 1)]) {
                if (decreaseNow) {
                    index = Math.max(index - INDEX_DECREMENT, minIndex);
                    nextReceiveBufferSize = SIZE_TABLE[index];
                    decreaseNow = false;
                } else {
                    decreaseNow = true;
                }
            } else if (actualReadBytes >= nextReceiveBufferSize) {
                index = Math.min(index + INDEX_INCREMENT, maxIndex);
                nextReceiveBufferSize = SIZE_TABLE[index];
                decreaseNow = false;
            }
        }

[Apache9]

@normanmaurer In spark we may receive GBs data and keep them in memory, so this is a big problem. Now we can only consolidate the ByteBufs if we found that we have already wasted enough memory, otherwise we will be killed by the container...

Is there a way to let the handlers give a hint to the RecvByteBufAllocator? For example, LengthFieldBasedFrameDecoder, in the handler we can get the exact size of the message, no need to guess. And also, is there a way to reuse the ByteBuf which is already passed to the ChannelPipeline? For example, still a length prefixed decoder, without netty, we can have a ByteBuffer which fits the message exactly, and we can use this ByteBuffer multiple times to read if it is not full.

Thanks.

[normanmaurer]

@Apache9 You could write your own RecvByteBufAllocator for this ?

[Apache9]

@normanmaurer I do not think I can implement the above logic with the current RecvByteBufAllocator interface? There is no way that a handler can tell RecvByteBufAllocator what is the exact size...

[Scottmitch]

@Apache9 see RecvByteBufAllocator.guess() and RecvByteBufAllocator.allocate() ... these are what is used by Netty to allocate the buffer. Can you have a reference from your handler to your custom RecvByteBufAllocator to provide input to this process?

[normanmaurer]

I will close this for no.. Please re-open if you still think there is something needed to be changed in netty

[brucelwl]

This looks like the same problem, and it's fixed #9555