Provide additional debug information for memory leaks via ByteBuf.touch [SPR-17427]

[spring-projects-issues]

Violeta Georgieva opened SPR-17427 and commented

Hi,

Netty provides the following API

io.netty.buffer.ByteBuf.touch()


    /**
     * Records the current access location of this object for debugging purposes.
     * If this object is determined to be leaked, the information recorded by this operation will be provided to you
     * via {@link ResourceLeakDetector}.  This method is a shortcut to {@link #touch(Object) touch(null)}.
     */
    ReferenceCounted touch();

When the implementation does additional transformations of the content or buffers this object, please use this touch() functionality. If there is a memory leak, when Netty dumps the stack traces of the components that accessed this object, it will be easier to find where the buffer was leaked.

Thanks,
Violeta

 

---

Affects: 5.1.1

[spring-projects-issues]

Rossen Stoyanchev commented

I've looked into this, and what I want to propose is to insert touch calls inside NettyDataBufferFactory and NettyDataBuffer. There we have access to the native ByteBuf without the need to downcast, and they capture all things that can happen to buffers (read, write, slice, collect, etc) in a general way.

The alternative of interspersing calls to touch above that layer, i.e. in the codecs, is not feasible I think. Take StringDecoder for example. It breaks up each input buffer along delimiters, then creates slices of chunks, buffers them up until each delimiter, then joins chunks into a stream of delimited Strings. For once it would be super noisy to insert calls for all this in StringDecoder, but it is also not possible to capture everything because joining goes into the DataBufferFactory where a composite is created. Likewise many decoders rely on DataBufferUtils#join which once again leads into the DataBufferFactory.

Based on my proposal the touch messages will be generic (e.g. "read", "write", "slice", etc) but since touch captures a stack trace that should show where the operation is coming from. In fact we might as well use the no-arg touch().

[spring-projects-issues]

Rossen Stoyanchev commented

I'm moving to 5.1.3 since we'll need time to agree and experiment with the outcome.

[spring-projects-issues]

Rossen Stoyanchev commented

I've tried inserting calls in operations of NettyDataBufferFactory and NettyDataBuffer, but after experimenting I realized that Netty already does this and I'm only duplicating records. For example AdvancedLeakAwareByteBuf intercepts and delegates every call to ResourceLeakTracker, that is unless "io.netty.leakDetection.acquireAndReleaseOnly" is set to true, which is false by default.

Given this I can't think of anything useful we can do by calling touch() that isn't already covered by the built-in record tracking. Our codecs either call operations on the input ByteBuf right away (which are tracked automatically) or we use buffering operators (e.g. bufferUntil, collectList) which will not release the input buffers anyway for garbage collection until finished, and on the next stage we'll operate on the buffer which again will leave records. So unless I'm missing something, in case of a leak we should already have all the records we can have, based on the way we operate on data buffers where we don't explicitly queue them ourselves.  

 

[rstoyanchev]

Adding a touch call here does make sense:

spring-framework/spring-core/src/main/java/org/springframework/core/io/buffer/NettyDataBufferFactory.java

Lines 92 to 94 in 57de2e0

	public NettyDataBuffer wrap(ByteBuf byteBuf) {
	return new NettyDataBuffer(byteBuf, this);
	}

because we're wrapping the buffer and keeping it.

The constructor however is not suitable because that's also called from places that slice or retain and those Netty already records, so there is no need for us to do it too.