Web Client throws OutOfMemoryError when response is larger than 1GB and contains UTF-16 characters

[mlichtblau]

We are observing OutOfMemoryErrors when the WebClient receives large responses containing characters that cannot be represented by Latin1 encoding.
This is especially critical, because the exception occurs here in decodeInternal before the data buffer is released and I suspect this is causing a memory leak.
I would expect a WebClientResponseException to be thrown and the data buffer to be released correctly.

Here is a demo that shows that WebClientResponseExceptions are thrown correctly for most cases (response larger 2GB, response larger maxInMemorySize), but not in the specific case where:

• response within maxInMemorySize
• AND response smaller Integer.MAX_VALUE
• AND response larger Integer.MAX_VALUE >> 1 (StringUTF16.MAX_LENGTH)
• AND response containing character that cannot be represented by Latin1.

response1500MBWithEmoji2000MBMaxMemoryThrowsOutOfMemoryError is the important test case.

If you have an idea where to throw a proper exception before attempting to create a String I would be happy to contribute.

[rstoyanchev]

Thanks for the report. It looks like we need to do the release in a finally:

T value;
try {
	value = decodeInternal(dataBuffer, charset);
}
finally {
	DataBufferUtils.release(dataBuffer);
}

[mlichtblau]

This makes sense in any case, I can open a PR for this.
Do you have a suggestions on where to throw an appropriate exception so a WebClientResponseException is thrown instead of OutOfMemoryError?

[rstoyanchev]

I'll await the PR, thanks. I didn't confirm but I expect an exception at the level of a decoder would be turned into a WebClientResponseException at the HttpMessageReader level.

[mlichtblau]

I believe the issue is that an OutOfMemoryError is deemed "fatal" by reactor (Docs):

Reactor, however, defines a set of exceptions (such as OutOfMemoryError) that are always deemed to be fatal. See the Exceptions.throwIfFatal method. These errors mean that Reactor cannot keep operating and are thrown rather than propagated.

I assume this is why no WebClientResponseException is thrown (as in this test case). I think the framework either needs to a) detect this situation before the exception is thrown and throw a different exception, or b) turn the OutOfMemoryError into a different exception that is not considered fatal by reactor.

Here is the stack trace of the OutOfMemoryError:

java.lang.OutOfMemoryError: UTF16 String size is 1576009739, should be less than 1073741823
	at java.base/java.lang.StringUTF16.newBytesFor(StringUTF16.java:50)
	at java.base/java.lang.String.<init>(String.java:577)
	at java.base/java.lang.String.<init>(String.java:522)
	at io.netty.buffer.ByteBufUtil.decodeString(ByteBufUtil.java:1371)
	at io.netty.buffer.AbstractByteBuf.toString(AbstractByteBuf.java:1246)
	at io.netty.buffer.AbstractByteBuf.toString(AbstractByteBuf.java:1241)
	at org.springframework.core.io.buffer.NettyDataBuffer.toString(NettyDataBuffer.java:334)
	at org.springframework.core.codec.StringDecoder.decodeInternal(StringDecoder.java:58)
	at org.springframework.core.codec.StringDecoder.decodeInternal(StringDecoder.java:43)
	at org.springframework.core.codec.AbstractCharSequenceDecoder.decode(AbstractCharSequenceDecoder.java:179)
	at org.springframework.core.codec.AbstractCharSequenceDecoder.decode(AbstractCharSequenceDecoder.java:50)
	at org.springframework.core.codec.AbstractDataBufferDecoder.decodeDataBuffer(AbstractDataBufferDecoder.java:108)
	at org.springframework.core.codec.AbstractDataBufferDecoder.lambda$decodeToMono$1(AbstractDataBufferDecoder.java:95)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)

What do you think about rethrowing a dedicated exception in one of NettyDataBuffer, StringDecoder, or AbstractCharSequenceDecoder to avoid it from bubbling up?

It's unfortunate that the StringUTF16 implementation throws an OutOfMemoryError...

[rstoyanchev]

Oh I see. I didn't realize the OOME was coming from String. I assumed it was some decoding issue, and the memory was exhaust was due the unreleased buffers.

Yes, we need to wrap it, and the releasing the buffer immediately. We have DataBufferLimitException that's thrown if we aggregate too much in the codec layer. I think it could be used for this, and it would carry the root cause.

[mlichtblau]

Sounds good, I will update the PR tomorrow.

[rstoyanchev]

Superseded by #35625.