OutOfMemoryError when uploading large file with RestTemplate and RestClient

[ZIRAKrezovic]

Affects: 6.1.8

Edit: Originally reproduced in 6.1.6, but present in 6.1.8 which is latest version.

When uploading large file using multipart upload using either RestTemplate or RestClient, virtual machine will crash with out of memory error when InterceptingClientHttpRequestFactory wraps the Client's request factory.

• For RestTemplate, this happens when at least one interceptor is added
• For RestClient, even plain RestClient.create() results in usage of InterceptingClientHttpRequestFactory

I have narrowed issue to FormHttpMessageConverter::writeMultipart

		if (outputMessage instanceof StreamingHttpOutputMessage streamingOutputMessage) {
			streamingOutputMessage.setBody(outputStream -> {
				writeParts(outputStream, parts, boundary);
				writeEnd(outputStream, boundary);
			});
		}
		else {
			writeParts(outputMessage.getBody(), parts, boundary);
			writeEnd(outputMessage.getBody(), boundary);
		}

When running the code with debugger, and no interceptor exists in RestTemplate, the execution is as follows

As soon as one interceptor is added, the execution is as follows

The InterceptingClientHttpRequest wrapped by InterceptingClientHttpRequestFactory does not implement StreamingHttpOutputMessage, so the second path is used - which in turn ends up using FastByteArrayOutputStream.

However, the InterceptingClientHttpRequest does seem to have code to handle StreamingHttpOutputMessage itself.

The resulting error is:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at org.springframework.util.FastByteArrayOutputStream.addBuffer(FastByteArrayOutputStream.java:325)
	at org.springframework.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:126)
	at org.springframework.http.converter.FormHttpMessageConverter$MultipartOutputStream.write(FormHttpMessageConverter.java:683)
	at java.base/java.io.InputStream.transferTo(InputStream.java:797)
	at java.base/sun.nio.ch.ChannelInputStream.transferTo(ChannelInputStream.java:268)
	at org.springframework.http.converter.ResourceHttpMessageConverter.writeContent(ResourceHttpMessageConverter.java:151)
	at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:140)
	at org.springframework.http.converter.ResourceHttpMessageConverter.writeInternal(ResourceHttpMessageConverter.java:46)
	at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:235)
	at org.springframework.http.converter.FormHttpMessageConverter.writePart(FormHttpMessageConverter.java:549)

I can circumvent this issue for RestTemplate by clearing any interceptors. However, we utilize spring cloud load balancer, which is written as a Request Interceptor, which makes RestTemplate's requestFactory wrapped as InterceptingClientHttpRequestFactory.

I am not sure how to do this for rest client.

I have prepared a reproducer, by generating a file of ~360 MB. It is important to run the test with small heap size, i.e. -Xmx256m is enough. I have updated maven-surefire-plugin to set this - so as long as you use "./mvnw clean verify" the issue will be obvious. My IntelliJ IDEA test configuration is as follows

Examine the MultipartReproducerApplicationTests and comments on how to reproduce the wrong and right scenarios.

https://github.com/ZIRAKrezovic/spring-multipart-reproducer

[bclozel]

I think this is the expected behavior.
When using interceptors with RestTemplate or RestClient, the contract requires the entire request body to be provided as a byte array. This means the entire body must be held in memory at once. This explains why message converters cannot leverage the streaming use case.

If you wish to intercept requests in a streaming fashion, you should use WebClient with its ExchangeFilterFunction contract. If you don't need interceptors, removing those is also a possibility.

I can circumvent this issue for RestTemplate by clearing any interceptors. However, we utilize spring cloud load balancer, which is written as a Request Interceptor, which makes RestTemplate's requestFactory wrapped as InterceptingClientHttpRequestFactory.

I cannot speak for the spring cloud load balancer, you should reach out to the team about this use case.