WebFlux WebClient loading entire file into direct buffer memory during multipart upload

[matthewwilson]

Affects: 5.1.8-RELEASE

We have an endpoint that we can upload files using multipart and are using WebClient for our uploads.

Our upload client code looks like this

@Component
class UploadClient(
    private val client: WebClient,
) {
    suspend fun upload(filePath: String) =
        client.post()
              .uri("/upload")
              .contentType(MULTIPART_FORM_DATA)   
              .body(BodyInserters.fromMultipartData(generateMultipartBody(filePath)))
              .retrieve()
              .bodyToMono(UploadResult::class.java)
              .awaitFirst()

    private fun generateMultipartBody(filePath: String): MultiValueMap<String, HttpEntity<*>> {
        val builder = MultipartBodyBuilder()
        builder.part("file", FileSystemResource(filePath))
        return builder.build()
    }
}

However when we upload a large file, (1.6gb) we are seeing that this entire file is loaded into direct memory:

As the file is uploaded, the memory is released, then when the next file is uploaded you can see the spike in memory again.

For contrast I tried replacing WebClient with https://github.com/AsyncHttpClient/async-http-client and the memory usage is much lower, ~60mb per upload

[poutsma]

I am guessing that the collect(Collectors.toList()) on this line is the culprit, as it buffers the entire part into a list. @rstoyanchev, what do you think?

[poutsma]

After more investigation, it seems that the real culprit is the code that reads from the file (i.e. DataBufferUtils::readAsynchronousFileChannel, which currently does not consider back pressure and reads the entire file into buffers, whether there is demand or not.

I am working on a fix.