WebFlux temporary file not always deleted with parallel uploads.

[davefarrelly]

I am working on a project using spring-webflux:6.0.8 and I have noticed an issue where temporary files created during a FilePart upload are not always getting deleted when uploads are done in parallel.

However, if I do slow sequential uploads, the temporary files are getting deleted consistently as expected.

I have a reproducer with a single endpoint that receives a FilePart. Sending ~30 requests in parallel with with 28MB file seems to reproduce the issue consistently.

spring-tempfile-reproducer.zip

[rstoyanchev]

I'm not able to reproduce the issue. There may be something more specific here that depends on how the client creates and uses connections. Can you share the client code as well?

This is what I used:

public class ClientApp {

    public static void main(String[] args) {
        FileSystemResource resource = new FileSystemResource("..."); // 30 MB file
        WebClient client = WebClient.create();
        Flux.range(1, 50)
                .flatMap(aLong -> client.post()
                        .uri("http://localhost:9890/upload")
                        .body(BodyInserters.fromMultipartData("uploadedFile", resource))
                        .retrieve()
                        .toBodilessEntity())
                .blockLast();
    }

}

[rstoyanchev]

Note that it's worth trying with 6.0.10 where there was a related change that could have an impact on this, see c1fe571.

[davefarrelly]

@rstoyanchev thanks for the response!

I was actually using JMeter with the "bzm - Parallel Controller" plugin to make the requests.

I will test against our real application with a client similar to what you provided and also try 6.0.10 and get back to you with the results.

[davefarrelly]

@rstoyanchev quick update, I tried updating to 6.0.10 and surprisingly none of the temporary files are getting deleted on this version. (Note. this is still using JMeter). Is there any config changes or anything needed with this version?

[davefarrelly]

Tested using the client code provided instead of JMeter and I am still seeing the same results:

6.0.8: Couple of temporary files remain on disk
6.0.10: All 50 temporary files remain on disk

[rstoyanchev]

Thanks for the additional info. I should have mentioned, might as well use the latest 6.0.13 (instead of 6.0.10), just in case. Also if you get a chance to run my sample code above, to confirm whether it reproduces it for you or not.

[davefarrelly]

Yeah I tested 6.0.13 and seems to be behaving the same as 6.0.10, none of the files are getting deleted.

I ran the sample you gave above and results were the same on my end 🤔

[davefarrelly]

Hi @rstoyanchev @poutsma I have tried updating to Spring Boot 3.2.0 released today (Spring Framework 6.1.1) and I am still seeing this issue. Where a couple of files (~2 out of 50) are not getting cleaned up.

[poutsma]

Hi @davefarrelly, we are aware that the issue remains in 6.1.1, and are still investigating.

[AlexejTimonin]

A tip to anyone experiencing temp folder filling up with zombie temp files due to this issue. Make sure to call delete on the FilePart when you're done working with it.

Given this cURL from 1 terminal session (You could call it MacGyver load testing), the file is 280kb curl manual:

while true; do curl -v -F uploadedFile=@curl-man.txt http://localhost:8080/upload; done

And this controller:

    @PostMapping
    public Mono<String> uploadFile(@RequestPart("uploadedFile") FilePart uploadedFile) {
        var newFile = Path.of("c:/test", UUID.randomUUID().toString());
        return uploadedFile.transferTo(newFile)
                .then(Mono.just(uploadedFile.filename()));
    }

The Temp folder will start to fill up with zombie temp files which are not deleted after about 30 seconds of curl spam:

To fix it, call delete on the FilePart when you are done:

    @PostMapping
    public Mono<String> uploadFile(@RequestPart("uploadedFile") FilePart uploadedFile) {
        var newFile = Path.of("c:/test", UUID.randomUUID().toString());
        return uploadedFile.transferTo(newFile)
                .then(uploadedFile.delete())
                .then(Mono.just(uploadedFile.filename()));
    }

With the controller above, I've ran curl from 2 sessions for 1 minute with no zombie Temp files detected.

Edit: Version - 6.1.4

[simonbasle]

@davefarrelly I think this has finally been resolved. The fix should be available soon in a snapshot and I've also backported it to the 6.0.x branch.

If you have an opportunity to test out the snapshot, let us know. thanks.