IllegalStateException handling multi-part form files

[pmlopes]

@woollybah commented on Tue Oct 18 2016

We have a client that posts a multi-part form containing 3 file attachments to our Linux vert.x 3.3.3 HTTPServer.
The client uses keep-alive to perform a sequence of uploads over a single connection.

In the vast majority of uploads, everything performs as expected.

However, occasionally we get an IllegalStateException within the internal code that handles the form file attachments :

Oct 17, 2016 1:39:32 PM io.vertx.core.impl.ContextImpl
SEVERE: Unhandled exception
java.lang.IllegalStateException: File handle is closed
        at io.vertx.core.file.impl.AsyncFileImpl.checkClosed(AsyncFileImpl.java:450)
        at io.vertx.core.file.impl.AsyncFileImpl.check(AsyncFileImpl.java:445)
        at io.vertx.core.file.impl.AsyncFileImpl.closeInternal(AsyncFileImpl.java:474)
        at io.vertx.core.file.impl.AsyncFileImpl.close(AsyncFileImpl.java:117)
        at io.vertx.core.http.impl.HttpServerFileUploadImpl.handleComplete(HttpServerFileUploadImpl.java
        at io.vertx.core.http.impl.HttpServerFileUploadImpl.resume(HttpServerFileUploadImpl.java:128)
        at io.vertx.core.http.impl.HttpServerFileUploadImpl.resume(HttpServerFileUploadImpl.java:38)
        at io.vertx.core.streams.impl.PumpImpl.lambda$new$0(PumpImpl.java:62)
        at io.vertx.core.file.impl.AsyncFileImpl.checkDrained(AsyncFileImpl.java:285)
        at io.vertx.core.file.impl.AsyncFileImpl.lambda$doWrite$0(AsyncFileImpl.java:151)
        at io.vertx.core.file.impl.AsyncFileImpl$1.lambda$completed$0(AsyncFileImpl.java:394)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:316)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
        at java.lang.Thread.run(Thread.java:745)

This failure occurs before our routing context handler is called, and therefore we appear unable to catch the exception and handle it better (like failing the context, for instance). Instead, the request hangs until the idleTimeout kicks in, and the request fails.

Looking in the "file-uploads" folder, the three files are present, and appear to be complete. And obviously, the files remain there (i.e. not cleaned up) as the process managing them has failed.

The error itself implies that an AsyncFile is being closed twice!

The issue only appears with one particular client of ours. All our other clients have not exhibited this behaviour.

Is there any way for us to catch and handle exceptions from there?

@woollybah commented on Thu Oct 20 2016

Here's a reproducer : https://github.com/woollybah/multipart-upload-reproducer

Running the test generates the exception consistently here. Out in the wilderness, the exception occurs fairly frequently (several times a day), which is somewhat of an issue for us.

[pmlopes]

After investigation the issue is not related to delete files using the reproducer with delete files as true or false renders the same behaviour.

Also from the stack trace it shows that the exception is being bubbled up from the core.

[simonext]

Hello,
I encounter exactly the same problem with the xml file upload.
I use an HTML5 form = "fyle".
On the Vertx side, I use routingContext.fileUploads ().

Thank you in advance I hope you would find the solution.

[romanpa]

Hi

Sometimes I have the same issue when uploading files. Usually it happens in tests and i don't know how to reproduce it. But succeed to catch it once.

May be this will help to investigate this issue:
The 'long' stack is when all works OK, the 'short' stack is when there is an Exception.
It looks like problem comes from Netty

This is a call-stack that I see when everything is OK

This is call-stack when there is a problem:

[vietj]

I've done a fix for Vert.x core here #1770 that fixes the reproducer provided by @woollybah

[vietj]

I'm closing this issue, please confirm with fix #1770 , if there is still an issue, reopen this one.

[vietj]

marked as invalid as fixed in another issue