mime/multipart: TempFile file hangs around on disk after usage in multipart/formdata.go

[nilslice]

Hey Go team,

I'd like to know if this is a bug or expected behavior. If it is a bug, I would like to write a patch to fix it.

---

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8 linux/amd64

What operating system and processor architecture are you using (go env)?

GOOS="linux"
GOARCH="amd64"

What did you do?

When uploading a file to a Go HTTP server, the call to req.ParseMultipartForm() subsequently calls readForm() and if the file size exceeds the maxMemory setting it creates a file on disk using the ioutil.TempFile() func, but the resulting file is never removed by the Go program. Unix behavior aside, based on the documentation for ioutil.TempFile(), the caller should remove the file once it is done using it. The file is only ever removed from the Go program if the io.Copy returns a non-nil error.

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.

I would need to show a lot of code to demonstrate this, but you can see in the Go src here how the file is never removed: https://github.com/golang/go/blob/master/src/mime/multipart/formdata.go#L78

A link on play.golang.org is best.

What did you expect to see?

I expect that the temp file is removed after the file is copied to it's designated location on disk.

What did you see instead?

my /tmp directory is littered with multipart-XXXXXXXX files

[bradfitz]

When you're done with the multipart files, you can call:
https://golang.org/pkg/mime/multipart/#Form.RemoveAll

... which is accessible here:
https://golang.org/pkg/net/http/#Request.MultipartForm

But, yeah, maybe this could be more automatic. Maybe the multipart.File's concrete type could delete-on-Close.

Feel free to work on it!

[bradfitz]

When you're done with the multipart files, you can call:
https://golang.org/pkg/mime/multipart/#Form.RemoveAll

... which is accessible here:
https://golang.org/pkg/net/http/#Request.MultipartForm

But, yeah, maybe this could be more automatic. Maybe the multipart.File's concrete type could delete-on-Close.

Feel free to work on it!

[nilslice]

Thanks for the suggestions! I wasn't aware of Form.RemoveAll - good to know.

From my perspective, the complete process of dealing with HTML forms / other HTTP clients uploading files to my server shouldn't involve removing the temp files w/ RemoveAll. Based on the UX of a typical Go programmer, automatically cleaning up those files doesn't feel overly magical to me.

I'll work on it and submit if something clicks.

[nilslice]

Thanks for the suggestions! I wasn't aware of Form.RemoveAll - good to know.

From my perspective, the complete process of dealing with HTML forms / other HTTP clients uploading files to my server shouldn't involve removing the temp files w/ RemoveAll. Based on the UX of a typical Go programmer, automatically cleaning up those files doesn't feel overly magical to me.

I'll work on it and submit if something clicks.

[kennygrant]

@nilslice still working on this?

We were hit by a similar problem (have small servers accepting file uploads, and tmp files filled the disk at times if not flushed regularly), and I too was unaware of Form.RemoveAll, but delete on close would be even better.

Happy to look at it if you're busy or leave it to you if you'd like to look at it.

[kennygrant]

@nilslice still working on this?

We were hit by a similar problem (have small servers accepting file uploads, and tmp files filled the disk at times if not flushed regularly), and I too was unaware of Form.RemoveAll, but delete on close would be even better.

Happy to look at it if you're busy or leave it to you if you'd like to look at it.

[nilslice]

@kennygrant - I have a partial fix, but have been swamped with other work. If it's imperative to your project that it's patched, go for it! Otherwise, I'd still like to finish it. But no problem whatsoever if you want to get the patch in.

[nilslice]

@kennygrant - I have a partial fix, but have been swamped with other work. If it's imperative to your project that it's patched, go for it! Otherwise, I'd still like to finish it. But no problem whatsoever if you want to get the patch in.

[kennygrant]

That's ok, I'll leave it to you if you have started. We have a workaround for now.

[kennygrant]

That's ok, I'll leave it to you if you have started. We have a workaround for now.

[Kagami]

But it's removed at the end of request?

https://github.com/golang/go/blob/release-branch.go1.10/src/net/http/server.go#L1553-L1555

[Kagami]

But it's removed at the end of request?

https://github.com/golang/go/blob/release-branch.go1.10/src/net/http/server.go#L1553-L1555

[nilslice]

@Kagami - it does appear to make the call, but the last time I tested it the files were still on disk.

The blame shows that the code has been in the tree for ~7 years, well before this issue was originally filed: https://github.com/golang/go/blame/678dede7bc22a7cccfe78c42711478a9b1ecf4d3/src/net/http/server.go#L1553-L1555

Have you tried this to verify that the files are cleaned up after the request?

I'm trying to get my CL out this month or early next (taking a couple days off from work to do long put-off open source tasks I have queued up!).

[nilslice]

@Kagami - it does appear to make the call, but the last time I tested it the files were still on disk.

The blame shows that the code has been in the tree for ~7 years, well before this issue was originally filed: https://github.com/golang/go/blame/678dede7bc22a7cccfe78c42711478a9b1ecf4d3/src/net/http/server.go#L1553-L1555

Have you tried this to verify that the files are cleaned up after the request?

I'm trying to get my CL out this month or early next (taking a couple days off from work to do long put-off open source tasks I have queued up!).

[gopherbot]

Change https://golang.org/cl/113055 mentions this issue: mime/multipart: properly remove temp files in case of error

[gopherbot]

Change https://golang.org/cl/113055 mentions this issue: mime/multipart: properly remove temp files in case of error

[lpar]

If this won't be fully fixed in 1.10, how about mentioning the need to call RemoveAll in the documentation for ParseMultipartForm?

[lpar]

If this won't be fully fixed in 1.10, how about mentioning the need to call RemoveAll in the documentation for ParseMultipartForm?

[dvdscripter]

I'm no expert but you should be able to remove files without closing them?

...
file, _, err := r.FormFile("file")
if err != nil {
	return err
}
defer file.Close()
...

Leaves no "multipart*" files behind.

[dvdscripter]

I'm no expert but you should be able to remove files without closing them?

...
file, _, err := r.FormFile("file")
if err != nil {
	return err
}
defer file.Close()
...

Leaves no "multipart*" files behind.