Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: HTTP large file upload fails with "bufio: buffer full" #26707

Closed
jvehent opened this issue Jul 31, 2018 · 9 comments

Comments

Projects
None yet
7 participants
@jvehent
Copy link

commented Jul 31, 2018

Uploading large files in HTTP multipart POST requests fails with multipart: NextPart: bufio: buffer full.

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ulfr/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ulfr"
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build508991886=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The following source code accepts uploads requests over HTTP. It works fine for files up to 15GB, but crashes for 20GB files with the error: multipart: NextPart: bufio: buffer full

package main

import (
        "log"
        "net/http"
)

func main() {
        http.HandleFunc("/upload", uploadHandler)
        http.ListenAndServe(":5050", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
        log.Printf("%s %s %s user-agent=%s", r.Method, r.Proto, r.URL.String(), r.UserAgent())
        if r.Method != http.MethodPost {
                http.Error(w, "only POST method is allowed", http.StatusBadRequest)
                return
        }
        log.Printf("parsing multipart form")
        // no more than 100MB of memory, the rest goes into /tmp
        r.ParseMultipartForm(100000000)
        file, handler, err := r.FormFile("uploadfile")
        if err != nil {
                http.Error(w, "failed to read uploadfile form field", http.StatusBadRequest)
                log.Printf("failed to read uploadfile form field: %v", err)
                return
        }
        defer file.Close()
        log.Println(handler.Filename)
}

Here's an example run with a 20GB file filed with zeroes.

$ go run multipart.go 
2018/07/31 08:29:18 POST HTTP/1.1 /upload user-agent=curl/7.59.0
2018/07/31 08:29:18 parsing multipart form
2018/07/31 08:29:43 failed to read uploadfile form field: multipart: NextPart: bufio: buffer full

upload with curl:

$ curl -vi -F uploadfile="@/home/ulfr/20gfile.bin" http://127.0.0.1:5050/upload
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5050 (#0)
> POST /upload HTTP/1.1
> Host: 127.0.0.1:5050
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Length: 20971520209
> Content-Type: multipart/form-data; boundary=------------------------879658a7528de349
> Expect: 100-continue
> 

< HTTP/1.1 100 Continue

< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Tue, 31 Jul 2018 12:29:43 GMT
< Content-Length: 37
< Connection: close
< 
failed to read uploadfile form field

* we are done reading and this is set to close, stop send
* Closing connection 0

Since my machine has 32GB of RAM, I tried boosting MAXMEMORY to 30GB by setting r.ParseMultipartForm(3000000000) but got the same results.

@ianlancetaylor ianlancetaylor changed the title HTTP large file upload fails with "bufio: buffer full" net/http: HTTP large file upload fails with "bufio: buffer full" Aug 1, 2018

@ianlancetaylor ianlancetaylor added this to the Go1.12 milestone Aug 1, 2018

@meirf

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2018

I wasn't able to reproduce this on go1.10.3 darwin/amd64.

@fraenkel

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2018

I did reproduce the error and then it magically went away after a few runs. I am trying to see if I can cause it to happen again.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 1, 2018

If anything, I suspect this is a "mime/multipart" bug and not a "net/http" bug.

@mvanotti

This comment has been minimized.

Copy link

commented Aug 2, 2018

Hi Julien!

I think the error is that you forgot to check the return value from ParseMultipartForm. When you call r.FormFile, it calls ParseMultiparForm again (because the form was not correctly parsed), and is now in an inconsistent state, making it fail.

It probably failed because mime/multipart/formdata.go writes to io.TempFile and you might have run out of space in your temp directory. The part that fails is the io.Copy writing the part body to the file.

The next time you call ParseMultipartForm, the buffer has already been partially consumed, and calls to NextPart fail to find the boundary.

I think after this kind of error (having a partially read buffer with some parts discarded), subsequent calls to ParseMultipartForm should fail immediately and with a clear error. Maybe ParseMultipartForm should support a directory where downloaded files should be stored.

@jvehent

This comment has been minimized.

Copy link
Author

commented Aug 2, 2018

Hey @mvanotti ! Long time no see!
I do indeed see an error on ParseMultipartForm due to low disk space: write /tmp/multipart-629374591: no space left on device, so this might well be the issue. I will try to reproduce on a different machine that has a larger /tmp directory and close this issue if possible. Thanks!

@jvehent

This comment has been minimized.

Copy link
Author

commented Aug 2, 2018

Confirmed on another system that the error I'm seeing is due to limited /tmp space, so this issue can be closed.

I'd be nice to not be limited by /tmp somehow, or at least be able to move that to a different directory, but the error message seems explicit enough (when not ignored, ah!) for the time being.

@jvehent jvehent closed this Aug 2, 2018

@mvanotti

This comment has been minimized.

Copy link

commented Aug 2, 2018

You can change the TempDir in linux by setting the TMPDIR environment variable (code link). It's going to be global to the whole program though.

If that's not OK, then setting the tmpfile folder for the multipart package would need to either allow to set it in the multipart.Reader struct or add a function something like ReadFormIntoFolder(dstFolder) which directly stores everything in a folder.

@mvanotti

This comment has been minimized.

Copy link

commented Aug 2, 2018

Regarding the error, I still think that the way ParseMultipartForm works is not OK. If that function fails (at least in this part), it doesn't make sense to allow it to be called again and make it do any work. It should fail immediately because there's nothing else it can do. Let's say that you have a multipart form with 2 parts, the first one is read completely but fails to be saved, then if you ignore the error, the next time you call it will give you just one part, without errors... Right? That seems like an error.

@iaburton

This comment has been minimized.

Copy link

commented Aug 3, 2018

Since ParseMultipartForm seems to be mostly a convenience method anyways, you could call MultipartReader and control the stream yourself, including where the data goes. Thats the more efficient solution anyways, and usually what I use for potentially large uploads (since you can stop it early and not have to download everything first).

I don't think the solution is to try and change where ParseMultipartForm puts data, but simply write a little more code using the great, already provided, alternative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.