net/http: io.Copy is throwing an error of runtime: out of memory

[rishiloyola]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.6.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/usr/lib/golang/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/golang"
GORACE=""
GOROOT="/usr/lib/golang"
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

I used io.pipe to solve this problem but still, it is not working fine. I have a machine with 1.8 GB ram. And I am trying to transfer the file of 1.6 GB.

Here is my code - https://github.com/vkuznet/transfer2go/blob/rishi-super/core/request.go#L74-L139

I want to transfer 1.6 GB file using HTTP protocol. But while reading the file chunk wise using io.pipe it is showing out of memory error.

[davecheney]

Can you please do the following

1. Upgrade to the latest Go 1.8 release or the soon to be released Go 1.9 release candidate and verify the bug stil exists.
2. Build your program using the -race flag and verify there are no reports of data races.
3. Include the full output from any runtime error message or stack trace.

[rishiloyola]

@davecheney I installed go1.8.3 version and compiled successfully using -race flag. But still, it is killing my process. Now it is just showing me this output - killed

[davecheney]

Please check dmesg, it will tell you if the oom killer chose to kill your process.

…

On Wed, 2 Aug 2017, 20:41 Rishi ***@***.***> wrote: @davecheney <https://github.com/davecheney> I installed go1.8.3 version and compiled successfully using -race flag. But still, it is killing my process. Now it is just showing me this output - killed — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21272 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA8AWflu0O0H1d8C9vgLebXrtD6m8ks5sUFJrgaJpZM4OqyOc> .

[davecheney]

Looking through your code I can see at least one place where the http response body is not being closed.

You stated earlier that you receive no message other than "killed" when you transfer a large file. How are you determining that the fault lies with Io.Copy?

[rishiloyola]

If I compile my code using -race flag then it will only show killed output. If I do compile without that then it is showing me the entire error.
Here is my error - https://gist.github.com/rishiloyola/287bf425041354eff03d6c12ce7874c0

I have added defer resp.Body.Close() in my code and then I retest it. Plus I am monitoring my process using top command. Suddenly it is taking 1.3 GB space of RAM.

The output of dmesg -

[1486504.349646] Out of memory: Kill process 4279 (transfer2go) score 857 or sacrifice child
[1486504.351559] Killed process 4279 (transfer2go) total-vm:3193268kB, anon-rss:1651520kB, file-rss:0kB, shmem-rss:0kB

By this, I am concluding that either there is a problem in my code or there is a bug in io package.

[davecheney]

The line numbers on your panic message do not match the code you provided above. Can you please check that these are accurate.

…

On Wed, Aug 2, 2017 at 10:02 PM, Rishi ***@***.***> wrote: If I compile my code using -race then it will only show killed output. If I do compile without that then it is showing me the entire error. Here is my error - https://gist.github.com/rishiloyola/ 287bf425041354eff03d6c12ce7874c0 I have added defer resp.Body.Close() in my code and then I retest it. Plus I am monitoring my process using top command. Suddenly it is taking 1.3 GB space of RAM. The output of dmesg - [1486504.349646] Out of memory: Kill process 4279 (transfer2go) score 857 or sacrifice child [1486504.351559] Killed process 4279 (transfer2go) total-vm:3193268kB, anon-rss:1651520kB, file-rss:0kB, shmem-rss:0kB By this, I am concluding that either there is a problem in my code or there is a bug in io package. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21272 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA9DiT3woWYjtG63XwM6oMErAGD4hks5sUGVMgaJpZM4OqyOc> .

[davecheney]

Specifically the code you linked above was from the logrus branch which uses io.Pipe as the target for the Multipart writer. The code from master, which matches the line numbers in the panic message uses a bytes.Buffer as the target for the multipart writer, https://github.com/vkuznet/transfer2go/blob/master/core/request.go#L81. This will cause the entire file to be buffered in ram, causing the OOM that you have found.

…

On Wed, Aug 2, 2017 at 10:09 PM, Dave Cheney ***@***.***> wrote: The line numbers on your panic message do not match the code you provided above. Can you please check that these are accurate. On Wed, Aug 2, 2017 at 10:02 PM, Rishi ***@***.***> wrote: > If I compile my code using -race then it will only show killed output. > If I do compile without that then it is showing me the entire error. > Here is my error - https://gist.github.com/rishil > oyola/287bf425041354eff03d6c12ce7874c0 > > I have added defer resp.Body.Close() in my code and then I retest it. > Plus I am monitoring my process using top command. Suddenly it is taking > 1.3 GB space of RAM. > > The output of dmesg - > > [1486504.349646] Out of memory: Kill process 4279 (transfer2go) score 857 or sacrifice child > [1486504.351559] Killed process 4279 (transfer2go) total-vm:3193268kB, anon-rss:1651520kB, file-rss:0kB, shmem-rss:0kB > > By this, I am concluding that either there is a problem in my code or > there is a bug in io package. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#21272 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AAAcA9DiT3woWYjtG63XwM6oMErAGD4hks5sUGVMgaJpZM4OqyOc> > . >

[bradfitz]

Questions about using Go should go to the mailing list. See https://golang.org/wiki/Questions.

There's no bug in Go here. It looks like your program has at least 5 paths where the http.Response.Body is not closed and your goroutine is not waited for and cleaned up regardless.

Let's discuss this on the mailing list.

If you want to open a bug for something, the better bug to open would be a feature request bug to make such multipart MIME documents easier to create. Currently multipart.Writer only works in terms of io.Writer, but a way to make multipart from io.Reader parts would be helpful.