Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: io.Copy is throwing an error of runtime: out of memory #21272

Closed
rishiloyola opened this issue Aug 2, 2017 · 8 comments

Comments

Projects
None yet
4 participants
@rishiloyola
Copy link

commented Aug 2, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.6.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN="/usr/lib/golang/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/golang"
GORACE=""
GOROOT="/usr/lib/golang"
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

I used io.pipe to solve this problem but still, it is not working fine. I have a machine with 1.8 GB ram. And I am trying to transfer the file of 1.6 GB.

Here is my code - https://github.com/vkuznet/transfer2go/blob/rishi-super/core/request.go#L74-L139

I want to transfer 1.6 GB file using HTTP protocol. But while reading the file chunk wise using io.pipe it is showing out of memory error.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2017

Can you please do the following

  1. Upgrade to the latest Go 1.8 release or the soon to be released Go 1.9 release candidate and verify the bug stil exists.
  2. Build your program using the -race flag and verify there are no reports of data races.
  3. Include the full output from any runtime error message or stack trace.
@rishiloyola

This comment has been minimized.

Copy link
Author

commented Aug 2, 2017

@davecheney I installed go1.8.3 version and compiled successfully using -race flag. But still, it is killing my process. Now it is just showing me this output - killed

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2017

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2017

Looking through your code I can see at least one place where the http response body is not being closed.

You stated earlier that you receive no message other than "killed" when you transfer a large file. How are you determining that the fault lies with Io.Copy?

@rishiloyola

This comment has been minimized.

Copy link
Author

commented Aug 2, 2017

If I compile my code using -race flag then it will only show killed output. If I do compile without that then it is showing me the entire error.
Here is my error - https://gist.github.com/rishiloyola/287bf425041354eff03d6c12ce7874c0

I have added defer resp.Body.Close() in my code and then I retest it. Plus I am monitoring my process using top command. Suddenly it is taking 1.3 GB space of RAM.

The output of dmesg -

[1486504.349646] Out of memory: Kill process 4279 (transfer2go) score 857 or sacrifice child
[1486504.351559] Killed process 4279 (transfer2go) total-vm:3193268kB, anon-rss:1651520kB, file-rss:0kB, shmem-rss:0kB

By this, I am concluding that either there is a problem in my code or there is a bug in io package.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2017

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2017

@bradfitz

This comment has been minimized.

Copy link
Member

commented Aug 2, 2017

Questions about using Go should go to the mailing list. See https://golang.org/wiki/Questions.

There's no bug in Go here. It looks like your program has at least 5 paths where the http.Response.Body is not closed and your goroutine is not waited for and cleaned up regardless.

Let's discuss this on the mailing list.

If you want to open a bug for something, the better bug to open would be a feature request bug to make such multipart MIME documents easier to create. Currently multipart.Writer only works in terms of io.Writer, but a way to make multipart from io.Reader parts would be helpful.

@bradfitz bradfitz closed this Aug 2, 2017

@mikioh mikioh changed the title io.copy is throwing an error of runtime: out of memory net/http: io.Copy is throwing an error of runtime: out of memory Aug 2, 2017

@golang golang locked and limited conversation to collaborators Aug 2, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.