New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: cannot map pages in arena address space on Windows #8491

Open
gopherbot opened this Issue Aug 7, 2014 · 22 comments

Comments

Projects
None yet
9 participants
@gopherbot

gopherbot commented Aug 7, 2014

by kaixinlaoshu2008:

What does 'go version' print?
go 1.3 windows/386
What steps reproduce the problem?

file, handler, err := r.FormFile("file")
...
defer file.Close()

Use FormFile to receive documents.
When i upload a flie(100M) 4 or 5 times will get errors.

What happened?

fatal error: runtime: cannot map pages in arena address space

What should have happened instead?

go free memory after the FormFile receive a document.
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Aug 7, 2014

Comment 1:

I think we need a lot more information.  What is FormFile?  Can you show us a complete
program?  Have you tried looking at a memory profile (see
http://blog.golang.org/profiling-go-programs)?

Labels changed: added repo-main, release-none, os-windows.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Aug 7, 2014

Comment 2:

OP means net/http.Request.FormFile but I'm also curious to see actual code.

Status changed to WaitingForReply.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Aug 8, 2014

Comment 3:

The platform is windows/386. So heap is small.
Alex
@gopherbot

This comment has been minimized.

gopherbot commented Aug 12, 2014

Comment 4 by kaixinlaoshu2008:

my code:
//golang
package main
import (
    "html/template"
    "io"
    "log"
    "net/http"
    "os"
)
var uploadTemplate = template.Must(template.ParseFiles("page.html"))
func indexHandle(w http.ResponseWriter, r *http.Request) {
    if err := uploadTemplate.Execute(w, nil); err != nil {
        log.Fatal("Execute: ", err.Error())
        return
    }
}
func uploadHandle(w http.ResponseWriter, r *http.Request) {
    log.Println("method:", r.Method)
    if r.Method == "POST" {
        //128MB
        r.ParseMultipartForm(128 << 20)
        file, _, err := r.FormFile("file")
        if err != nil {
            log.Println(err)
            return
        }
        defer file.Close()
        path := "./"
        log.Println("create file")
        t, err := os.Create(path + "file")
        if err != nil {
            log.Println(err)
            return
        }
        defer t.Close()
        io.Copy(t, file)
        http.Redirect(w, r, "/", 302)
        return
    }
}
func main() {
    http.HandleFunc("/", indexHandle)
    http.HandleFunc("/upload", uploadHandle)
    http.ListenAndServe(":8080", nil)
}
//html
<html>
    <head>
        <title>Golang upload</title>
    </head>
    <body>
        <form id="uploadForm" method="POST" enctype="multipart/form-data" action="/upload">
            <p>Golang upload</p>
            <input type="file" id="file" name="file" />
            <input type="submit" value="upload">
        </form>
    </body>
</html>
@gopherbot

This comment has been minimized.

gopherbot commented Aug 14, 2014

Comment 5 by kaixinlaoshu2008:

who can help me?
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Aug 14, 2014

Comment 6:

kaixinlaoshu2008, I can run your program from #4 with no problems. How do I make it
crash? Thank you.
Alex
@gopherbot

This comment has been minimized.

gopherbot commented Aug 14, 2014

Comment 7 by kaixinlaoshu2008:

today i try go 1.3 windows/386 on win7(x64),the same error.
Keep uploading a file(120MB) for 4 times in a short time on my computer what will make
it crash.
@gopherbot

This comment has been minimized.

gopherbot commented Aug 14, 2014

Comment 8 by kaixinlaoshu2008:

today i try go 1.3.1 windows/386 on win7(x64),the same error.
@gopherbot

This comment has been minimized.

gopherbot commented Aug 14, 2014

Comment 9 by kaixinlaoshu2008:

today i try go 1.3.1 windows/386 on win7(x64),not "go 1.3 windows/386".
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Aug 14, 2014

Comment 10:

The problem is that your address space is quite small - 32-bit windows program have only
2GB address space. So Go cannot allocate a lot of memory. If you try to run
windows/amd64 (64-bit) version of Go, the limit is much high. So it should not crash as
quickly.
@Brad, in your opinion, how much memory should that server use under described
conditions? I don't have time to investigate it properly now. Thank you.
Alex
@gopherbot

This comment has been minimized.

gopherbot commented Aug 15, 2014

Comment 11 by kaixinlaoshu2008:

Immediately free memory when a file upload is complete,the problem will be resolved.
In fact,the program don't immediately free memory when a file upload is complete,and
memory goes up with the size of the uploaded files before crash.
Monitoring the changes of program's memory,i found the program don't free memory when a
file upload is complete and time has passed for 5 minutes or more.
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Aug 15, 2014

Comment 12:

kaixinlaoshu2008,
You use r.ParseMultipartForm in your program. It uses a lot of memory, because it parses
multi-part form in memory, and, if large file is part of that form, then
ParseMultipartForm will use as much memory as you've allowed. You have allowed to go up
to (128 << 20) bytes. So ParseMultipartForm tries to get as much memory as
possible on your computer.
You use windows/386 version of Go, so your total runtime heap is about 500M, so you
don't have much when you are playing with 300M files. You cannot increase heap, because
your OS allows for max of 2G virtual address space, and you need to fit many other
things apart from Go runtime heap.
I think your program behaves as expected. But I will let Brad to make that decision. He
wrote this code, so he might have better suggestions.
Sorry.
Alex

Status changed to New.

@gopherbot

This comment has been minimized.

gopherbot commented Aug 18, 2014

Comment 13 by kaixinlaoshu2008:

If the program immediately free memory,the runtime heap(500M) is enough.Because i am 
uploading a single file(size<120MB) in every time.
@gopherbot

This comment has been minimized.

gopherbot commented Aug 25, 2014

Comment 14 by kaixinlaoshu2008:

where is Brad?

@gopherbot gopherbot added new labels Aug 25, 2014

@bradfitz bradfitz removed the new label Dec 18, 2014

@meilihao

This comment has been minimized.

meilihao commented Mar 27, 2015

Today I tried again on go1.4.2 linux/amd64,but got the same problem.go does't free the uploaded file's memory.
ps:程序还是不会立即释放文件上传所使用过的内存,导致内存随文件上传而不断减少直至crash.

@minux

This comment has been minimized.

Member

minux commented Mar 27, 2015

@meilihao

This comment has been minimized.

meilihao commented Mar 27, 2015

It happens on ubuntu 14.04 x86_84/go1.4.2 linux/amd64 too.The program eats memory each time until crash when files are uploaded.

My core questions is "how to free memory immediately".

And my whole code is in this issue ,a go file and a html file.you can test it.

@randall77

This comment has been minimized.

Contributor

randall77 commented Mar 27, 2015

I looked into this a little bit (with 1.3.3). I don't see quite the behavior you are.

I tried uploading the same 90MB file 10+ times. The heap grows to about 500MB over the first three or four uploads and then stops. After I stop uploading, GC eventually gets back to <1MB. It takes two background cycles to get there because of the concurrent sweep. You can speed that up by calling runtime.GC(), although in your example program there really isn't a good place to do that (you'd need it inside ListenAndServe after the http.Request is dropped).

We should ideally be able to handle this workload in ~256MB. I'm not entirely sure why it gets to 500MB.

@meilihao

This comment has been minimized.

meilihao commented Mar 28, 2015

I am so sorry,my tests are not carefully enough on amd64.The program on amd64 is ok,not crash.

"r.ParseMultipartForm(128 << 20)" makes the heap grows to about 511MB and then stops.If not use it, the heap upper limit is 256MB. And the memory eventually gets back to <17MB after 9 Minutes.The long time is ok?

@rsc rsc removed the os-windows label Apr 10, 2015

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc added OS-Windows and removed release-none labels Apr 10, 2015

@mikioh

This comment has been minimized.

Contributor

mikioh commented May 7, 2015

@mikioh mikioh changed the title from runtime: cannot map pages in arena address space on Windows to runtime: fatal error: cannot map pages in arena address space on Windows May 7, 2015

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented May 7, 2015

@mikioh please create new issue instead. Thank you. I suspect some tests use too much memory for windows/386.

Alex

@mikioh

This comment has been minimized.

Contributor

mikioh commented May 7, 2015

Filed #10737.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment