用SaveToFile文件上传后，一直占着虚拟内存没有释放 #42

mrcuix · 2013-04-20T20:29:46Z

发现在windows中用SaveToFile保存上传后的文件，一直占着虚拟内存没有释放，再上传多几次C盘就满了
有什么方法比较优化的呢？
代码如下：

package main

import (
    "github.com/astaxie/beego"
)

type UploadController struct {
    beego.Controller
}
func (this *UploadController) Post() {
    _, h, _ := this.GetFile("uploadfile")
    this.SaveToFile("uploadfile", "./static/files/"+h.Filename)
    this.Ctx.Redirect(302, "/upload")

}

func main() {
    beego.Router("/upload", &UploadController{})
    beego.Run()
}

The text was updated successfully, but these errors were encountered:

astaxie · 2013-04-21T02:13:48Z

我想说关于虚拟内存请看这个http://golang.org/doc/faq#Why_does_my_Go_process_use_so_much_virtual_memory

这个和beego的保存文件好像没啥关系。

还有你说的C盘爆满是什么意思？

astaxie · 2013-04-23T07:39:06Z

如果你有什么问题重新打开这个issue吧，给你发了也没有反馈

mrcuix · 2013-04-23T15:07:18Z

我发现当我用原生的来写是可以马上释放 C盘的临时文件空间的

import (
    "crypto/md5"
    "fmt"
    "html/template"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

func upload(w http.ResponseWriter, r *http.Request) {
    fmt.Println("method:", r.Method) //获取请求的方法
    if r.Method == "GET" {
        crutime := time.Now().Unix()
        h := md5.New()
        io.WriteString(h, strconv.FormatInt(crutime, 10))
        token := fmt.Sprintf("%x", h.Sum(nil))

        t, _ := template.ParseFiles("up.tpl")
        t.Execute(w, token)
        // t.Execute(w, nil)
    } else {
        r.ParseMultipartForm(32 << 20)
        file, handler, err := r.FormFile("uploadfile")
        if err != nil {
            fmt.Println(err)
            return
        }
        defer file.Close()
        fmt.Fprintf(w, "%v", handler.Header)
        f, err := os.OpenFile("./static/files/"+handler.Filename, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
        if err != nil {
            fmt.Println(err)
            return
        }
        defer f.Close()
        io.Copy(f, file)
    }
}

func main() {

    http.HandleFunc("/upload", upload)
    err := http.ListenAndServe(":8080", nil)
    if err != nil {
        log.Fatal("listenandserver: ", err)
    }

}

astaxie · 2013-04-23T15:48:09Z

C盘的临时空间是指什么？我没在window下面测试过呢
还有我看了你的代码和我处理的应该是一模一样的，唯一不一样就是我的
MaxMemory = 1 << 26 //64MB 比你大一倍

mrcuix · 2013-04-23T15:51:36Z

就是C盘的空间啊，一般上传会占用一定的空间，但上传完会马上释放的，现在用beego是不会释放，用原生的写是可以释放

astaxie · 2013-04-23T15:54:22Z

内存空间？还是什么？你怎么查看的啊？

mrcuix · 2013-04-23T15:57:15Z

就临时文件啊，直接实时刷新看C盘的剩余容量，而且我需要用360安全卫士才能释放一下空间，也不是每次都能释放到

mrcuix · 2013-04-23T16:21:03Z

经过我刚才测试发现以下代码是可以马上释放 C盘占用了的空间的

test.go 文件

package main

import (
    "fmt"
    "github.com/astaxie/beego"
    "io"
    "os"
)

type UploadController struct {
    beego.Controller
}


func (this *UploadController) Get() {
    this.TplNames = "upload.tpl"
}

func (this *UploadController) Post() {

    this.Ctx.Request.ParseMultipartForm(32 << 20)
    file, handler, err := this.GetFile("uploadfile")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()
    f, err := os.OpenFile("./static/files/"+handler.Filename, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
    if err != nil {
        fmt.Println(err)
        return
    }
    defer f.Close()
    io.Copy(f, file)

    this.Ctx.Redirect(302, "/upload")

}

func main() {
    beego.Router("/upload", &UploadController{})
    beego.Run()
}

app.conf 文件

autorender = true
autorecover = false

upload.tpl 文件

<html>
<head>
    <title>上传文件</title>
</head>
<body>

<form enctype="multipart/form-data" action="/upload" method="post">
  <input type="file" name="uploadfile" />
  <input type="submit" value="upload" />
</form>

</body>
</html>

astaxie · 2013-04-23T16:23:12Z

我在想是不是和你掉用了两次这个有关系

_, h, _ := this.GetFile("uploadfile")
this.SaveToFile("uploadfile", "./static/files/"+h.Filename)

mrcuix · 2013-04-23T16:27:04Z

而且我尝试过改写你源码的如下：

controller.go

func (c *Controller) SaveToFile(fromfile, tofile string) error {
    c.Ctx.Request.ParseMultipartForm(32 << 20)
    file, _, err := c.Ctx.Request.FormFile(fromfile)
    if err != nil {
        return err
    }
    defer file.Close()
    f, err := os.OpenFile(tofile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
    if err != nil {
        return err
    }
    defer f.Close()
    io.Copy(f, file)
    return nil
}

在把你的beego 重新 go build 然后 go install 后
再用我这种方式尝试了一次，发现也是没有马上释放C盘占用的空间的

package main

import (
    "github.com/astaxie/beego"
)

type UploadController struct {
    beego.Controller
}
func (this *UploadController) Post() {
    _, h, _ := this.GetFile("uploadfile")
    this.SaveToFile("uploadfile", "./static/files/"+h.Filename)
    this.Ctx.Redirect(302, "/upload")

}

func main() {
    beego.Router("/upload", &UploadController{})
    beego.Run()
}

astaxie · 2013-04-23T16:29:19Z

 _, h, _ := this.GetFile("uploadfile")
this.SaveToFile("uploadfile", "./static/files/"+h.Filename)

把上面两行换成

this.SaveToFile("uploadfile", "./static/files/xxx.txt“)

astaxie · 2013-04-23T16:33:30Z

我觉得是this.GetFile("uploadfile")这个掉用了两次引起的

mrcuix · 2013-04-23T16:43:16Z

果然是用了两次，应该是第一次获取的时候获取文件名，然后内存占用了没有释放。
哎呀我没有认真考虑到呢

mrcuix · 2013-04-23T16:48:37Z

在第一次获取名称的时候改成下面的就可以正常了,上传完后占用的释放到了

 f, h, _ := this.GetFile("uploadfile")
 f.Close()
this.SaveToFile("uploadfile", "./static/files/"+h.Filename)

astaxie · 2013-04-23T16:49:58Z

我怎么感觉是go的内置bug呢，不知道go1.1存在不存在这个问题，你用的是go1.0.3版本吧

mrcuix · 2013-04-23T17:08:40Z

执行go version 返回以下信息

C:\Users\Administrator>go version
go version devel +d58997478ec6 Mon Apr 08 00:09:35 2013 -0700 windows/amd64

NormanZhang · 2014-01-09T13:40:48Z

不是吧，你们解决了？我怎么还是内存不释放啊
我的代码真的简单到爆啊。
code:
func (this *CommonController) UpLoadFile() {
this.SaveToFile("Filedata", "./static/files/xxx.rmvb")
}
只有这样啊。
开启程序只有1.5M内存，上传了一个100多兆的rmvb，保存完了之后，内存占用了370M，然后一直不释放。
但是用360回收一下内存，内存又恢复1.8M了。
用这个我写个文件存储服务，存几个大文件就爆了了。。。求解~~~~~~~~~~！！！！

NormanZhang · 2014-01-09T13:42:37Z

我只是用任务管理器看一下程序占用的内存

NormanZhang · 2014-01-09T13:46:44Z

怎么解决啊，我用上面原生的代码也是这样的情况

slene · 2014-01-09T14:23:26Z

@NormanZhang 你可以看下处理文件上传中，讲述 MaxMemory 的部分，你可以调小 beego.MaxMemory

NormanZhang · 2014-01-09T14:25:12Z

@slene 我调了，完全不起作用啊，任务管理器里面还是占用这么多

NormanZhang · 2014-01-09T14:25:45Z

@slene 是这句话吧？
this.Ctx.Request.ParseMultipartForm(32 << 20)

slene · 2014-01-09T14:34:45Z

默认是 64M，调小一下，你调的多少呢？试一下 10M
在 main.go 里

beego.MaxMemory = 10485760

NormanZhang · 2014-01-09T14:56:39Z

无效，只上传一个100兆的文件，内存占用依然需要200M，而且上传结束之后，内存不释放
T T

NormanZhang · 2014-01-09T14:56:52Z

@slene 忘了@了

slene · 2014-01-09T14:58:34Z

不立即释放，是对的，但内存占用减少了吧？

NormanZhang · 2014-01-09T15:11:49Z

@slene 木有啊

tossp · 2014-07-17T18:56:22Z

@astaxie 1.3还是一样的崩溃，默认配置，上传一个70MB的文件，4到6次，必定崩

2014/07/18 02:53:49 [log.go:63] [I] | beego     | 862.0493ms       | POST       | /api/v1/server/upload/

fatal error: runtime: cannot map pages in arena address space

goroutine 48 [running]:
runtime.throw(0xb96c03)
        D:/Develop/go/src/pkg/runtime/panic.c:520 +0x71 fp=0x1ef5d0 sp=0x1ef5c4
runtime.SysMap(0x0, 0x1000000, 0x1ef601, 0xba1c58)
        D:/Develop/go/src/pkg/runtime/mem_windows.c:117 +0x96 fp=0x1ef5f0 sp=0x1ef5d0
runtime.MHeap_SysAlloc(0xbaab20, 0x1000000)
        D:/Develop/go/src/pkg/runtime/malloc.goc:616 +0xf7 fp=0x1ef620 sp=0x1ef5f0
MHeap_Grow(0xbaab20, 0x800)
        D:/Develop/go/src/pkg/runtime/mheap.c:319 +0x57 fp=0x1ef648 sp=0x1ef620
MHeap_AllocLocked(0xbaab20, 0x800, 0x0)
        D:/Develop/go/src/pkg/runtime/mheap.c:222 +0x2f4 fp=0x1ef668 sp=0x1ef648
runtime.MHeap_Alloc(0xbaab20, 0x800, 0x0, 0x129e0101)
        D:/Develop/go/src/pkg/runtime/mheap.c:178 +0x8b fp=0x1ef67c sp=0x1ef668
largealloc(0x1, 0x1ef6d4)
        D:/Develop/go/src/pkg/runtime/malloc.goc:224 +0x97 fp=0x1ef6a0 sp=0x1ef67c
runtime.mallocgc(0xfffe00, 0x7cffc1, 0x1)
        D:/Develop/go/src/pkg/runtime/malloc.goc:169 +0xbc fp=0x1ef6d4 sp=0x1ef6a0
cnew(0x7cffc0, 0xfffe00, 0x1)
        D:/Develop/go/src/pkg/runtime/malloc.goc:836 +0xad fp=0x1ef6e4 sp=0x1ef6d4
runtime.cnewarray(0x7cffc0, 0xfffe00)
        D:/Develop/go/src/pkg/runtime/malloc.goc:849 +0x3f fp=0x1ef6f4 sp=0x1ef6e4
makeslice1(0x7c2880, 0xfffe00, 0xfffe00, 0x1ef734)
        D:/Develop/go/src/pkg/runtime/slice.goc:55 +0x4b fp=0x1ef700 sp=0x1ef6f4
runtime.makeslice(0x7c2880, 0xfffe00, 0x0, 0xfffe00, 0x0, 0x0, 0xfffe00, 0xfffe00)
        D:/Develop/go/src/pkg/runtime/slice.goc:36 +0xb2 fp=0x1ef720 sp=0x1ef700
bytes.makeSlice(0xfffe00, 0x0, 0x0, 0x0)
        D:/Develop/go/src/pkg/bytes/buffer.go:191 +0x72 fp=0x1ef744 sp=0x1ef720
bytes.(*Buffer).ReadFrom(0x13ba92c0, 0x186a48, 0x62353bb0, 0x7ffc8e, 0x0, 0x0, 0x0)
        D:/Develop/go/src/pkg/bytes/buffer.go:163 +0xbe fp=0x1ef7c0 sp=0x1ef744
io.Copy(0x186b80, 0x13ba92c0, 0x186a48, 0x62353bb0, 0x0, 0x0, 0x0, 0x0)
        D:/Develop/go/src/pkg/io/io.go:349 +0x124 fp=0x1ef840 sp=0x1ef7c0
io.CopyN(0x186b80, 0x13ba92c0, 0x186b98, 0x129e8810, 0x4000001, 0x0, 0x170ce4, 0x7c27c0, 0x0, 0x0)
        D:/Develop/go/src/pkg/io/io.go:318 +0xd1 fp=0x1ef88c sp=0x1ef840
mime/multipart.(*Reader).ReadForm(0x13bf6740, 0x4000000, 0x0, 0x0, 0x0, 0x0)
        D:/Develop/go/src/pkg/mime/multipart/formdata.go:68 +0x7b5 fp=0x1efa38 sp=0x1ef88c
net/http.(*Request).ParseMultipartForm(0x13c4c9a0, 0x4000000, 0x0, 0x0, 0x0)
        D:/Develop/go/src/pkg/net/http/request.go:796 +0x19e fp=0x1efb14 sp=0x1efa38
github.com/astaxie/beego/context.(*BeegoInput).ParseFormOrMulitForm(0x129e8240, 0x4000000, 0x0, 0x0, 0x0)
        D:/Users/zh/IdeaProjects/golang/src/github.com/astaxie/beego/context/input.go:282 +0xa4 fp=0x1efb84 sp=0x1efb14
github.com/astaxie/beego.(*ControllerRegistor).ServeHTTP(0x129b0c40, 0x186ab0, 0x13c4caf0, 0x13c4c9a0)
        D:/Users/zh/IdeaProjects/golang/src/github.com/astaxie/beego/router.go:597 +0x1fc4 fp=0x1efdfc sp=0x1efb84
net/http.serverHandler.ServeHTTP(0x129b4b40, 0x186ab0, 0x13c4caf0, 0x13c4c9a0)
        D:/Develop/go/src/pkg/net/http/server.go:1673 +0x155 fp=0x1efe30 sp=0x1efdfc
net/http.(*conn).serve(0x13be80a0)
        D:/Develop/go/src/pkg/net/http/server.go:1174 +0x8c6 fp=0x1effc8 sp=0x1efe30
runtime.goexit()
        D:/Develop/go/src/pkg/runtime/proc.c:1445 fp=0x1effcc sp=0x1effc8
created by net/http.(*Server).Serve
        D:/Develop/go/src/pkg/net/http/server.go:1721 +0x2be

astaxie · 2014-07-18T01:08:08Z

https://code.google.com/p/go/issues/detail?id=1464

YouZhengChuan · 2017-04-11T02:50:10Z

f, h, _ := this.GetFile("uploadfile")
defer f.Close()
最好改成这样
defer func() {
if err := f.Close(); err != nil {
log.Fatal("f.Close(): ", err.Error())
return
}
}()

如果用户POST里面的Header中，name不等于“uploadfile”，那么f.Close()会触发这个报错：
[C] [asm_amd64.s:514] Handler crashed with error runtime error: invalid memory address or nil pointer dereference
[C] [asm_amd64.s:514] /usr/local/go/src/runtime/asm_amd64.s:514
... ...

bestplay · 2017-07-29T10:12:22Z

@YouZhengChuan

f, h, _ := this.GetFile("uploadfile")
f, h, err := this.GetFile("uploadfile")
if err != nil {}
defer f.Close()

自己把 error 处理省掉了。

defer 之前处理 error 不就好了

beego 的文档里关于文件上传的章节，也写的有问题。没有把错误处理放到前面去

astaxie closed this as completed Apr 23, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

用SaveToFile文件上传后，一直占着虚拟内存没有释放 #42

用SaveToFile文件上传后，一直占着虚拟内存没有释放 #42

mrcuix commented Apr 20, 2013

astaxie commented Apr 21, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

slene commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

slene commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

slene commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

tossp commented Jul 17, 2014

astaxie commented Jul 18, 2014

YouZhengChuan commented Apr 11, 2017

bestplay commented Jul 29, 2017 •

edited

用SaveToFile文件上传后，一直占着虚拟内存没有释放 #42

用SaveToFile文件上传后，一直占着虚拟内存没有释放 #42

Comments

mrcuix commented Apr 20, 2013

astaxie commented Apr 21, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

mrcuix commented Apr 23, 2013

astaxie commented Apr 23, 2013

mrcuix commented Apr 23, 2013

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

slene commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

slene commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

slene commented Jan 9, 2014

NormanZhang commented Jan 9, 2014

tossp commented Jul 17, 2014

astaxie commented Jul 18, 2014

YouZhengChuan commented Apr 11, 2017

bestplay commented Jul 29, 2017 • edited

bestplay commented Jul 29, 2017 •

edited