Memory is not cleaned up while doing some stress testing

[vunhatminh241191]

I am a newbie with IRIS, so I am appreciated anyone input to my problem

Let's talk about my setup:
First: I have a simple IRIS server, which is only accepting a HTTP POST request to upload and store file (10MB).
Second: I uses vegeta to do some stress testing, where I create 1000 connections per second and post a same file to my server.

My problem is: IRIS server uses all of the memory that I have (16GB). We guess these connections do not release these memory after they finished their thread.
So I don't know if it is a bug or a new feature should be add on? Can anyone help me out please?

Thanks

[kataras]

Hello @vunhatminh241191 , can you verify and prove the issue you are describing? We serve 1million requests every day with a lot fewer memory, and it is cleaned up correctly, in fact this is not Iris "magic", it is just Go + net/tcp and net/http that does the memory cleanup and etc. Iris has nothing to do with "cleanup" memory on a simple app, so it is hard to believe that it has something to do with Iris, can you please post the progress you followed along with the code snippet you used?

Thanks!

[jjhesk]

if that is a concurrent process, i think we need to take a deep look of it.

[jjhesk]

yeap, i think it is due to connection without epolling that cause residual leaks.

[kataras]

@vunhatminh241191 by-default Iris will make use of a tcp keep alive connection in iris.Addr, you can change the underline listener by passing the iris.Listener on the Application#Run instead, test it :

ln, err := net.Listen("tcp", ":8080")
if err != nil {
  // [handle error.]
}
 
// [...]
app.Run(iris.Listener(ln))

[kataras]

@jjhesk it is not that, not everything is releated with the epolling and things like that you all gophers reading the last days from an article that posted some days ago. People that are working with net/http, net and golang professionally, like me, we all know these things for quite some years now before an X article published and puts people on a missleading position. The std library and go authors are not stupid, don't worry.

[jjhesk]

thanks for the updates

[vunhatminh241191]

@kataras I think you are right. I found some documents which is related to Go's memory leak such as this and this. However, here is my code snippet, please let me know if I do sth wrong:

app.Post("/upload_file", func(ctx iris.Context) {
// Get the max post value size passed via iris.WithPostMaxMemory.
fmt.Println("ehhe", ctx.FormFile)
maxSize := ctx.Application().ConfigurationReadOnly().GetPostMaxMemory()
fmt.Println("maxsize", maxSize)
err := ctx.Request().ParseMultipartForm(maxSize)
if err != nil {
fmt.Println(err.Error(), "huhuhu")
ctx.StatusCode(iris.StatusInternalServerError)
ctx.WriteString(err.Error())
return
}

            form := ctx.Request().MultipartForm
            fmt.Println(form, "form here")

            files := form.File["files[]"]
            fmt.Println(files, "files here")
            failures := 0
            for _, file := range files {
                    _, err = saveUploadedFile(file, "./uploads")
                    if err != nil {
                            fmt.Println(err.Error(), "hixhix")
                            failures++
                            ctx.Writef("failed to upload: %s\n", file.Filename)
                    }
            }
            ctx.Writef("%d files uploaded", len(files)-failures)
    })
    // to start a new server listening at :80 and redirects
    // to the secure address, then:
    target, _ := url.Parse("https://0.0.0.0:443")
    go host.NewProxy("0.0.0.0:8080", target).ListenAndServe()

    // start the server (HTTPS) on port 443, this is a blocking func
    app.Run(iris.TLS("0.0.0.0:8443", "mycert.crt", "mykey.key"))

}

func saveUploadedFile(fh *multipart.FileHeader, destDirectory string) (int64, error) {
src, err := fh.Open()
if err != nil {
return 0, err
}
defer src.Close()
fh.Filename = strconv.FormatInt(time.Now().Unix(), 10) + "-" + fh.Filename
out, err := os.OpenFile(filepath.Join(destDirectory, fh.Filename),
os.O_WRONLY|os.O_CREATE, os.FileMode(0666))

    if err != nil {
            return 0, err
    }
    defer out.Close()

    return io.Copy(out, src)

}

I am very appreciated you guys input. Thanks