net/http: ParseMultipartForm using 8x maxMemory

[git001]

What version of Go are you using (go version)?

$ go version
go version go1.18.1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOENV="/home/alex/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/alex/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/alex/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2728191052=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have created a small upload tool for the caddy v2 server
https://github.com/git001/caddyv2-upload

I set the MaxBytesReader ( https://github.com/git001/caddyv2-upload/blob/main/upload.go#L125 ) and ParseMultipartForm ( https://github.com/git001/caddyv2-upload/blob/main/upload.go#L126 ) to limit the memory usage.

What we have observed is that even when we limit the ParseMultipartForm() maxMemory parameter is the initial memory consumption 7-8 times higher then the given maxMemory.
git001/caddyv2-upload#2

What did you expect to see?

We expect that the memory usage is not 7-8 times higher at the initial upload phase then the configured one for ParseMultipartForm

What did you see instead?

We see that the memory usage 7-8 times higher then configured maxMemory in ParseMultipartForm

[seankhliao]

Please try profiling to identify where the memory might be used.
Note maxMemory refers to the max file size that will be stored in memory, not the maximum memory that will be used during the operation.

[git001]

Note maxMemory refers to the max file size that will be stored in memory, not the maximum memory that will be used during the operation.

thank you for the info. I will profile the app.

[git001]

I have now created a simple module which shows my observations.
https://github.com/git001/golang-multiparttest

What I have seen is that the parameter maxMemory is a very important "flag" between memory usage and filesystem usage.
My conclusion is that the Multipart upload requires 2*times the memory of maxMemory but I was not able to use the tool pprof in that way to find out why this happen.

Finally I understand now the sentence in the doc which explains exactly the same which could be the reason for the doubled memory usage.

The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.