runtime: VSZ(virtual memory size) Increased by Approx. 500MB: GO Apps Upgraded from go1.19.3 to go1.22.0

[denver1111]

Go version

go version go1.22.0 linux/amd64

Output of go env in your module/workspace:

Build command:
#GOOS=linux GOARCH=arm64 go build -mod=readonly -o myapplication1
Run command:
#"start" : "/usr/bin/env GOTRACEBACK=crash myapplication1

No extra param used apart from above mentioned during build or run myapplication1

#go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/.go_1.22/pkg/mod'
GONOPROXY='bitbucket.xyz.com'
GONOSUMDB='bitbucket.xyz.com'
GOOS='linux'
GOPATH='/root/.go_1.22'
GOPRIVATE='bitbucket.xyz.com'
GOPROXY='https://artifactory.xyz.com/artifactory/go-remote/'
GOROOT='/usr/local/go_1.22/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go_1.22/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/root/application1/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build534398313=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I upgraded all my Go application in my platform from go1.19.3 to go1.22.0. I have total 3 application written in GO in my platform.

What did you see happen?

After the upgrade, I observed an approximate increase of 500MB in the VSZ (Virtual Memory Size) as displayed in the 'top' command (a Linux command) for all my Go applications. Below are the VSZ values captured from the 'top' command for each application running on go1.22.0 (after the upgrade):

The VSZ is reported to be approximately 1200MB for each of my Go applications individually.

What did you expect to see?

Before the Go upgrade, I was using go1.19.3 to build my applications, and here's what I used to observe.

The VSZ (virtual memory size) was reported to be approximately 700MB for each of my Go applications individually.

Please note:
If I rebuild any one of these three applications using go1.19.3 and run it, I observe that the VSZ (virtual memory size) of that particular application returns to the normal reading of 700MB. However, the other two applications continue to run with a VSZ of 1200MB.

I attempted to build my applications with Go version go1.22.2 as well, but the issue persists.

[seankhliao]

given it's just virtual memory, why is this an issue?

[denver1111]

given it's just virtual memory, why is this an issue?

Higher virtual memory consumption can signify various issues. With limited physical memory on my system, excessive virtual memory usage could result in resource contention, possibly indicating memory leaks or inefficient memory management. Each Go application is utilizing 122% of virtual memory(see %VSZ), potentially impacting other processes and leading to decreased system performance or responsiveness.

My concern lies with the significant 500MB increase in virtual memory usage observed in Go releases post 1.19.3. What factors are contributing to this surge in virtual memory consumption?

[adonovan]

With limited physical memory on my system, excessive virtual memory usage could result in resource contention, possibly indicating memory leaks or inefficient memory management. Each Go application is utilizing 122% of virtual memory(see %VSZ), potentially impacting other processes and leading to decreased system performance or responsiveness.

I'm not quite sure what you mean by "resource contention" or "inefficient memory management". An application that maps a single block of virtual addresses (even one much larger than physical memory) doesn't use more than a constant amount of additional RAM: the only resources being used in this case are an extra entry in the kernel's page table and one in the processor's TLB. And unbacked virtual address mappings may be used to implement algorithms that save actual RAM.

Each Go application is utilizing 122% of virtual memory(see %VSZ), potentially impacting other processes and leading to decreased system performance or responsiveness.

Are you using busybox top? It differs from the standard one: its %VSZ is the ratio of virtual address space to MemTotal (physical memory, used + free) (code), so it is quite normal for it to be more than 100%, or even much higher.

[denver1111]

Can anyone help me by answering this.

"My concern lies with the significant 500MB increase in virtual memory usage observed in Go releases post 1.19.3. What factors are contributing to this surge in virtual memory consumption?"

[adonovan]

What factors are contributing to this surge in virtual memory consumption?"

I don't know, but you could try running your application in a debugger such as lldb or dlv, with a breakpoint on the mmap system call, conditional on a size (reg. SI or R1) larger than some big value, and see what the Go stack looks like at that point.

[cherrymui]

Go's virtual address mapping changes over time. It may map or reserve memory address space differently, but the actually used memory space should not increase much.

potentially impacting other processes and leading to decreased system performance or responsiveness.

Could you explain what impact actually occurred? Thanks.

[mknyszek]

See also https://go.dev/doc/gc-guide#A_note_about_virtual_memory, which echoes what others have said here already.

Given the versions involved, it's possibly also related to changes to how huge pages are managed in Go 1.21, but that should not have affected virtual memory footprint. See https://go.dev/doc/gc-guide#Linux_transparent_huge_pages anyway, in case it offers you some insights.

[denver1111]

Could you explain what impact actually occurred?

I am getting occasional crash when doing stress testing of my application. And this crashes I am seeing only after Go version upgrade to 1.22.0

And each crash is producing CORE file which is giving similar stack trace
.... .... 6 0x000000000004adcc in runtime.crash at /usr/local/go/src/runtime/signal_unix.go:1005 7 0x000000000004adcc in runtime.fatalthrow.func1 at /usr/local/go/src/runtime/panic.go:1203 8 0x000000000004ad48 in runtime.fatalthrow at /usr/local/go/src/runtime/panic.go:1192 9 0x000000000004a940 in runtime.throw at /usr/local/go/src/runtime/panic.go:1023 10 0x0000000000026c98 in runtime.badPointer at /usr/local/go/src/runtime/mbitmap.go:286 11 0x0000000000026dfc in runtime.findObject at /usr/local/go/src/runtime/mbitmap.go:329 12 0x0000000000045f3c in runtime.wbBufFlush1 at /usr/local/go/src/runtime/mwbbuf.go:240 13 0x00000000000787e4 in runtime.wbBufFlush.func1 at /usr/local/go/src/runtime/mwbbuf.go:181 14 0x000000000007ec0c in runtime.systemstack at /usr/local/go/src/runtime/asm_arm64.s:243 15 0x000000000007eb88 in runtime.systemstack_switch at /usr/local/go/src/runtime/asm_arm64.s:200 16 0x0000000000045e3c in runtime.wbBufFlush at /usr/local/go/src/runtime/mwbbuf.go:180 17 0x000000000007e434 in gcWriteBarrier at /usr/local/go/src/runtime/asm_arm64.s:1294 18 my application code trace <not added here> which is not making any call to GC ... ...
Till frame 18, I am getting application stack traces but suddenly it is going into GC space which has no direct call from my application. hence I am loosing the trace here. Please note, I am not claiming this impact is due to increased VSZ.

While doing this analysis I observed about this 500MB increase of virtual space in Go application, hence asked a question here to know what has changed in recent GO version so that it is showing consumed VSZ increased by 500MB.

It may map or reserve memory address space differently, but the actually used memory space should not increase much.

Yes, I agree to this as actual memory consumption of my application didn't increase even with increased VSZ.

With all the explanation given before, can I safely assume this VSZ increased with new Go version 1.22.0 has no impact on Go applications as it is only related to Virtual memory?

[randall77]

I am getting occasional crash when doing stress testing of my application. And this crashes I am seeing only after Go version upgrade to 1.22.0

And each crash is producing CORE file which is giving similar stack trace

Could you open a new issue for that crash?
It would help if you could paste in the crash report stack trace that the runtime prints. It has additional information that's not in the stack trace above.
Please run the race detector (-race) on your program. Data races can generate this kind of crash.

With all the explanation given before, can I safely assume this VSZ increased with new Go version 1.22.0 has no impact on Go applications as it is only related to Virtual memory?

Yes.

I'm going to close this issue, as I think the virtual memory part at least is resolved.