runtime: sporadic memory allocation issues when sufficiently large dependencies are brought in by a program that sets memory rlimits #69105
Comments
ianlancetaylor
changed the title
gopherbot
added
the
compiler/runtime
|
If a program runs out of memory, it can't continue. 1G is not going to be enough for a large program. What are you suggesting that we do here? |
ianlancetaylor
added
the
WaitingForInfo
|
I am suggesting that something leaks, since it's very unlikely that simple program would run out of 1G of memory |
|
I don't think it's unlikely at all. You are limiting the size of virtual memory, but the program itself takes up virtual memory. |
|
Hmm, the core dump size is also nowhere near 1G. What am I missing here? |
|
The Go runtime reserves large portions of address space, see https://go.dev/doc/gc-guide#A_note_about_virtual_memory. RLIMIT_AS is not a good way to limit the memory of Go programs. cc @mknyszek |
|
Thank you! this link is super useful.
Thank you. I am assuming those come from the "sufficiently large dependencies" |
|
If you are running on Linux, I recommend using a memory cgroup to limit memory use of an application. Memory cgroups measure actual memory usage more precisely and aren't quite so trivially circumvented (RLIMIT_AS is immediately circumvented by fork()). |
So we follow up on the Setrlimit call with a |
|
Ah, do you actually want to apply the rlimit to a (non-Go) process you are exec'ing to? I think there is reasonable room for a proposal to add rlimits to https://pkg.go.dev/syscall#SysProcAttr so that |
|
Note that at least on Linux you can also do that by executing the process via the prlimit command. |
|
So playing around a bit with this new finding, I am getting larger VSS numbers in the core dump than I was expecting. I acknowledge that I might be doing something terribly wrong, but, would 1.4Gi be expected with the below program? package main
import (
"fmt"
"net/http"
)
func main() {
fmt.Println(http.Client{})
panic("test")
}$ eu-readelf -l reprducer-cgo-pthread.core | awk '{sz=strtonum($6); if (sz > 50 * 1024 * 1024) {print sz}}' | awk '{n += $1}; END{print n}'
1505267712 |
and others
Go version
go version go1.22.6 linux/amd64
Output of
go envin your module/workspace:What did you do?
Running a simple program that sets a 1G memory rlimit and does basically nothing,
with a dep on a sufficiently large library (k8s for the purpose of demonstration)
akalenyu/kubernetes@b88d05b
To reproduce:
What did you see happen?
What did you expect to see?
No sporadic ENOMEM
The text was updated successfully, but these errors were encountered: