Description
What version of Go are you using (go version
)?
1.21, etc
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env
)?
More of an issue on MacOS, but exists anywhere I guess.
What did you do?
Tried to limit memory usage. Specifically, I wanted the allocator to fail (yes, I know this panics and can't be recover
ed) rather than more memory to be used.
What did you expect to see?
I don't honestly know. Memory usage is poorly-defined and inconsistent at the best of times.
What did you see instead?
So, modern MacOS just doesn't let you do anything much like ulimit
anymore for memory usage. And modern Linux pretty much always does overcommit, and then if too much memory gets used, it kills... Something. Arbitrarily. Which may or may not be the thing you wanted.
I'm looking at some code which has an unbounded memory usage problem, which I want to fix, and first I wanted to write a test case that would confirm that it was failing. And discovered that I can't sanely do this; I can't actually make the test fail due to excessive memory usage, because the amount of memory I can allocate (way over 60GB) takes long enough to populate that tests time out, and anyway a test that takes over ten minutes to run isn't a great fit for ever getting anything done.
And we have GOMEMLIMIT
, but that's advisory. And what I sort of want is, roughly, a GOMEMLIMITHARD
, where when runtime is about to request more memory from the OS, it checks whether doing so would cause the runtime's total memory-requested-from-OS to exceed the limit, and if it would, to fail as though a request from the OS was denied.
FAQ
Q: But that will just kill your program!
A: Yes, exactly. It will kill this program, rather than causing the kernel to pick some arbitrary program to kill. That is, in fact, highly preferable.
Q: But you can just use the OS's limits!
A: Apparently not on MacOS in the last few years (?!?!). Still seems to work on Linux via ulimit -v
, although it's a bit approximate, because memory "usage" is not well-defined. (For instance, mmapped files sometimes but not always count as some kinds of memory.)
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Status