Skip to content

proposal: runtime: a hard limit version of GOMEMLIMIT for testing #63338

Open
@seebs

Description

@seebs

What version of Go are you using (go version)?

1.21, etc

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

More of an issue on MacOS, but exists anywhere I guess.

What did you do?

Tried to limit memory usage. Specifically, I wanted the allocator to fail (yes, I know this panics and can't be recovered) rather than more memory to be used.

What did you expect to see?

I don't honestly know. Memory usage is poorly-defined and inconsistent at the best of times.

What did you see instead?

So, modern MacOS just doesn't let you do anything much like ulimit anymore for memory usage. And modern Linux pretty much always does overcommit, and then if too much memory gets used, it kills... Something. Arbitrarily. Which may or may not be the thing you wanted.

I'm looking at some code which has an unbounded memory usage problem, which I want to fix, and first I wanted to write a test case that would confirm that it was failing. And discovered that I can't sanely do this; I can't actually make the test fail due to excessive memory usage, because the amount of memory I can allocate (way over 60GB) takes long enough to populate that tests time out, and anyway a test that takes over ten minutes to run isn't a great fit for ever getting anything done.

And we have GOMEMLIMIT, but that's advisory. And what I sort of want is, roughly, a GOMEMLIMITHARD, where when runtime is about to request more memory from the OS, it checks whether doing so would cause the runtime's total memory-requested-from-OS to exceed the limit, and if it would, to fail as though a request from the OS was denied.

FAQ

Q: But that will just kill your program!
A: Yes, exactly. It will kill this program, rather than causing the kernel to pick some arbitrary program to kill. That is, in fact, highly preferable.

Q: But you can just use the OS's limits!
A: Apparently not on MacOS in the last few years (?!?!). Still seems to work on Linux via ulimit -v, although it's a bit approximate, because memory "usage" is not well-defined. (For instance, mmapped files sometimes but not always count as some kinds of memory.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    FeatureRequestIssues asking for a new feature that does not need a proposal.NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.Proposalcompiler/runtimeIssues related to the Go compiler and/or runtime.

    Type

    No type

    Projects

    Status

    Todo

    Status

    Incoming

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions