Ability to constrain the total heap used by v8 and Node.js (--max-heap-size)

[giltayar]

Is your feature request related to a problem? Please describe.

Running NodeJS applications in a memory-constraining environment like Kubernetes is today problematic, because there is no way to tell NodeJS (and the v8 GC) how much total memory it has to work with. --max-old-space-size is a good start, but due to the way Buffer uses memory allocated outside of v8's heap (called "external memory"), this is not enough.

The problem in essence is this: when running under a memory constraint (i.e. docker/k8s limiting the process' memory), one can specify --max-old-space-size that is below the memory constraint, have memory that is below that usage, and yet the process still gets OOMKilled, because most of that memory lives in Buffer-s, and thus in the external heap, which v8 does not know about and thus does not feel like it needs to trigger a GC.

I created a Github repo that reproduces this problem: https://github.com/giltayar/node-memory-constraining-problem. Follow the README to reproduce the problem and to see a more in depth examination of this problem.

Describe the solution you'd like

I would like a --max-heap-size parameter to NodeJS that constrains v8's heaps and the external heap to not pass a specified limit.

This solution was proposed by @psmarshall and @bmeurer from the v8 team after discussing the problem with them.

Describe alternatives you've considered

A --max-external-heap-size parameter that constrains the external memory size used by Buffer and others.

[giltayar]

I have opened a sibling issue for v8: https://bugs.chromium.org/p/v8/issues/detail?id=8431

[bnoordhuis]

A script is going to get killed if it tries to allocate memory in excess of a limit, whether that's a Node.js limit or a kernel limit, so I don't know what you expect.

Buffer.alloc() could, in theory, throw an exception when a threshold is exceeded, but then what? Decades of C code not checking malloc()'s return value hints at what to expect and it's nothing good.

because most of that memory lives in Buffer-s, and thus in the external heap, which v8 does not know about and thus does not feel like it needs to trigger a GC.

Node.js informs V8 about external memory and V8's garbage collector uses that as an input signal. In other words, the GC runs more often when there's more external memory.

[bnoordhuis]

Buffer.alloc() could, in theory, throw an exception when a threshold is exceeded, but then what?

I suppose it could also perform an emergency GC when going over the threshold, in hopes of reclaiming enough Buffers / typed arrays to get below it again.

However, I don't expect that to be very effective because the GC is already quite aggressive. It seems unlikely there are generally enough unreachable objects to make a big difference.

[psmarshall]

This is about the specific case where you do have enough unreachable memory for the GC to be effective, but the GC is never triggered. A way to configure a total memory (or even just an external memory limit in this case) would let V8 attempt a GC.

[bnoordhuis]

Okay, I guess that's something we can try out. I'm confused by this:

A way to configure a total memory (or even just an external memory limit in this case) would let V8 attempt a GC.

If you mean a V8 flag or setting, then there's no need for an issue here, right? That trickles down to us eventually.

If instead the idea is that Node.js tracks external memory usage and invokes the garbage collector when it exceeds a threshold, then the first order of business is to come up with something more official than v8::EmbedderHeapTracer::GarbageCollectionForTesting().

[giltayar]

@bnoordhuis: if NodeJS does track external memory usage and notifies v8 of it, then you're right: it's just a matter of v8 adding a --max-total-heap-size flag as per the v8 bug I added, and enabling node to trickle it down to v8 (I'm assuming NodeJS still needs to trickle it down to v8).

But I wasn't sure whether this was tracked by node and reported to v8, hence this issue.

If instead the idea is that Node.js tracks external memory usage and invokes the garbage collector when it exceeds a threshold, then the first order of business is to come up with something more official than v8::EmbedderHeapTracer::GarbageCollectionForTesting()

I don't think we need this, as long as external memory is reported to v8 and v8 has a max-total-heap-size or max-external-heap-size flag.

[bnoordhuis]

Node.js calls Isolate::AdjustAmountOfExternalAllocatedMemory() when buffers are created and freed. I assume that's enough?

[bnoordhuis]

I'll go ahead and assume that no response means no action is needed on our side. Let me know if I need to reopen this.

[psmarshall]

Yes, we'll track this as a v8 issue in the linked bug and open a new Node bug if anything needs to change here.

[psmarshall]

FYI work has begun on this here: https://bugs.chromium.org/p/v8/issues/detail?id=9306

[mhio]

So it appears a --max-heap-size flag was added to v8 under 9306 and has trickled into nodejs 12/14.

The flag configures both the young and old generation limits

but it doesn't appear to account for external memory as in 8341
Should I use --max-heap-size as is or will it change behaviour to include the external memory at some point?