JavaScript heap out of memory

[sanepal]

I'm running into the following issue when synthesizing an app that has stacks in many regions:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
   1: node::Abort() [/usr/local/Cellar/node@8/8.16.0/bin/node]
   2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/Cellar/node@8/8.16.0/bin/node]
   3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/Cellar/node@8/8.16.0/bin/node]
   4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/Cellar/node@8/8.16.0/bin/node]
   5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/Cellar/node@8/8.16.0/bin/node]
   6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/Cellar/node@8/8.16.0/bin/node]
   7: v8::String::WriteUtf8(char*, int, int*, int) const [/usr/local/Cellar/node@8/8.16.0/bin/node]
   8: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [/usr/local/Cellar/node@8/8.16.0/bin/node]
   9: node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding) [/usr/local/Cellar/node@8/8.16.0/bin/node]
  10: node::Buffer::(anonymous namespace)::CreateFromString(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/Cellar/node@8/8.16.0/bin/node]
  11: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/local/Cellar/node@8/8.16.0/bin/node]
  12: v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/Cellar/node@8/8.16.0/bin/node]
  13: v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/Cellar/node@8/8.16.0/bin/node]
  14: 0x131f3a6842fd
  15: 0x131f3a73edb6

<--- Last few GCs --->
  
  [52557:0x104001a00]    65671 ms: Mark-sweep 1078.2 (1204.6) -> 1077.9 (1208.6) MB, 189.6 / 0.0 ms  allocation failure GC in old space requested
  [52557:0x104001a00]    65891 ms: Mark-sweep 1077.9 (1208.6) -> 1077.9 (1172.1) MB, 219.6 / 0.0 ms  last resort GC in old space requested
  [52557:0x104001a00]    66104 ms: Mark-sweep 1077.9 (1172.1) -> 1077.9 (1167.6) MB, 213.3 / 0.0 ms  last resort GC in old space requested
  
  
  <--- JS stacktrace --->
  
  ==== JS stack trace =========================================
  
  Security context: 0x150ee2725891 <JSObject>
      1: fromString(aka fromString) [buffer.js:314] [bytecode=0x150e49bdf279 offset=164](this=0x150e69b022d1 <undefined>,string=0x150e2480b949 <Very long string[172886547]>,encoding=0x150ee2735b29 <String[4]: utf8>)
      2: from [buffer.js:177] [bytecode=0x150e49b99c09 offset=11](this=0x150e71b35dd9 <JSFunction Buffer (sfi = 0x150ee277e529)>,value=0x150e2480b949 <Very long string[172886547]>,encodi...
  

Is it possible to configure the memory allocated to Node?

Additionally, the above exception is not surfaced appropriately to the end user. This is what I saw:

Exception in thread "main" java.lang.NullPointerException
        at software.amazon.jsii.JsiiRuntime.requestResponse(JsiiRuntime.java:83)
        at software.amazon.jsii.JsiiClient.callMethod(JsiiClient.java:185)
        at software.amazon.jsii.JsiiObject.jsiiCall(JsiiObject.java:57)
        at software.amazon.awscdk.core.App.synth(App.java:65)

From there, I had to debug JsiiRuntime to get the stderr and stdout to find this error.

[RomainMuller]

Hey @sanepal,

It would help if you could show us the code that causes this error to happen. I understand this may be impossible, so I reckon the next best thing would be if you can give us some idea of:

• How many stacks are in your app
• How many resources are in your stacks (approximately 😅)

A workaround I can propose is to try and bump the node process' maximum heap size by setting the NODE_OPTIONS=--max-old-space-size=8192 environment variable (this sets the maximum heap to 8GiB).

[RomainMuller]

Then, I reckon the issue is caused by the fact the jsii runtime does not forward finalizations from the Java process to the node process, meaning any allocated object ever would be retained forever in the node heap.

See aws/jsii#643

[jeshan]

I got the memory issue from Python. The workaround works fine but it's not going to scale since we may hit the limit again later.

For my case, I have 4 top-level stacks, each of which nests a vpc stack. They share most of my ~12 constructs.
My estimated resource count across all of them is 297. They are all in the same account in different regions.

[pkspks]

This is happening to us as well. Getting a nullpointer with no other details. @sanepal , how did you get JSII to spit out the logs/trace?

[gaocan1992]

I got the memory issue from Python. The workaround works fine but it's not going to scale since we may hit the limit again later.

For my case, I have 4 top-level stacks, each of which nests a vpc stack. They share most of my ~12 constructs.
My estimated resource count across all of them is 297. They are all in the same account in different regions.

Agree, I also use this to fix the issue but once I added more resources. It happened again and I found that sometime you can build the entire CDK package but when you try to deploy an individual stack this will also happen.

[gaocan1992]

Hey @sanepal,

It would help if you could show us the code that causes this error to happen. I understand this may be impossible, so I reckon the next best thing would be if you can give us some idea of:

• How many stacks are in your app
• How many resources are in your stacks (approximately 😅)

A workaround I can propose is to try and bump the node process' maximum heap size by setting the NODE_OPTIONS=--max-old-space-size=8192 environment variable (this sets the maximum heap to 8GiB).

Is this because we have too many resources in one stack or we have too many stacks. Or both?

[hastarin]

Just a note that I ran into a similar issue with a simple stack and found this issue.

In my case (Windows 10, C# based stack) uninstalling nodejs 32-bit and installing the 64-bit version resolved the issue.

[github-actions]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.