FATAL ERROR: JavaScript heap out of memory

[mikemonaco]

@lukehoban (from slack)

We are running into a strange issue when using pulumi.StackReference. We are using two stacks, one for infrastructure and one for services as recommended by the docs. The infrastructure stack deploys without any issues. In the services stack, we are creating a StackReference and calling getOutput to get/use variables that were exported from the infrastructure stack.

index.ts

import * as pulumi from "@pulumi/pulumi";

const env = pulumi.getStack();
const infraStackReference = new pulumi.StackReference("username/pulumi-infra/"+env);

//const vpcOutput = infraStackReference.getOutput("vpc");
//const clusterOutput = infraStackReference.getOutput("cluster");
//const albOutput = infraStackReference.getOutput("internalAlb");

When I run pulumi up --yes the index.ts file show above is successful. If I un-comment just one of the commented lines, it will still work, but it takes about 1-2 minutes on the pulumi:pulumi:Stack pulumi-services-project-staging running read pulumi:pulumi:StackReference username/pulumi-infra/project-staging step. For each of the getOutput lines that I un-comment, the build process takes an additional 2 minutes or so. If all three are uncommented, the process takes about 5-6 minute and then fails due to FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory (see full_log.txt below)

Our infrastructure index.ts file is not very large, maybe 500 or so lines creating a single VPC, 2 load balancers, an S3 Bucket, DynamoDb Table, an ECS cluster, an EC2 instance, and various IAM roles.

I've attached a log from the failed build processes. full_log.txt

Our build is taking place on AWS CodeBuild with 7 GB memory and 4 vCPUs.

What is interesting is the following log file. This is the log when only calling getOutput("vpc") and leaving the other two getOutput calls commented.

Notice the HUGE dump on line 191 ReadResource RPC finished: vpc_only_log.txt

Does that look normal, or does it seem to be exposing the issue?

[lukehoban]

It is vaguely possible this is related to #3071?

[mikemonaco]

It is vaguely possible this is related to #3071?

@lukehoban we are not using secrets, so it seems unlikely. Does the size of the output on line 191 of the previously attached vpc_only_log.txt seem suspicious to you?

[mikemonaco]

@lukehoban could this be related to #3257?

[pbn4]

I am hitting the similar problem on Node v10.16.3:

Diagnostics:
  pulumi:pulumi:Stack (sm-secure-infra-prod):
    FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
     1: 0x10003b348 node::Abort() [/usr/local/opt/node@10/bin/node]
     2: 0x10003b521 node::OnFatalError(char const*, char const*) [/usr/local/opt/node@10/bin/node]
     3: 0x10017cbbf v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/opt/node@10/bin/node]
     4: 0x10017cb60 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/opt/node@10/bin/node]
     5: 0x1004328e4 v8::internal::Heap::UpdateSurvivalStatistics(int) [/usr/local/opt/node@10/bin/node]
     6: 0x100434523 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/opt/node@10/bin/node]
     7: 0x100431bbe v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/opt/node@10/bin/node]
     8: 0x1004309dd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/opt/node@10/bin/node]
     9: 0x1004391be v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/opt/node@10/bin/node]
    10: 0x100439208 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/opt/node@10/bin/node]
    11: 0x1004199e3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/opt/node@10/bin/node]
    12: 0x1005f30ea v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/opt/node@10/bin/node]
    13: 0x3cb22765be3d
    14: 0x3cb227a0f4eb

    <--- Last few GCs --->
    [14615:0x104002a00]   472294 ms: Scavenge 1141.6 (1439.1) -> 1135.3 (1439.6) MB, 10.8 / 0.0 ms  (average mu = 0.210, current mu = 0.197) allocation failure
    [14615:0x104002a00]   472351 ms: Scavenge 1147.2 (1439.6) -> 1139.3 (1439.6) MB, 7.4 / 0.0 ms  (average mu = 0.210, current mu = 0.197) allocation failure
    [14615:0x104002a00]   472375 ms: Scavenge 1148.7 (1439.6) -> 1146.1 (1441.1) MB, 11.3 / 0.0 ms  (average mu = 0.210, current mu = 0.197) allocation failure
    <--- JS stacktrace --->
    ==== JS stack trace =========================================
        0: ExitFrame [pc: 0x3cb22765be3d]
    Security context: 0x1d6f2411e6e9 <JSObject>
        1: all [0x1d6f24111081](this=0x1d6f24110ef9 <JSFunction Promise (sfi = 0x1d6f6ca94a09)>,0x1d6f6ae7fe69 <JSArray[2]>)
        2: /* anonymous */(aka /* anonymous */) [0x1d6f6ae78061] [/Users/michal/Projekty/supermassive_secure_infra/node_modules/@pulumi/pulumi/output.js:~84] [pc=0x3cb227a612fc](this=0x1d6f6ae78129 <JSProxy>,func=0x1d6fcfaf8821 <JSFunction out...

    error: an unhandled error occurred: Program exited with non-zero exit code: -1

[lukehoban]

Does anyone who has seen this have any more details for a repro? We have not yet been able to reproduce this, and don't have any great guesses for what would be wrong here based on details shared so far unfortunately.

[julienvincent]

No details for a repro but I can confirm that I am getting this as well. My stack outputs do not have secrets

[lukehoban]

I believe this is likely fixed with #3850. If anyone still sees issues like this - please do open a new issue.