salto crashed during (initial) fetch from Salesforce

[grzn]

$ salto fetch
✔ Finished loading workspace for environment prod
Currently, the state of the salesforce service is unknown to Salto.

Fetching and applying changes from the service(s)
This might take a few minutes. You can go grab your favorite beverage.
The steps are: I. Fetching configs, II. Calculating difference and III. Applying the changes


  >>> Fetching the latest configs from: salesforce
  vvv Finished fetching the latest configs from: salesforce


  >>> Calculating the difference between local and remote
  vvv Finished calculating the difference between local and remote


  >>> Applying 6580 changes to the local workspace

<--- Last few GCs --->

[85765:0x102827000]   178065 ms: Mark-sweep 1858.1 (2088.1) -> 1858.0 (2082.8) MB, 1012.6 / 0.0 ms  (average mu = 0.089, current mu = 0.012) allocation failure scavenge might not succeed
[85765:0x102827000]   179526 ms: Mark-sweep 1864.7 (2089.6) -> 1864.5 (2085.3) MB, 1445.8 / 0.0 ms  (average mu = 0.044, current mu = 0.011) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2bbe587c0919 <JSObject>
    0: builtin exit frame: concat(this=0x2bbe15b82a71 <JSArray[19624]>,0x2bbe15b83351 <JSArray[1]>,0x2bbe15b82a71 <JSArray[19624]>)

    1: postprocess(aka arrpush) [0x2bbe7a37d9c9] [0x2bbed89c04d1 <undefined>:~70] [pc=0x30ab77d5f3fc](this=0x2bbe859cc7d9 <Rule map = 0x2bbe223f7779>,0x2bbe15b83281 <JSArray[2]>)
    2: arguments adaptor frame: 3->1
    3: finish [0x2bbe53323681] [0x2bbed89c04d1 <u...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100074f04 node::Abort() [/usr/local/bin/salto]
 2: 0x100075033 node::OnFatalError(char const*, char const*) [/usr/local/bin/salto]
 3: 0x10016876a v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/salto]
 4: 0x1002f32c5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/salto]
 5: 0x1002f4be9 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/salto]
 6: 0x1002f136d v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/salto]
 7: 0x1002eeff2 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/salto]
 8: 0x1002fb744 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/salto]
 9: 0x1002fb7bf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/salto]
10: 0x1002c6a7e v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::RootIndex, int, v8::internal::Object, v8::internal::AllocationType) [/usr/local/bin/salto]
11: 0x1002d6778 v8::internal::Factory::NewJSArrayStorage(v8::internal::Handle<v8::internal::JSArray>, int, int, v8::internal::ArrayStorageAllocationMode) [/usr/local/bin/salto]
12: 0x1002d65ec v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::AllocationType) [/usr/local/bin/salto]
13: 0x10042da3f v8::internal::ElementsAccessor::Concat(v8::internal::Isolate*, v8::internal::Arguments*, unsigned int, unsigned int) [/usr/local/bin/salto]
14: 0x1001d7003 v8::internal::(anonymous namespace)::Fast_ArrayConcat(v8::internal::Isolate*, v8::internal::BuiltinArguments*) [/usr/local/bin/salto]
15: 0x1001d4ae6 v8::internal::Builtin_Impl_ArrayConcat(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/salto]
16: 0x10094e999 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/usr/local/bin/salto]
17: 0x30ab77d5f3fc
18: 0x1008c683c Builtins_ArgumentsAdaptorTrampoline [/usr/local/bin/salto]
19: 0x30ab7814537b
20: 0x30ab77e47c28
[1]    85765 abort      salto fetch

I ran this command with telemetry turned off, I'll run it again so you guys will have the stack trace in your backend.

Also, it would be nice to print the timestamps/time lapses so we'll know how long each step took.

[ori-moisis]

Hi Guy,

Thank you for reporting the issue, we will look into it (having telemetry enabled will certainly help so thanks for that as well).

At first glance this seems to be a memory consumption issue that you might be able to work around by configuring NodeJS (which is our underlying engine for the salto CLI).
Try setting the following environment variable prior to running salto:

NODE_OPTIONS=--max_old_space_size=4096

(this can be done on linux / mac by running export NODE_OPTIONS=--max_old_space_size=4096)

Hope this helps

[grzn]

It still crashes, reached 4GB: ``` $ salto fetch ✔ Finished loading workspace for environment prod Currently, the state of the salesforce service is unknown to Salto. Fetching and applying changes from the service(s) This might take a few minutes. You can go grab your favorite beverage. The steps are: I. Fetching configs, II. Calculating difference and III. Applying the changes

>> Fetching the latest configs from: salesforce

vvv Finished fetching the latest configs from: salesforce

>> Calculating the difference between local and remote

vvv Finished calculating the difference between local and remote

>> Applying 6580 changes to the local workspace

<--- Last few GCs ---> [94415:0x102827000]   204196 ms: Mark-sweep 3881.5 (4135.8) -> 3881.4 (4135.6) MB, 1420.1 / 0.0 ms  (average mu = 0.102, current mu = 0.013) allocation failure scavenge might not succeed [94415:0x102827000]   205588 ms: Mark-sweep 3885.9 (4140.0) -> 3885.8 (4140.0) MB, 1384.0 / 0.0 ms  (average mu = 0.062, current mu = 0.006) allocation failure scavenge might not succeed <--- JS stacktrace ---> ==== JS stack trace ========================================= Security context: 0x181256940919 <JSObject> 0: builtin exit frame: concat(this=0x18121d600f89 <JSArray[30223]>,0x18121d6018a1 <JSArray[1]>,0x18121d600f89 <JSArray[30223]>) 1: postprocess(aka arrpush) [0x1812345dc761] [0x1812dac804d1 <undefined>:~70] [pc=0x286ee4e3461c](this=0x1812da500139 <Rule map = 0x1812f35b6d79>,0x18121d6017d1 <JSArray[2]>) 2: arguments adaptor frame: 3->1 3: process [0x181233a35f61] [0x1812dac804d1 <... FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x100074f04 node::Abort() [/usr/local/bin/salto] 2: 0x100075033 node::OnFatalError(char const*, char const*) [/usr/local/bin/salto] 3: 0x10016876a v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/salto] 4: 0x1002f32c5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/salto] 5: 0x1002f4be9 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/salto] 6: 0x1002f136d v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/salto] 7: 0x1002eeff2 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/salto] 8: 0x1002fb744 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/salto] 9: 0x1002fb7bf v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/usr/local/bin/salto] 10: 0x1002c6a7e v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::RootIndex, int, v8::internal::Object, v8::internal::AllocationType) [/usr/local/bin/salto] 11: 0x1002d6778 v8::internal::Factory::NewJSArrayStorage(v8::internal::Handle<v8::internal::JSArray>, int, int, v8::internal::ArrayStorageAllocationMode) [/usr/local/bin/salto] 12: 0x1002d65ec v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::AllocationType) [/usr/local/bin/salto] 13: 0x10042da3f v8::internal::ElementsAccessor::Concat(v8::internal::Isolate*, v8::internal::Arguments*, unsigned int, unsigned int) [/usr/local/bin/salto] 14: 0x1001d7003 v8::internal::(anonymous namespace)::Fast_ArrayConcat(v8::internal::Isolate*, v8::internal::BuiltinArguments*) [/usr/local/bin/salto] 15: 0x1001d4ae6 v8::internal::Builtin_Impl_ArrayConcat(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/salto] 16: 0x10094e999 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/usr/local/bin/salto] [1]    94415 abort      salto fetch ``` I don't see a point in setting it to 8GB, do you?

…

On Wed, May 06, 2020 at 12:14 PM, ori-moisis < ***@***.*** > wrote: Hi Guy, Thank you for reporting the issue, we will look into it (having telemetry enabled will certainly help so thanks for that as well). At first glance this seems to be a memory consumption issue that you might be able to work around by configuring NodeJS (which is our underlying engine for the salto CLI). Try setting the following environment variable prior to running salto: NODE_OPTIONS=--max_old_space_size=4096 (this can be done on linux / mac by running export NODE_OPTIONS=--max_old_space_size=4096 ) Hope this helps — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub ( #997 (comment) ) , or unsubscribe ( https://github.com/notifications/unsubscribe-auth/AABKYFVFPEMU7WORAKDV5BDRQES7TANCNFSM4M2IBZ2A ).

[ori-moisis]

Since this is still failing on out of memory and not something else, going bigger might work.

We would be happy to dive deeper into your scenario to understand what goes wrong there, would you like to join our Slack workspace for more direct communication?

[grzn]

sure.

[roironn]

Closing this issue as it was solved in v0.1.15 (https://github.com/salto-io/salto/releases/tag/v0.1.15)