OOM Crashes are still happening in the worker

[josephjclark]

We've tightened up some processing in the worker to make the main thread more resiliant.

But we're still seeing OOM explosions - impact-water is blowing up a couple of times a week

Here's one in GCP for 832eaf62-4bed-4c8b-bebe-6a0712e70aaa

Couple of notes:

• the last payload is about 5mb on disk pretty-printed. So not big I'd say.
• I think the worker blew up around 1gb of memory, but I'm not good at understanding the stack trace
• I don't think the worker is under particularly high load

I suspect this has more to do with the main worker thread processing many smaller events, than being blown up by one big event. But I'm really not sure.

Might be related to #897

Might be related to #826

[josephjclark]

Perhaps more than anything I'd really really like a way to notify us and the app about these OOM kills. But the nature of it means we can't do much: IIRC we do get a callback hook but it's strictly synchronous.

[josephjclark]

So I finally have a repro for this:

Start the worker with a limited heap:

NODE_OPTIONS=--max_old_space_size=50 npm start

That's a 50mb main thread limit. Node's own debugging tells me the heap is around double that

(question: if the main parent has this limit, do the children also have it? That's driven by config in the worker itself right?)

Now run a workflow with this step:

fn((state) => {
    state.data = []
    let counter = 1e4;
    while(--counter) {
      const str = new Array(1e3).fill(1).join('')
      state.data.push(str)
    }
    console.log(state.data.length)
    return state;
})

I see this error locally:

[R/T] ❯ Final memory usage: [step 47mb] [system 170mb]
[SRV] ℹ ac10e787-1b60-4d90-a23f-f3b54e2b28de :: run:log :: OK
[SRV] ℹ ac10e787-1b60-4d90-a23f-f3b54e2b28de :: run:log :: OK
[SRV] ℹ ac10e787-1b60-4d90-a23f-f3b54e2b28de :: run:log :: OK

<--- Last few GCs --->

[100237:0x3ae5c000]   209159 ms: Mark-Compact 49.1 (54.6) -> 48.7 (56.4) MB, pooled: 2 MB, 7.37 / 0.00 ms  (average mu = 0.970, current mu = 0.095) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xe1bdfe node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0x123e800 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0x123ead7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x146c615  [node]
 5: 0x1485e89 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 6: 0x145a558 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 7: 0x145b485 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x143387e v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
 9: 0x1422974 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Tagged<v8::internal::Map>, v8::internal::AllocationAlignment) [node]
10: 0x1423e3e v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
11: 0x143f219 v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<char const>, v8::internal::AllocationType) [node]
12: 0x125ad1d v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [node]
13: 0x10d0977  [node]
14: 0x10d0aaf node::StringDecoder::DecodeData(v8::Isolate*, char const*, unsigned long*) [node]
15: 0x10d100d  [node]
16: 0x1d535e2  [node]
Aborted (core dumped)

I think this has reproduced the case where:

• the inner worker thread has created/downloaded a payload too large
• somewhere on transferring that to the main thread, the main thread has blown up

I believe it's possible for this to occur naturally, just with a big payload, but a worker under load, or that has been running for a longer time, will have less main thread heap available and so be more likely to see this issue. I've had to limit the memory quite extensively here.

[josephjclark]

nodejs: Understanding and Tuning Memory

[josephjclark]

Setting --run-memory gives me a killed status on the run and the worker survives, as expected

[josephjclark]

Ok I'm about convinced that there's no magic here. It all depends on the numbers.

The above example is fine if state is always redacted from events, or if you set --payload-memory to a really low value. Suddenly it all works fine.

So nothing is wrong, exactly. If you send a payload to the main process, and the main process runs out of memory while processing it, then the process will blow up.

So what we're looking at is simply too much memory being used in the main process. If the main process is full then event a simple js object will be enough to cause a blow up.

That gives me two places to go:

1. There's clearly a memory leak in production. If every run adds 1mb to the main process heap, then after enough runs a big payload will kill us. But if I can track this down I should be able to restore parity.
2. Is there any way we can isolate any risky areas of processing? Compilation, autoinstall and event handling are all places where we could have memory problems. Worker threads aren't isolated, they share memory - so if a worker thread blows the heap, the whole process goes down (90% sure this is true but worth verifying again). The best defence I think I can see is the push the websocket connection down into the child process. Now all payload processing is shared by the child process, and any issues belonging to the run will be caught there.

This means opening up #544 - an old issue where I put a lot of thought into this. It's quite a big deal, a lot of code needs rearranging.