engine: memory leak

[josephjclark]

Worker memory seems to steadily increase over time:

When idle, the worker's memory should really drop down to base levels.

[josephjclark]

More suspicion of this today:

Note how CPU usage drops right off at idle times, but memory never really drops down. It keeps increasing.

This is the tale of woe of pod global-worker-5dd5d4cf5f-9pck8

We actually think ephemeral storage (disk space) is what's done for this pod, which is a discussion to be had elsewhere. But this would be the next bullet in the chamber

[josephjclark]

Memory leak confirmed.

I've set up a local test which sends a large-ish payload. This runs absolutely fine, no problem. But after running it 6 times, suddenly the main worker thread is running out of memory

In other words, it looks like every run is adding Nmb of memory to main worker thread. So eventually even a small payload coming into the worker will kill it, triggering losts.

[josephjclark]

And this is coming out of the engine, not the worker.

[josephjclark]

Ran a little test in the engine and disabled the logic that creates a new worker_thread for each run. Same problem. So it's not the worker thread per se.

[josephjclark]

The inverse is also true: creating a new child process for each run also causes memory errors.

[josephjclark]

Ok and this is super interesting: I've bypassed the worker pool and called the runtime directly from the engine. No worker thread, no child process. And that blows up after 10 or so runs!

So something in the engine or the runtime is hogging a few mb of memory each time and not releasing it.

EDIT: and even more damning, if I skip out the whole runtime and just return a big object, that also blows up! It runs further than the runtime (makes sense) but still fails to release some memory.

So the problem is in the engine but outside the workerpool and indeed the main execution loop.

[josephjclark]

Found a reference to state objects in the engine which isn't getting garbage collected. I'm not really sure why the reference is hanging around but I have found a way to forcibly remove it and encourage garbage collection to do its thing.