Attempt to stabilise worker memory

[josephjclark]

Short Description

This PR removes a possible memory leak which may be triggering Lost runs.

Fixes #1096 (I hope)

Implementation Details

This PR absolutely does something for me locally. Running stress tests on the Worker with a low memory limit, on this branch I seem to have total stability, where on main heap errors are frequent. So that's good.

But will this help us in production? I'm really not sure. This PR may take out the memory leak, but if our workers in production are blowing up on 40% memory capacity, then I don't think we can blame the memory leak?

Here's what I know about production:

• We have some internal QA workflows which are sending large payloads through the app on regular cron jobs. This is straining the system during times of high load, squeezing the margins.
• Those have been disabled for now, but those workflows should absolutely be allowed to run if the services team wants them to

My suspicion remains that that when the worker is processing several large JSON payloads at once in the main thread, and its working memory happens to be quite high, it's running out of heap memory. The problem here may just be that parsing large JSON objects requires a lot of memory. In which case solutions are:

• reduce the payload limit down from 10mb to 4 or 5mb
• Find a more optimal way to process those JSON objects

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

48 hours of worker memory usage. The first 24 hours-ish are before this patch, the second 24hours are after it was deployed