Memory Usage

[charliematters]

Hi,

I'm using BullMQ as a queue for handling email sending for a booking system. There's a sender in the main web-app, and a worker in a Render.com Background Worker which takes the order IDs and generates PDF before sending them.

I notice that the worker process consumes 300MB of memory in an idle state, and was wondering whether there was something I could do to reduce this?

My main reason for asking is that Render.com has its cheapest tier of background worker top out at 256MB, and the next tier is 500MB. The actual generation of PDFs takes about 10MB, so I'd hoped that we'd comfortably fit in the cheapest tier.

I tried "bee" as an alternative, and noticed that have the same behaviour, but it does seem quite memory-intensive

[manast]

Can you write a minimal test case where a dummy worker indeed consumes 300Mb? if you do we can look into it, there is no reason why a "dummy" worker would take so much memory,.

[charliematters]

Now I'm feeling foolish. I stripped everything back and it was still using 300MB, then I realised that I was writing it in Typescript and running ts-node ./worker.ts, so all the extra usage was TS. I used tsc to convert it to normal js and just ran it in node - 80MB

[williamolojede]

Hi @charliematters, I'm curious does your background worker read from the same DB as the web app? I'm trying to split worker processes from a web app, and I'm wondering if it makes sense to read/write from/to the same DB the web app uses

[charliematters]

Hi William, yes it does. Both the worker and the (Remix) web app use Prisma connections to the same Postgres database. The only thing that the worker fetches is the detail of the customer order, and it only does that on a successful payment, so it's not really taxing the database. The main reason I moved the email-sending into a worker was because the PDF-creation bit was slow, and moving that off the main thread has hugely eased the burden

[manast]

Now I'm feeling foolish. I stripped everything back and it was still using 300MB, then I realised that I was writing it in Typescript and running ts-node ./worker.ts, so all the extra usage was TS. I used tsc to convert it to normal js and just ran it in node - 80MB

Yes, ts-node is known for being both slower and consuming more memory. The best approach I have found for using Typescript with BullMQ (and in any other project really) is this one: https://blog.taskforce.sh/using-typescript-with-bullmq/
Basically, building both in dev and prod, and running standard JS transpiled code.

[charliematters]

Yeah, that's essentially what I ended up with - a tsc build step and then you just run the produced file with node. It's been ticking away very happily for weeks now!