Various ways to use background workers/queues with Blitz

[flybayer]

We're starting to have people use Blitz with background workers and queues. At some point we want Blitz to have this functionality backed in, or at least really nicely integrated.

So this is a thread for people to post what they are doing and/or ask questions.

---

The first one I've seen is by @johncantrell97 on how to use BullMQ with Blitz when deploying it to a server (not serverless):

https://medium.com/@johncantrell97/background-workers-in-blitz-31fe5f3b6ce6

[davidharting]

I am not deeply familiar with Blitz internals and am by no means an expert in the area of background workers. So as someone who is very excited to use Blitz in production, I am wondering if you have the goal to make it so that developers can write jobs once and then easily switch the backing store via configuration / adapter libraries (similar to ActiveJob in Rails)?

---

The question comes to mind because I think it would unlock a lot for Blitz developers as potentially all of these could happen without any worker code changes:

1. Change between serverless and serverful deployments.
2. Change the backing store for your jobs as your application grows
   For instance, from in memory, to Postgres, to Redis, to SQS
3. Allow for jobs to be executed synchronously
   This could be a useful configuration option for running end-to-end tests in CI.

I believe that the first point is a particular challenge for frameworks that seek to be equally strong at serverful and serverless deployments. While cloud providers have their own recommendations (e.g., SQS + Lambda), 2nd tier serverless cloud providers such as Vercel and Netlify do not have standard background worker implementations yet.

Perhaps in some cases, the adapters / worker implementations would only need to map from the Blitz API to popular libraries. For instance, the serverful Redis adapter could be a (hopefully) thin wrapper around BullMQ.

[flybayer]

Hi David! Thank you for this. I definitely want a standardized job interface like ActiveJob. And basically everything else you described!

I haven't yet spent any time thinking about how to make this all work, but anyone is welcome to do some R&D on this and report their findings! It'll still be some time before I can give any attention to this (currently focused on authentication!)

[HeyGarrison]

After spending the night trying to find something that fits with postgres I found this: https://github.com/timgit/pg-boss

I'm a little concerned about not going through prisma, but the DB directly. So it might be that I end up having to convert pg-boss to use prisma instead? Does anyone know if this is a bad idea? :D

[flybayer]

Having other things connect to your db directly is fine, but you probably need to run prisma introspect afterwards so your schema is updated with whatever pgboss adds.

However, I'd really recommend not using your database for queuing. That can cause a lot of problems (I know from experience!). You're always better off using redis or something else for background processing.

[HeyGarrison]

I just have this image in my head of only having to run a node server and a DB, but I guess that might be too idealistic. ;)

[flybayer]

You could get away with that for small things without many users. Just depends on what you are doing :)

[sehmaschine]

Disclaimer: I've not been using Blitz so far, but I've recently listened to an interesting Podcast (FSJam) and I'm generally trying to simplify our architecture. I'm a bit surprised background tasks and job queues are not yet supported – when reading the manifesto, my assumption was that this topis is already covered.

So here's my usecase. We usually have two different kind of background tasks with our apps:

1. Background tasks which are independent of a user action, e.g. sending daily mails or doing backups. These tasks are usually part of a distinct application/microservice – mainly because we can use a small server instance to handle the workload independent of the main app. But I'm happy to integrate this with the main application. Triggered by a cronjob (or something similar).
2. Long running jobs which have to run in the background, e.g. generating PDFs. These tasks are part of the main app and are triggered by a user action (e.g. clicking a button). Then the user either stays on the page and needs updates on the process or the user leaves the page and we will inform her about the process later on. With this usecase, we need to update a single job with the current state (e.g. 23 of 80 PDFs created) regularly.

We currently use RabbitMQ/Celery, but the setup and maintenance is quite complicated. Something more lightweight would be awesome. I've been looking at Bull, but it's not convincing. Quirrel and Repeater don't seem to handle usecase 2 very well. graphile-worker looks quite ok.

[sehmaschine]

I really wish we had better answers for this by now, but we all have limited time and can only do so much! 😅 We'll get there :)

Sorry, that probably came across a bit rude. No offense meant.

Why do you say that? I think both should handle use case 2 very well, but you may need to run your app as a server instead of on Vercel for example.

Quirrel for example has a price structure which is based on API calls. With usecase 2 that is not an option.
Answer from Repeater on usecase 2: it is totally doable, but you may need some creative architecture.

So yes, it is possible with both apps. I'll give Repeater a try and will share my experience ...

[flybayer]

With this usecase, we need to update a single job with the current state (e.g. 23 of 80 PDFs created) regularly.

Quirrel for example has a price structure which is based on API calls. With usecase 2 that is not an option.

I guess I'm not following here. How I envision this working is user clicks button, this triggers 1 Quirrel call which starts a job that loops through and processes 80 PDFs. After each pdf processed, the job updates the status in the DB. The frontend then polls the current processed status to update the user that 23 of 80 have been processed.

[sehmaschine]

User clicks button > Quirrel call (start job) > Quirrel calls our API endpoint (starts generating PDFs) > Each loop updates the job (X of Y PDFs processed) by calling Quirrel > Frontend calls job status (Quirrel call)

If I'm using the DB (or Redis) in order to process the job updates, I don't need something like Quirrel or Repeater in the first place. Unless I'm missing something very obvious here.

[Skn0tt]

Hi @sehmaschine! Quirrel focuses on the scheduling (as in "wake up your application when a job is due") and distribution (as in "spread out work across multiple lambdas to speed up") part of job queueing, and is very similar to Repeater in that regard.
It does not aim to keep track of state like "number of PDFs processed", and there are no immediate plans to change that.

I'd recommend handling the PDF-Progress state outside of Quirrel. If it's ephemeral, you could just publish updates using an approach similar to the one demonstrated here: https://demo.quirrel.dev/distribute

On the pricing model: I decided to go with API-Call-Based pricing since it's closely-tied to my own operating cost. I totally get that this doesn't work for some applications - feel free to reach out to info@quirrel.dev and I'm sure we'll figure out something that works for both of us.
And: Quirrel is MIT-Licensed, so you're totally free to host your own instance!

[sehmaschine]

Thanks @Skn0tt. Guess I might need to rethink the architecture.

[labe-me]

I usually use Bull too (https://github.com/OptimalBits/bull).

Typescript, baked by redis, easy to configure and use, gives enough control on jobs (promises, watch, cronjobs, scheduling, retry, etc.), you can put your workers in separate processes, control execution parallelism and you can even use bull-board (https://github.com/vcapretz/bull-board) to get a quick view of what's going on.

[Skn0tt]

Just wanted to throw Quirrel into the ring: https://github.com/quirrel-dev
Quirrel was developed in response to this very problem in Blitz.js.
It's MIT-Licensed, there's a managed version and it has a really nice DX.

Especially for serverless deployments, I think it's a very nice fit.

Disclaimer: I'm the author ☺️

[labe-me]

Interesting, does it support parallelism limits (wait for job to complete before pushing the next one) ?

[Skn0tt]

It does! You can set the exclusive: true option to forbid concurrency. It can beset per-job or per-queue.

[PlanetIrata]

Hi,

Just want to know if there's any news regarding background intensive tasks with Blitz JS as stated in the landing page:

Blitz is set up for server intensive tasks like sending emails, cron jobs, background processing, generating PDFs, etc. Currently we have minimal backend guides, but are working on developing more patterns and adding more docs.

I don't really understand the purpose of the quirrel solution suggested above... it seems to be a cron/queue app which call our own API, so any intensive task like sending 10.000 emails or generating PDFs would lock the main app thread anyway.

To keep our blitz app monolithic without needing to develop a backend for intensive tasks, we need a solution to use Node's worker_threads and/or an external thread pool manager.

The ability to run a file _server.js when the server starts would be enough to init/create the worker(s) in most cases.

Any progress on this point ?

Thx

[maluramichael]

I would be fine with a simple way of calling the blitz console and some mutations. This way i could send emails via my sendMail mutation triggered by a cronjob.

Is there no easier way than this? https://javascript.plainenglish.io/how-i-automated-my-weekly-newsletter-using-blitz-js-and-cron-4c38ec244762