-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a queue management system to perform long-running async jobs #1645
Comments
I like this idea of a Postgres-based queue: https://github.com/timgit/pg-boss |
I think with https://github.com/apricote/nest-pg-boss it should be really easy to implement. We would only need to create the db/schema to store the queues and add the setup for each job type (notifications, workspace deletion, ... ). But I think this is would be best handled with the first ticket in need of the queueing system? |
@brendanlaschke I didn't look into it into details but this PR from @martmull could be a good candidate to refactor with queues > #2132 (webhooks/triggers should be performed asynchronously) |
Oh yes interesting one, now I don't think it is that good of a candidate as we are handling an external request - I experimented with a queue and am kind a stuck on one problem:
deleteOneHook(@Args() args: DeleteOneHookArgs): Promise<Hook>
Currently at 2. the hook promise would be returned ( But now adding the job to the queue and handling this queued job at some point later we cannot respond with a correct status? Instead there could be only a "job has been queued" response? As you as a team have a bit more experience as I have - have I missed something here or do you have an idea to solve this by using a different structure? |
@brendanlaschke I'm sorry I think I led you into the wrong direction. The typical use-case for a queue system is when you have to make an http request to an external service. Because you don't have control over the response time of that external service, you don't want to make your own response time dependent of it, so in that case "job has been queued" is indeed considered good enough to return a 200 immediately. Then it's the role of the queue management system to monitor the job execution and let someone knows if things didn't go as expected. |
We are performing various task across the app such as sending notifications which would be better handled asynchronously.
We need to add a system to manage jobs/queues.
NestJS has a built-in package that is a wrapper around BullMQ/Bull.
We should use BullMQ and not Bull (Bull is in maintenance mode)
One additional requirement is that we don't want to make setting up this app locally more complex, so if QUEUE_DRIVE=sync (default value in .env.example) then the job should be processed immediatly as it is created.
The text was updated successfully, but these errors were encountered: