add message queue integration

[mahendraHegde]

closes #1645
closes #827

[FelixMalfait]

1 queue = 1 broader category, no? Like "notifications" would be a single queue in my opinion, not one queue topic per notification. The main benefit I see of splitting into multiple queues would be to prioritize/protect one queue from blocking another (e.g. email overload doesn't block payment or whatever more important queue) and for debugging

[mahendraHegde]

correct each queue will handle specific event, i was not looking to have a single queue for all notifications, because some notifications could be more important and some notifcations are triggered very often, so its better to separate queue based on actions or use cases, this way worker can do "handle" the event, for example, we might want to send email when user signs up also might want to create some external accounts, all these could be handled by signup event and worker.

[mahendraHegde]

also if we have separate queue for notifications, producers would have to publish to notification queue separately, apart from the async tasks queue they might be publishing, so it makes publishers to be coupled with the notifications, so I feel we should avoid that.

[FelixMalfait]

Ah yes I get what you mean, something more event-based than action-based.
But even if we do event-based I'm not sure if this isn't too much granularity? Having one queue topic per events feels off to me. But I don't have much experience with queue, I'll let someone else give his opinion!

[magrinj]

I agree with @mahendraHegde on this, but I think the decision is more on product side, because it really depend on our needs.
If we answer yes to this question, multiple queues is the way to go:
Do we need to prioritize certain types of events over others ?

[FelixMalfait]

I feel like the reason it's confusing is because we are mixing two separate things:
(1) building a pub/sub system with events emitters and listeners
(2) building a mechanism for asynchronous job processing

Both are covered by NestJS with core modules, but the core module for (2) adds a dependence on Redis which we don't want. We should stick to rebuild (2), I feel like here we are trying to rebuild both

[mahendraHegde]

@FelixMalfait you are right, My bad, im going to change naming convention to avoid any confusions, intention is not to build the pub/sub but to build a queue for async job processing.
however that makes local driver implementation difficult, nodejs event emiter is pub/sub by default(multiple consumers will get same event), so implementing a local sync queue would require some custom implementation(basically in memory queue), so my question is,is local driver even needed? as mentioned before pg-boss doesnt require any additional setup, so it makes sense to use it as primary driver for local development also wdyt?

[FelixMalfait]

Yes I agree, I'm not sure the local driver is relevant either since, as you say, Postgres is already there.
So maybe there's a little over-engineering in the current solution? It's sometimes frustrating to remove code but we wouldn't have discovered/understood this without your code

Also @magrinj found this package:
https://www.npmjs.com/package/@nestjs-enhanced/pg-boss that is another wrapper. But the code isn't even on Github. It's not something we should use - just referencing for additional inspiration

[mahendraHegde]

@FelixMalfait I have pushed changes, also added example usage in doc, let me know if you still want me to follow the decorator way of handling stuff as done by the above npm library. i feel this approach is simpler.

[mahendraHegde]

as per the worker, we have 2 options

1. run the worker part of the server process.
2. run it as a standalone nest process

i'm planing to go with 2nd approach, so that we get the max of the queue aproach and dont block the main proces's event loop when we do some heavy computation in the future,

[FelixMalfait]

Great work! Thanks

[magrinj]

Really good job, thanks a lot for your work ! 🎉

[magrinj]

I agree with @mahendraHegde on this, but I think the decision is more on product side, because it really depend on our needs.
If we answer yes to this question, multiple queues is the way to go:
Do we need to prioritize certain types of events over others ?

[mahendraHegde]

@magrinj @FelixMalfait i have made the decorator change as we discussed.

[magrinj]

@mahendraHegde Great jobs ! Few changes requested :)
I saw decorators on @nestjs/bull allowing the creation of consumer class, maybe it's something we can add later !

[magrinj]

@mahendraHegde Perfect, I've approved !
Can you just rebase it on main and run a yarn install to fix the lock file please

[github-actions]

CLA

Hello there and welcome to our project!
By submitting your Pull Request, you acknowledge that you agree with the terms of our Contributor License Agreement.
Although we don't have a dedicated legal counsel, having this kind of agreement can protect us from potential legal issues or patent trolls.
Thank you for your understanding.

Generated by 🚫 dangerJS against 3fa0700

[charlesBochet]

A huge thanks! This is deeply appreciated. Merging it!

A note about the queue, I think it should not be restricted to "message" queue. It should be a global queue feature that we could use for anything

[charlesBochet]

I think this should be less specific but we can work on that later :)
We should have a notificationQueue, paymentQueue, ...