Sequential parallel queues - Question

[atullal]

Implementing many sequential queue which are parallel to each other

Hi,
For my use case I need to implement many parallel queue which will contain jobs which needs to run in sequence. The jobs would be small and not very resource intensive, what is the best way to implement this with bull.js and if it is even possible.

PS - Using Nest js, solution in that would be great otherwise, node js will do.

[manast]

There is nothing fundamentally wrong in scaling queues horizontally as in your diagram, however it may be complicated to handle from admin point of view if you have an unbound number of queues. It will also be difficult to visualize with current UIs. Finally it may be problematic if "n" becomes too large since you may run out of connections to redis.

[atullal]

Thanks for the reply, I plan to create queues on demand and destroy queue when it is drained of jobs. Having an overlord queue to manage creation and deletion of queues as well as managing the number of queues "n" could be a solution here. This type of implementation should be okay with bull, right?

[EmanuelAguilar74]

can you share your insights about your implementation? how scalable is this?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[n1ru4l]

@Stale no

[manast]

Currently I do not have a solution for these kind of requirements in plain javascript, as a Redis module I have one on the drawboard, but I know modules are problematic since not everybody is able to use them.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.