[Feature Request] Worker Rate Limit without Duration

[lukepolo]

It would be nice to be able to rate limit without a duration for a group

For example:
A job that downloads something from a remote machine as fast as possible but only have 5 concurrent downloads available.

Currently we are using redis to keep track of the number of jobs currently running and then delay them if there's 5 already running.

Im sure theres a better way , and open to ideas

[manast]

Ok, so more like a global concurrency parameter instead of the current per worker concurrency right?

[lukepolo]

Yup!

[manast]

Note for future implementer: the challenge here is not so much limiting the amount of active jobs at any given time, but how to "wake" the workers that will be sleeping while the limit has been reached. We want the workers to wake up as soon as they are needed and in a robust way so that they "always" start working as soon as the active number goes below the threshold. Events could be used but the biggest drawback is that events will be sent to all sleeping workers, if there is a large pool of workers this will imply that the workers will all try to start processing new jobs but maybe only a handful will succeed, creating unnecessary stress on Redis.

[xiaosha007]

A global concurrency feature would be useful for me as well

[nemphys]

+1 on this one, super useful if implemented.

[hicaro]

+1 on this. I agree this would be extremely useful

[manast]

Closing this issue as this has been already implemented for a while: https://docs.bullmq.io/bullmq-pro/groups/concurrency