Looking for guidance: controlling throughput based on resource availability

[suchakraborty]

Firstly, thank you all for this wonderful project and all the community that is using it. I decided to use Bull for a project that has come up and I am looking for guidance on how to do so. In short I have jobs with non-linear resource requirements and I would like to control the throughput based on such resource constraints.

Imagine something like a torrent, where one magnet link leads to several downloads and all the downloads need to complete to stitch up a file, which is then further processed. Also, in my case I do not know how many such part downloads I have in the beginning. Obviously, I cannot start downloading more files than I have disk space to wrangle and finish processing.

I imagine the general requirement to control throughput based on some resource availability is common. Can someone point me towards some best practices in implementing this?

I have not dealt with this type of problem, so the best I have is to keep the first job unresolved until I know the resource requirements and if that is more than a heuristic threshold, pause the queue and start a polling management job/feedback from the last job to know when resources become available to unpause the queue.
I suppose the turnkey method would be to generalize limiter in the queue to use some abstraction of the resource constraint. That would be an enhancement.

To be clear, I am not looking to stop a job and push it back in the queue, I understand its not possible and I am perfectly comfortable working around it with failing a job. I am looking for a good way to control throughput, and if Bull is not the way, that is fine too, I am willing to look into other solutions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.