The problem of how GPU resources are dynamically allocated locally

[lincong8722]

Dear Nextflow author,

I have written an NF script that includes multiple processes (A, B, C) that need to run on a GPU. Currently, I am running the script in a local environment, but I am facing an issue. Since I only have one GPU available, running processes A, B, and C simultaneously results in insufficient GPU memory, causing the processes to fail.

I have already tried setting maxForks to 1 for each process, but this doesn't solve the problem because in the workflow design, there are situations where A, B, and C need to run concurrently, leading to GPU memory shortage.

I would like to know how to address this issue. Is there a way to allocate the GPU resources dynamically, so that only one process runs at a time, utilizing the available GPU memory efficiently?

Thank you for your assistance.

Best regards

[bentsherman]

Hi @lincong8722 , unfortunately the accelerator directive is not supported for the local executor, so you'll have to control it indirectly through another setting.

maxForks was a good idea but doesn't work in this case because it is applied per-process.

executor.queueSize limits the number of parallel tasks across the entire pipeline, so you could set that to 1, but it might be overkill if you have other tasks that can still run.

You could also use the cpus or memory directive to limit the parallelism. For example, if your machine has 8 cpus, you could assign 4 cpus to each GPU process. Nextflow reserves one cpu for itself, so for these three GPU processes, only one would be able to run at a time, while other non-GPU tasks could still run in parallel if they use no more than 3 cpus.