Running -with-docker and mounting volumes

[nightdew]

Hi, I'm running my pipeline using the -with-docker option and I need to mount some folders from host as the input data.

With the -with-docker option, each process will run in a container on its own. How do I make all the containers mount the same input data folder automatically.

I know there's a docker scope with the runOptions which I can specify in nextflow.config but this would need to specify up front and the input folder which I mount is only specified during run time via command line.

Is there a way that we can specify these dynamically and loaded by all containers?

Thanks in advance!

[bentsherman]

In general, you shouldn't need to mount any volumes explicitly in this way. When Nextflow executes a task, it looks at all of the path inputs to the process, stages those files into the task directory (via link rather than copy), and mounts the task directory into the container.

So as long as you provide all of the input files as path inputs, rather than try to use their absolute path in the process script, it should just work 👍

[nightdew]

Thanks! Issue is I have many pipelines which does not have proper path inputs and is just reading from the path directly passed via val. A lot of refactoring would need on my end for this. Hence, trying to see whether there's an alternative way to do it

[bentsherman]

You can use the containerOptions directive which is basically the same as docker.runOptions but can be per-process and dynamic based on the task inputs:

containerOptions { "-v ${in1} -v ${in2} ..." }

Personally I think refactoring the vals to paths would be less work, but to each their own... 😄

[nightdew]

Yes refactoring would be the next to go as those pipelines are not written by us but this would suffice for now.