Loading environment modules

[azzaea]

Hello,

I'm wondering if cromwell has support for loading environment modules somehow? I need to port the same workflow between AWS (so, the docker runtime attribute is handy) and a slurm cluster (which uses module environment).

Can they be specified as a runtime parameter for example? Can they be part of the configuration file to cromwell? If yes, any suggestions as to how?

Thank you,
Azza

[wleepang]

Hi @azzaea,

The AWS backend for Cromwell integrates with AWS Batch for job scheduling and execution. As such it pretty much only uses tasks that use Docker containers.

My understanding is that Cromwell can be configured with multiple backends (e.g. AWS and FilesystemLocal) and that tasks can be parameterized via inputs to the workflow to choose which backend it runs on.

[azzaea]

Hi @wleepang

Yes, you are right. The problem I have is particularly when using a slurm backend, as I don't find an easy way to load environment modules except to actually modify the individual command section of each task definition in my workflow. This is inconvenient because when I'm running the same pipeline on AWS batch, there are no environment modules, so my tasks fail unless I explicitly remove all the module load <module name> from the command part of each task.

I would like to switch back and forth between cloud and cluster without having to touch the pipeline script (i.e. individual tasks) itself. Put differently, can I specify a runtime attribute called module much like the docker attribute, and then somehow modify the backend configuration settings to have cromwell load this module?

Did I explain myself better this time?

[wleepang]

I'll largely defer to @cjllanwarne, @mcovarr, or @danbills on the specifics, but it seems that you could specify the runtime attribute you need and how to interpret it by customizing the SLURM backend in the config:

https://cromwell.readthedocs.io/en/stable/backends/SLURM/

If I'm reading the docs correctly, it might be possible to inject your module load command into the --wrap argument.

[wkc1986]

I ran into a related issue while running the ENCODE HiC pipeline via Caper on SLURM. I opened an issue there too. On our HPC I need to module load cuda/11.7 to use the nvcc binary. I tried --wrap='module load cuda/11.7' but while this gets passed to the sbatch command it returns a script argument not permitted error, possibly because module isn't a binary but a bash function? Are there any other options for using Caper/Cromwell with the module system?