Support for Linux Containers and environment variables

[HenrikBengtsson]

@bwcompton, I think I've added support for what you need. Please install future.batchtools (>= 0.20.0-9024) using:

remotes::install_github("futureverse/future.batchtools", ref="develop")

Then try:

library(future)
plan(future.batchtools::batchtools_slurm, resources = list(
  modules = c("apptainer/latest"),
  envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf,$APPTAINER_BINDPATH"),
  rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript"),
  rscript_args = c("--no-restore", "--quiet", "--no-save")
))

f <- future(sessionInfo())
info <- value(f)
print(info)

[bwcompton]

Not quite.

> library(future)
> plan(future.batchtools::batchtools_slurm, resources = list(
+    modules = c("apptainer/latest"),
+    envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf,$APPTAINER_BINDPATH"),
+    rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript"),
+    rscript_args = c("--no-restore", "--quiet", "-no-save")
+ ))
> f <- future(sessionInfo())
> info <- value(f)
Error: Future (<unnamed-1>) of class BatchtoolsSlurmFuture expired, which indicates that it crashed or was killed.
Post-mortem details:
Future state: ‘running’
Batchtools status: ‘defined’, ‘expired’, ‘submitted’
Slurm job ID: [n=1] ‘43167910’
Slurm 'squeue' job status: <empty>
Slurm 'sacct' job status: 43167910|FAILED|1:0
The last few lines of the logged output:
Setting environment variables: [n=1]
/var/spool/slurm/slurmd/job43167910/slurm_script: line 16: APPTAINER_BINDPATH: unbound variable
In addition: Warning messages:
1: batchtools::waitForJobs(..., timeout = 2592000) returned FALSE 
2: In delete.BatchtoolsFuture(future) :
  Will not remove batchtools registry, because the status of the batchtools was ‘error’, ‘defined’, ‘expired’, ‘submitted’ and future backend argument 'delete' is ‘on-success’: ‘/work/pi_cschweik_umass_edu/marsh_mapping/salt-marsh-mapping/.future/20250914_124140-CG808r/batchtools_1637863878’
> 

[HenrikBengtsson]

Setting environment variables: [n=1]
/var/spool/slurm/slurmd/job43167910/slurm_script: line 16: APPTAINER_BINDPATH: unbound variable

Use:

plan(future.batchtools::batchtools_slurm, resources = list(
   modules = c("apptainer/latest"),
   envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf"),
   rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript"),
   rscript_args = c("--no-restore", "--quiet", "--no-save")
))

instead - that should do it.

Details

You get this error because your original:

export APPTAINER_BINDPATH="/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf,$APPTAINER_BINDPATH"

tries to prepend to any pre-existing value of APPTAINER_BINDPATH, but the latter is not set in your job. The job-script is conservative in order to protect against mistakes (from it using set -e in https://future.batchtools.futureverse.org/reference/batchtools_slurm.html#details) and gives an error saying "unbound variable", which means "not defined".

The proper way to handle this is either to (a) know that APPTAINER_BINDPATH is not already set by the system and just use:

export APPTAINER_BINDPATH="/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf"

or (b) be agile and use it if set, otherwise default to nothing, which is done in Bash as:

export APPTAINER_BINDPATH="/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf,${APPTAINER_BINDPATH:-}"

[bwcompton]

That got us a little farther.

> library(future)
> plan(future.batchtools::batchtools_slurm, resources = list(
+    modules = c("apptainer/latest"),
+    envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf"),
+    rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript"),
+    rscript_args = c("--no-restore", "--quiet", "-no-save")
+ ))
> f <- future(sessionInfo())
> info <- value(f)
Error: Future (<unnamed-1>) of class BatchtoolsSlurmFuture expired, which indicates that it crashed or was killed.
Post-mortem details:
Future state: ‘running’
Batchtools status: ‘defined’, ‘expired’, ‘submitted’
Slurm job ID: [n=1] ‘43172534’
Slurm 'squeue' job status: <empty>
Slurm 'sacct' job status: 43172534|FAILED|1:0
The last few lines of the logged output:
Setting environment variables: [n=1]
 - APPTAINER_BINDPATH=/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf
Load environment modules:
- modules: apptainer/latest
/var/spool/slurm/slurmd/job43172534/slurm_script: line 21: module: command not found
ERROR: future.batchtools job script failed on line 21
In addition: Warning messages:
1: batchtools::waitForJobs(..., timeout = 2592000) returned FALSE 
2: In delete.BatchtoolsFuture(future) :
  Will not remove batchtools registry, because the status of the batchtools was ‘error’, ‘defined’, ‘expired’, ‘submitted’ and future backend argument 'delete' is ‘on-success’: ‘/work/pi_cschweik_umass_edu/marsh_mapping/salt-marsh-mapping/.future/20250914_155841-cZA1oS/batchtools_373884062’

[HenrikBengtsson]

Load environment modules:

• modules: apptainer/latest
  /var/spool/slurm/slurmd/job43172534/slurm_script: line 21: module: command not found

Hmm... weird. In mlr-org/batchtools#314 (comment) you say your admin told you to use:

## Call batchtools inside container
module load apptainer/latest
export APPTAINER_BINDPATH="/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf,$APPTAINER_BINDPATH"

apptainer exec /modules/admin-resources/ood-dev/unity-r_4.4.0.sif Rscript --no-restore --quiet --no-save -e 'batchtools::doJobCollection("<%= uri %>")'

The error message you know get complains about module load apptainer/latest, i.e. the module command is not defined. So, what you've been told to use, is not 100% correct. Or, it could be that there are other instructions for you to submit jobs such that the module system is available. Anyway, let's just try to skip loading of modules and assume that apptainer is available by default. Try:

plan(future.batchtools::batchtools_slurm, resources = list(
   envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf"),
   rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript"),
   rscript_args = c("--no-restore", "--quiet", "--no-save")
))

[bwcompton]

Here we go! (note a typo in --no-save)

> plan(future.batchtools::batchtools_slurm, resources = list(
+    envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf"),
+    rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript"),
+    rscript_args = c("--no-restore", "--quiet", "--no-save")
+ ))
> f <- future(sessionInfo())
> info <- value(f)
> print(info)
R version 4.4.1 (2024-06-14)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future_1.67.0

loaded via a namespace (and not attached):
 [1] crayon_1.5.2           vctrs_0.6.5            cli_3.6.2              rlang_1.1.4            stringi_1.8.4          brew_1.0-10            data.table_1.15.4      glue_1.7.0             batchtools_0.9.17-9000
[10] prettyunits_1.2.0      listenv_0.9.1          backports_1.5.0        hms_1.1.3              fansi_1.0.6            rappdirs_0.3.3         tibble_3.2.1           base64url_1.4          progress_1.2.3        
[19] lifecycle_1.0.4        compiler_4.4.1         codetools_0.2-20       fs_1.6.4               pkgconfig_2.0.3        rstudioapi_0.16.0      digest_0.6.35          R6_2.5.1               utf8_1.2.4            
[28] pillar_1.9.0           parallelly_1.45.1      parallel_4.4.1         magrittr_2.0.3         checkmate_2.3.3        tools_4.4.1            withr_3.0.0            globals_0.18.0   

The other part of those instructions I didn't mention are in launching Rstudio OnDemand. The field [Advanced] Override Rstudio Image Location needs to be /modules/admin-resources/ood-dev/unity-r_4.4.0.sif. Dunno if that's the missing link?

[HenrikBengtsson]

Here we go!

Woohoo!

... (note a typo in --no-save)

I've fixed this in all my comments above, just in case someone comes to this later and tries to cut'n'paste from my instructions.

The other part of those instructions I didn't mention are in launching Rstudio OnDemand. The field [Advanced] Override Rstudio Image Location needs to be /modules/admin-resources/ood-dev/unity-r_4.4.0.sif.

I see; You're accessing the system via OnDemand [in the browser] that your HPC environment provides. From there you launch an RStudio interactive job. You're using a custom R/RStudio Apptainer image. Then, you run all the above R code from that.

... Dunno if that's the missing link?

Not sure I'm following. It looks like it's all working now. Are you referring to the "module: command not found" we got above?

[HenrikBengtsson]

BTW, it's only now that I realize you can drop all those Rscript arguments that you've been told to use, because when using Rscript, contrary to R, --no-restore, --quiet, and --no-save are all redundant, i.e. has no effect. So, now we're down to:

plan(future.batchtools::batchtools_slurm, resources = list(
   envs = c(APPTAINER_BINDPATH = "/run/munge,/var/run/munge,/etc/slurm,/var/spool/slurm/slurmd/conf-cache/slurm.conf"),
   rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript")
))

And I'm pretty sure it would even work with just:

plan(future.batchtools::batchtools_slurm, resources = list(
   rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript")
))

[bwcompton]

Both of these new shorter versions work too. I'll give these a try with batchtools this week.

My comment about the missing link was in reference to your comment about module command not being found. No matter, it's working now.

[HenrikBengtsson]

Both of these new shorter versions work too.

Nice! Depending on how your Apptainer image is defined, it might even work with just:

plan(future.batchtools::batchtools_slurm, resources = list(
   rscript = c("/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript")
))

and possibly even (although less likely) with:

plan(future.batchtools::batchtools_slurm, resources = list(
rscript = "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif"
))

... I'll give these a try with batchtools this week.

So, if you want to go bare-bone batchtools right now, you need to use your own custom cluster functions via future.batchtools::makeClusterFunctionsSlurm2 and then pass that to batchtools when setting it up.

With this success story, and hopefully with more from others soon, I'm closer to proposing a patch to batchtools for the original problem (mlr-org/batchtools#314).

Until fixed in batchtools, and depending on your use case, I'd recommend that you give futureverse a spin. Already now, you can try:

library(future)
library(future.apply)

plan(future.batchtools::batchtools_slurm, resources = list(
   rscript = c("apptainer", "exec", "/modules/admin-resources/ood-dev/unity-r_4.4.0.sif", "Rscript")
))

y <- future_lapply(1:10, function(x) {
  Sys.sleep(10) ## emulate slowness
  sqrt(x)
})

and those ten slow sqrt(x) calculations will be done in one Slurm job each. You can of course do much heavier processing too. You can think of future.batchtools as a simplified abstraction on-top of batchtools.

[bwcompton]

Thanks! I'm glad this is working. I've got a batchtools patch that's fine for now, but it'll be good to just use the package, as I plan to share my package that calls it when I get around to doing a bit of polishing.

I'm amped about future.batchtools! I used future for the first time last year, in a Shiny app that does a fair bit of processing of GIS data that takes a few seconds, too long for a web app. I send the processing off into the future and let the user focus on entering a title and description for the report. It works great.

[HenrikBengtsson]

And thanks for your help here. It'll help lots of others!