Nextflow doesn't set SINGULARITY_TMPDIR env variable

[laguem]

Bug report

We want to use Nextflow on a HPC cluster with limited space in the /tmp folder (20GB). Nextflow and Singularity are setup using Conda. The /tmp folder fills up pretty fast by unsquashfs (with rootfs-***** folders) when using nf-core/ampliseq for example, and we get the "Write on output file failed because No space left on device" error Halfway throught the pipeline execution. We would like Singularity to use the /scratch partition instead which is over 1TB to uncompress the images.

We tried to set all environment variables we could find to /scratch without success (test_scratch.yml):

env.NXF_TEMP : '/scratch'
env.TMP: '/scratch/'
env.TMPDIR : '/scratch/'
env.SINGULARITY_TMPDIR : '/scratch/'
env.SINGULARITY_LOCALCHACHEDIR : '/scratch/'
env.NXF_SINGULARITY_TMPDIR : '/scratch/'

singularity.enabled : true
singularity.autoMounts:true
singularity.tmpDir :'/scratch/'
singularity.temp : '/scratch/'
singularity.runOptions : '-B /scratch:/tmp/'

NXF_SINGULARITY_CACHEDIR and NFX_CONDA_CACHEDIR are setup in ~/.bashrc to point to the right caches.

When SINGULARITY_TMPDIR is set, Singularity will use /scratch when used alone, but not when using Nextflow.

Expected behavior and actual behavior

Setting env.NXF_SINGULARITY_TMPDIR : '/scratch/' should set SINGULARITY_TMPDIR to /scratch when Nextflow calls Singularity. Singularity should then be using /scratch instead of /tmp to store uncompressed images.

Instead, when Nextflow calls Singularity, all images are still uncompressed in /tmp and the pipeline crashes when /tmp doesn't have enough space available to uncompress the images ("Write on output file failed because No space left on device" error).

Steps to reproduce the problem

nextflow run nf-core/ampliseq -profile test,singularity -params-file test_scratch.yml

::::::::::::::
test_scratch.yml
::::::::::::::
env.NXF_TEMP : '/scratch'
env.TMP: '/scratch/'
env.TMPDIR : '/scratch/'
env.SINGULARITY_TMPDIR : '/scratch/'
env.SINGULARITY_LOCALCHACHEDIR : '/scratch/'
env.NXF_SINGULARITY_TMPDIR : '/scratch/'

singularity.enabled : true
singularity.autoMounts : true
singularity.tmpDir : '/scratch/'
singularity.temp : '/scratch/'
singularity.runOptions : '-B /scratch:/tmp/'

Program output

INFO: Converting SIF file to temporary sandbox...
FATAL: while extracting /isilon/lethbridge-rdc/users/ortegapoloro/singularity_cache/depot.galaxyproject.org-singularity-bioconductor-dada2-1.22.0--r41h399db7b_0.img: root filesystem extraction failed: extract command failed: WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
WARNING: Skipping mount /etc/hosts [binds]: /etc/hosts doesn't exist in container
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
WARNING: Skipping mount proc [kernel]: /proc doesn't exist in container
WARNING: Skipping mount /home/AAFC-AAC/laguem/miniconda3/envs/nextflow/var/singularity/mnt/session/tmp [tmp]: /tmp doesn't exist in container
WARNING: Skipping mount /home/AAFC-AAC/laguem/miniconda3/envs/nextflow/var/singularity/mnt/session/var/tmp [tmp]: /var/tmp doesn't exist in container
WARNING: Skipping mount /home/AAFC-AAC/laguem/miniconda3/envs/nextflow/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container

Write on output file failed because No space left on device

FATAL ERROR:writer: failed to write file /image/root/usr/local/lib/R/library/mgcv/libs/mgcv.so

FATAL ERROR:dir_scan: failed to make directory /image/root/usr/local/lib/R/library/stats/R, because No space left on device
Parallel unsquashfs: Using 128 processors
32150 inodes (37880 blocks) to write

: exit status 1

Environment

• Nextflow version: nextflow version 21.10.6.5660
• Java version: openjdk version "11.0.9.1-internal" 2020-11-04
• Operating system: Linux
• Bash version: GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

Additional context

1. Solution presented here Singularity rusn out of space in /tmp nf-core/rnaseq#566 doesn't work with singularity installed using conda.

2. I can have singularity use /scratch folder if I add SINGULARITY_TMDIR='/scratch' in front of singularity command in nxf_launch function in .command.run script like this

nxf_launch() {
    set +u; env - PATH="$PATH" ${TMP:+SINGULARITYENV_TMP="$TMP"} ${TMPDIR:+SINGULARITYENV_TMPDIR="$TMPDIR"} SINGULARITYENV_NXF_DEBUG=${NXF_DEBUG:=0} SINGULARITY="/scratch" singularity exec -B /isilon/ottawa-rdc/users/shared/abrsu/nf-core_troubleshoot/test_sge_b
iocluster/work -B /home/AAFC-AAC/laguem/.nextflow/assets/nf-core/ampliseq/bin -B "$PWD" /isilon/lethbridge-rdc/users/ortegapoloro/singularity_cache/depot.galaxyproject.org-singularity-bioconductor-dada2-1.22.0--r41h399db7b_0.img /bin/bash
 -c "cd $PWD; eval $(nxf_container_env); /bin/bash /isilon/ottawa-rdc/users/shared/abrsu/nf-core_troubleshoot/test_sge_biocluster/work/02/cd99c270bb548d1b22a17083bfd7d6/.command.run nxf_trace"
}

[bentsherman]

Possible duplicate of #2685

[bentsherman]

@laguem can you try the solutions I proposed in the above issue and let me know if any of them work?

[laguem]

I was able to fix the problem by adding this line to ~/.bashrc

export SINGULARITY_TMPDIR=/singularity_tmpdir

and those lines in the .conf file

singularity {
enabled = true
autoMounts = true
runOptions = '-B $SINGULARITY_TMPDIR:/tmp -B $SINGULARITY_TMPDIR:/scratch'
//Used to allow Singularity to access bashrc variables
envWhitelist = ['SINGULARITY_TMPDIR']
}

Nextflow won't be able to access SINGULARITY_TMPDIR set in the ~/.bashrc file without the envWhitelist line.