Test fails when running pipeline on HPC with singularity: "destination is already in the mount point list"

[FloWuenne]

Hi there,

thanks for the great implementation and documentation. I was trying to run the rnaseq pipeline on our HPC this week and I was running into the following issue. This might be related to nextflow but I couldn't figure out how to fix this so any help or pointing me to the right nextflow documentation to fix this would be highly appreciated.

I am running the following command for testing :

nextflow run nf-core/rnaseq -r 1.4.2 -profile test,singularity

I am using the singularity image which is stored in /work/singularity. The pipeline starts fine but then immediately throws this error:

Command error:
  �[33mWARNING:�[0m destination /lustre03/project/6003727/wueflo00/Programs/nf_core_RNAseq_pipeline/work/9f/74b1fede3aabe30fd75ebde7e3a4b3 already in mount list: destination is already in the mount point list

Nextflow report error:

When I enter the singularity shell and run the commands for get_software_versions inside the container, it completes without an error. I think this might be related to a bind path issue but adding the following to my .bashrc did not solve the issue:

export SINGULARITY_BIND="/home/wueflo00/projects/rrg-glettre/wueflo00/Programs/nf_core_RNAseq_pipeline/work,/scratch,/home,/project"

I would really appreciate any help with this since I was unable to fix it for now...

[robsyme]

It's possible that you need to unset the SINGULARITY_BIND variable and tell Nextflow to automount.

On systems that run singularity, I have a nextflow config file (~/.nextflow/config) that includes lines like:

singularity {
    enabled = true
    autoMounts = true
}

Setting autoMounts = true and removing the SINGULARITY_BIND setting might resolve your problem.

[drpatelh]

@robsyme autoMounts should already be set by default when using -profile singularity. We added this to the pipeline template a while back because we were frequently seeing similar errors

rnaseq/nextflow.config

Line 125 in 3b6df9b

singularity.autoMounts = true }

@FloWuenne I wouldn't add the bind paths to your .bashrc. You can add them in like so using a custom config file and pass it to the pipeline using -c custom.config:
https://github.com/nf-core/configs/blob/b23e9d5a9f8772d83acfdf0dfb8c9d68de5d8087/conf/genouest.config#L11

[FloWuenne]

Thanks for your help @robsyme and @drpatelh!

So I have configured a custom config like the following:

singularity {
  enabled = true
  autoMounts = true
  runOptions = '-B /home/wueflo00/projects/rrg-glettre/wueflo00/Programs/nf_core_RNAseq_pipeline/work -B /scratch/wueflo00 -B /home/wueflo00'
}

and run the command like this:

/home/wueflo00/projects/rrg-glettre/wueflo00/Programs/nextflow_20.01.0/nextflow run -c ./custom.config nf-core/rnaseq -r 1.4.2 -profile test,singularity

But still, get the exact same error. Did I specify anything wrong?

The pipeline, by the way, creates all files correctly until the following command for some reason:

echo $(bamCoverage --version 2>&1) > v_deeptools.txt

[robsyme]

What is the contents of v_rseqc.txt?

[FloWuenne]

  File "/home/wueflo00/.local/lib/python3.6/site-packages/numpy/core/__init__.py", line 40, in <module>
    from . import multiarray
  File "/home/wueflo00/.local/lib/python3.6/site-packages/numpy/core/multiarray.py", line 12, in <module>
    from . import overrides
  File "/home/wueflo00/.local/lib/python3.6/site-packages/numpy/core/overrides.py", line 6, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libmkl_rt.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/nf-core-rnaseq-1.4.2/bin/read_duplication.py", line 25, in <module>
    from qcmodule import SAM
  File "/opt/conda/envs/nf-core-rnaseq-1.4.2/lib/python3.6/site-packages/qcmodule/SAM.py", line 19, in <module>
    from bx.binned_array import BinnedArray
  File "/opt/conda/envs/nf-core-rnaseq-1.4.2/lib/python3.6/site-packages/bx/binned_array.py", line 17, in <module>
    from numpy import *
  File "/home/wueflo00/.local/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/wueflo00/.local/lib/python3.6/site-packages/numpy/core/__init__.py", line 71, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using /opt/conda/envs/nf-core-rnaseq-1.4.2/bin/python),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: libmkl_rt.so: cannot open shared object file: No such file or directory

[FloWuenne]

Could it be due to me having numpy installed on in my home and the singularity container mounting my home and replacing the numpy from the container?

I usually would try to circumvent this by running singularity with --no-home but how would I do it in this case?

[robsyme]

That's exactly it. I think that @poquirion has a nice solution, which I'm sure he'll include here.

[poquirion]

the python/conda installed in the container is looking in your ~/.local library installation.

You can prevent that by setting the PYTHONNOUSERSITE variable to something. perhaps in a nextflow.config file in your working directory:

env.PYTHONNOUSERSITE = 1

Also I would recommend to the the dev team in charge of the container to set that in there system. The Dockerfile seems like a good place to set that variable. @drpatelh ?

[FloWuenne]

Indeed running the pipeline with a custom config file as described above that contains this:

//Profile config names for nf-core/configs
singularity {
  enabled = true
  autoMounts = true
  runOptions = '-B /home/wueflo00/projects/rrg-glettre/wueflo00/Programs/nf_core_RNAseq_pipeline/work -B /scratch/wueflo00 -B /home/wueflo00'
}

env {
  PYTHONNOUSERSITE = 1
}

Finally it makes this step of the pipeline work without a problem!

Thanks for the help @poquirion and everyone else!
I would also suggest to include this option as default into the docker container, as most users would assume that the container is isolated and doesn't load python packages from the users home!

[drpatelh]

Great debugging guys 😎 We have indeed added that to the pipeline template already:
https://github.com/nf-core/tools/blob/ebf63095ab222e4df987c4e950ce3839e8784d3e/nf_core/pipeline-template/%7B%7Bcookiecutter.name_noslash%7D%7D/nextflow.config#L81-L84

This will be added to the rnaseq pipeline when we merge the automated template synchronisation PR:
#382

I think the rnaseq pipeline is ripe for a release soon anyway at which point it will be fixed. Thanks for the input guys and glad you got it working @FloWuenne !

Did you figure out whether the bind paths are an essential requirement? If not, I would be tempted to try re-running the pipeline with this custom config:

env {
  PYTHONNOUSERSITE = 1
}

[FloWuenne]

I just tested without the bind paths and worked as well. So for anyone still running into the same issue in the future, the only essential component is to make a custom.config script and add this block to it:

env {
  PYTHONNOUSERSITE = 1
}

[rkacharya]

I'm having a similar error to the issue posted here with both singularity 2.6.1 and 3.5.3 running with the latest sarek pipeline (version 2.6.1)

I've tried adding the PYTHONNOUSERSITE variable as suggested and the issue persists. Any suggestions? Any ideas are greatly appreciated!

[liuxiii]

I have put the env into xin.config and run
nextflow run -c xin.config nf-core/rnafusion/download-references.nf -profile singularity --download_all --outdir . ...

Still got the following error:
FATAL: container creation failed: unabled to /data/rds/DMP/DUDMP/TRANSGEN/xliu/RNAfusion/nfcore/work/e5/eddf65803a86c6b43
4127985522e55 to mount list: destination /data/rds/DMP/DUDMP/TRANSGEN/xliu/RNAfusion/nfcore/work/e5/eddf65803a86c6b4341279855
22e55 is already in the mount point list

[el239]

Has anyone else had luck resolving this? Singularity version 3.8.7-1.el8
Tried supplying a custom config file containing:
env { PYTHONNOUSERSITE = 1 }
Tried different things like disabling autobind and setting the bind variable separately, specifying a path for the variable SINGULARITY_BIND, nothing works. Keep seeing
Command error: WARNING: While bind mounting '/erdos/elavelle/vigna/work/2d/4d1bc7f760018224f722bfde125d25:/erdos/elavelle/vigna/work/2d/4d1bc7f760018224f722bfde125d25': destination is already in the mount point list [ERROR] The server couldn't fulfill the request. [ERROR] Status: 400 Bad Request. Both list of IDs and query_key are empty

[neusmf]

Hi,
I am also experiencing the issue. I have nextflow 22.04.4 and Singularity 3.7

Command error:
WARNING: While bind mounting '/nfs/users/fgebauer/NEUS/BIOINFO/20220711_Public_melanoma_RNAseq/Tang_dataset/SRA_files/work/09/bb1d99726d7a9a333df661d506b95f:/nfs/users/fgebauer/NEUS/BIOINFO/20220711_Public_melanoma_RNAseq/Tang_dataset/SRA_files/work/09/bb1d99726d7a9a333df661d506b95f': destination is already in the mount point list

The .command.sh is the following:
more .command.sh
#!/bin/bash -euo pipefail
echo SRR14593545 > id.txt
sra_ids_to_runinfo.py
id.txt
SRR14593545.runinfo.tsv \

cat <<-END_VERSIONS > versions.yml
"NFCORE_FETCHNGS:SRA:SRA_IDS_TO_RUNINFO":
python: $(python --version | sed 's/Python //g')
END_VERSIONS

The previous solution "env { PYTHONNOUSERSITE = 1 } did not worked"

Thanks!