Singularity missing? #68

apeltzer · 2018-10-31T08:33:58Z

> nextflow run nf-core/eager -with-singularity /apps/containers/nf-core-eager.simg  --pairedEnd --reads "/path/sample_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome ${ref} --saveReference true  -name textEager3
N E X T F L O W  ~  version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: 897fca777a [master]
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
=========================================
 nf-core/eager v2.0.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.0.0
Run Name       : textEager3
Reads          : /path/sample_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/user/eager/work
Container Engine: singularity
Container      : /apps/containers/nf-core-eager.simg
Current home   : /home/user
Current user   : user
Current path   : /home/user/fastdir/eager
Script dir     : /home/user/.nextflow/assets/nf-core/eager
Config Profile : standard
=========================================
[warm up] executor > SLURM
[c5/898ed5] Submitted process > get_software_versions
[e1/cadfbc] Submitted process > fastqc (sample)
[06/d3a688] Submitted process > adapter_removal (sample)
[7f/7a6a9a] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Process `get_software_versions` terminated with an error exit status (127)

Command executed:

  echo 2.0.0 &> v_pipeline.txt
  echo 0.32.0 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  bwa &> v_bwa.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  AdapterRemoval -version  &> v_adapterremoval.txt 2>&1 || true
  picard MarkDuplicates --version &> v_markduplicates.txt  2>&1 || true
  dedup -v &> v_dedup.txt 2>&1 || true
  preseq &> v_preseq.txt 2>&1 || true
  gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
  vcf2genome &> v_vcf2genome.txt 2>&1 || true
  fastp --version &> v_fastp.txt 2>&1 || true
  bam --version &> v_bamutil.txt 2>&1 || true
  qualimap --version &> v_qualimap.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  127

Command output:
  (empty)

Command error:
  env: singularity: No such file or directory

Work dir:
  /fast/users/user/eager/work/c5/898ed52fe4a02f48caaf2f0130963b

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[e1/cadfbc] NOTE: Process `fastqc (sample)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete

Originally posted by @yassineS in #67 (comment)

The text was updated successfully, but these errors were encountered:

apeltzer · 2018-10-31T08:36:56Z

Can you confirm that singularity is installed properly?


Command exit status:
  127

Command output:
  (empty)

Command error:
  env: singularity: No such file or directory

This suggests its not installed on your cluster and Nextflow can't call singularity properly. Maybe some missing module activation? You can by the way also simply run like this:

-profile singularity

without requiring the -with-Singularity option at all. The image will then be pulled automatically and stored in the work directory of the current run.

yassineS · 2018-10-31T09:34:13Z

I have rerun with the changes suggested above and on the other thread, I still get the 127 error:

N E X T F L O W  ~  version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: 4ac6d7a330 [dev]
WARN: Access to undefined parameter `readPaths` -- Initialise it to a default value eg. `params.readPaths = some_value`
=========================================
 nf-core/eager v2.0.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name       : textEager3
Reads          : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/a1222423/eager/work
Container Engine: singularity
Container      : shub://nf-core/eager:latest
Current home   : /home/a1222423
Current user   : a1222423
Current path   : /home/a1222423/fastdir/eager
Script dir     : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : singularity
=========================================
[warm up] executor > SLURM
[da/19ef48] Submitted process > fastqc (NGSD2_r2)
[4b/c2f569] Submitted process > get_software_versions
[38/a1632e] Submitted process > adapter_removal (NGSD2_r2)
[10/71f24c] Submitted process > output_documentation
ERROR ~ Error executing process > 'fastqc (NGSD2_r2)'

Caused by:
  Process `fastqc (NGSD2_r2)` terminated with an error exit status (127)

Command executed:

  fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /fast/users/a1222423/eager/work/da/19ef48c42b7eda36c089592a5536f6: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /fast/users/a1222423/eager/work/da/19ef48c42b7eda36c089592a5536f6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
[nf-core/eager] Pipeline Complete

apeltzer · 2018-10-31T09:43:47Z

Is singularity present on that machine?

yassineS · 2018-10-31T09:45:34Z

Yes!

$ singularity --version
2.5.2-dist

apeltzer · 2018-11-02T13:23:18Z

Okay, it looks a bit weird because the error here:


  Process `fastqc (NGSD2_r2)` terminated with an error exit status (127)

Command executed:

  fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz

Command exit status:
  127

Suggests via exit code = 127 that fastqc can't be found.

Also, this here:

Config Profile : singularity
=========================================
[warm up] executor > SLURM

states that you use a SLURM scheduler but didn't specify a profile for this. Do you use your own local configuration to submit jobs to a scheduler? In case so, I can add a configuration for your local hpc environment that should load required modules e.g. singularity automatically, as it could be the case it doesn't load singularity on cluster nodes right now.

yassineS · 2018-11-04T05:33:38Z

There’s no special configuration required to launch a slurm job on our HPC. Does NF require any special configuration? Y Cheers, Yassine Souilmi

…

________________________________ From: Alexander Peltzer <notifications@github.com> Sent: Friday, November 2, 2018 11:53:19 PM To: nf-core/eager Cc: Yassine Souilmi; Mention Subject: Re: [nf-core/eager] Singularity missing? (#68) Okay, it looks a bit weird because the error here: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) Command executed: fastqc -q NGSD2_r2_1.fastq.gz NGSD2_r2_2.fastq.gz Command exit status: 127 Suggests via exit code = 127 that fastqc can't be found. Also, this here: Config Profile : singularity ========================================= [warm up] executor > SLURM states that you use a SLURM scheduler but didn't specify a profile for this. Do you use your own local configuration to submit jobs to a scheduler? In case so, I can add a configuration for your local hpc environment that should load required modules e.g. singularity automatically, as it could be the case it doesn't load singularity on cluster nodes right now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#68 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABr4IVBd7-pBmQR2c2S2Ptr9aIBQ3HL6ks5urEdHgaJpZM4YDlNE>.

apeltzer · 2018-11-04T09:57:04Z

Depends on the cluster quite a bit! Your error message could be explained by submitting jobs to slurm / cluster nodes, but without loading a Singularity module there and thus the tools inside the container cannot be resolved/used (error 127).

How do you ensure that Singularity is up and running on the cluster? Do you load a module or something similar?

yassineS · 2018-11-04T10:00:39Z

Yes! I load the module and double check that it’s callable. Never faced an issue where a module needs to be loaded on the compute nodes. I have been doing some digging, and the problem might be similar to (nf-core/rnaseq#29 (comment) <nf-core/rnaseq#29 (comment)>). But I am not really sure. Thanks for your help.

…

On 4 Nov 2018, at 8:27 pm, Alexander Peltzer ***@***.*** ***@***.***>> wrote: Depends on the cluster quite a bit! Your error message could be explained by submitting jobs to slurm / cluster nodes, but without loading a Singularity module there and thus the tools inside the container cannot be resolved/used (error 127). How do you ensure that Singularity is up and running on the cluster? Do you load a module or something similar? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABr4IVthTf9vtO6PrNkNmGvO8eWjJqCLks5urrnxgaJpZM4YDlNE>.

apeltzer · 2018-11-04T10:03:44Z

Hey!
Can you let me know how you load the module and maybe just give me a name for the cluster? I can then come up with a -profile <your_cluster_name> that should run 👍

module load modules/singularity/2.5.2

yassineS · 2018-11-04T10:05:36Z

The cluster is called Phoenix, and I load the module using the following command: module load Singularity/2.5.2-GCC-5.4.0-2.26

…

On 4 Nov 2018, at 8:33 pm, Alexander Peltzer ***@***.*** ***@***.***>> wrote: Hey! Can you let me know how you load the module and maybe just give me a name for the cluster? I can then come up with a -profile <your_cluster_name> that should run 👍 module load modules/singularity/2.5.2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABr4ITNGP7cNkVm8gX1dw-PxLr51TWF8ks5urruBgaJpZM4YDlNE>.

apeltzer · 2018-11-04T10:13:41Z

Thanks a bunch!

I added a config profile for you to test - its called -profile phoenix and is currently only found in the dev branch, therefore run your command using this:

nextflow run [your previous options] -profile phoenix -r dev

yassineS · 2018-11-04T10:35:11Z

Unfortunately, this didn't work. It first returned the following error:

WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /fast/users/a1222423/eager/work/singularity
Pulling Singularity image docker://nfcore/eager:latest [cache /fast/users/a1222423/eager/work/singularity/nfcore-eager-latest.img]
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Failed to pull singularity image
  command: singularity pull --name nfcore-eager-latest.img docker://nfcore/eager:latest > /dev/null
  status : 255
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
    WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
    ERROR: You must install squashfs-tools to build images
    ABORT: Aborting with RETVAL=255
    ERROR: pulling container failed!



 -- Check '.nextflow.log' file for details
[nf-core/eager] Pipeline Complete

Which I tried to circumvent by downloading the image manually, and than rerunning the workflow:

$ singularity pull --name nfcore-eager.img shub://nf-core/eager:latest
Progress |===================================| 100.0%
Done. Container is at: /fast/users/a1222423/eager/nfcore-eager.img
$
$
$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true  -profile phoenix -with-singularity nfcore-eager.img  -name textEager3
N E X T F L O W  ~  version 0.32.0
Project `nf-core/eager` currently is sticked on revision: dev -- you need to specify explicitly a revision with the option `-r` to use it
[a1222423@l03 eager]$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true  -profile phoenix -with-singularity nfcore-eager.img -r dev -name textEager3
N E X T F L O W  ~  version 0.32.0
Launching `nf-core/eager` [textEager3] - revision: a6bc45406e [dev]
=========================================
 nf-core/eager v2.1.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name       : textEager3
Reads          : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/a1222423/eager/work
Container Engine: singularity
Container      : nfcore-eager.img
Current home   : /home/a1222423
Current user   : a1222423
Current path   : /home/a1222423/fastdir/eager
Script dir     : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : phoenix
=========================================
[warm up] executor > slurm
[2b/b9706a] Submitted process > get_software_versions
[8e/3ae677] Submitted process > fastqc (NGSD2_r2)
[68/3dd262] Submitted process > adapter_removal (NGSD2_r2)
[c1/03664f] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Process `get_software_versions` terminated with an error exit status (127)

Command executed:

  echo 2.1.0dev &> v_pipeline.txt
  echo 0.32.0 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  bwa &> v_bwa.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  AdapterRemoval -version  &> v_adapterremoval.txt 2>&1 || true
  picard MarkDuplicates --version &> v_markduplicates.txt  2>&1 || true
  dedup -v &> v_dedup.txt 2>&1 || true
  preseq &> v_preseq.txt 2>&1 || true
  gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
  vcf2genome &> v_vcf2genome.txt 2>&1 || true
  fastp --version &> v_fastp.txt 2>&1 || true
  bam --version &> v_bamutil.txt 2>&1 || true
  qualimap --version &> v_qualimap.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /fast/users/a1222423/eager/work/2b/b9706a04822a46b640fa5fc67d10c6: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /fast/users/a1222423/eager/work/2b/b9706a04822a46b640fa5fc67d10c6

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[8e/3ae677] NOTE: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete

apeltzer · 2018-11-04T10:42:08Z

Ok, the first issue can be resolved by installing squashfs-tools on the server once. Maybe ask IT to do so, we had to as well ;-)

The second issue seems indeed to be related to the link you posted: nf-core/rnaseq#29 (comment)

It looks like your cluster admin (?) didn't specify certain configuration parameters for singularity to access certain paths on your system. For example, we have a system where we can only run images and access files stored under /beegfs/<....> and this has to be configured in the Singularity installation once for all users.

I can help you in getting that up and communicate it appropriately if you want to, but we'd need to have a shared screen session / Skype to talk about this ...

yassineS · 2018-11-04T10:46:26Z

Sounds good to me, and thank you again for your support!

apeltzer · 2018-11-04T10:53:02Z

No worries, happy to have some people testing the pipeline and giving feedback like this - thanks for testing and send me a DM/email if we should do a screen share or similar session to figure out whats going on.

apeltzer · 2018-11-06T13:15:41Z

Heads up to document for future cases what goes on:

Head node uses a different home than worker nodes on cluster, this leads to entire work directories not being readable, thus the error of missing things. Had similar issues on another cluster before, where the homes were linked as

/home-link/<yourusername> and then on the node they were present as /home/<yourusername>...

uoabowen · 2018-11-08T03:27:33Z

Hi @apeltzer,

I am the guy who installed and configured singularity on Phoenix which @yassineS is using.

We use $SINGULARITY_BINDPATH to bind directory, see https://www.sylabs.io/guides/2.6/user-guide/bind_paths_and_mounts.html#binding-with-overlay for details

This is my test script

/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ echo $SINGULARITY_BINDPATH 
/fast/users/a1222423/eager/work,/data
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ cat test-bowen.sh 
#!/bin/bash -x
#SBATCH -p test
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --mem=1G
#SBATCH --time=15:00

ml Singularity/2.5.2-GCC-5.4.0-2.26

export PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:$PATH

singularity exec /fast/users/a1222423/eager/work/singularity/nf-core-eager.img /bin/bash -c "pwd; cd $PWD; pwd; ls -al; /bin/bash /fast/users/a1222423/eager/work/8b//12a213d46743cf1ccdd3e71e23f23f/.command.stub"

This is what I got

[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ cat slurm-12963871.out
+ LMOD_PKG=/usr/share/lmod/lmod
+ LMOD_DIR=/usr/share/lmod/lmod/libexec
+ LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod
+ MODULESHOME=/usr/share/lmod/lmod
+ export LMOD_PKG
+ export LMOD_CMD
+ export LMOD_DIR
+ export MODULESHOME
+ LMOD_VERSION=6.3.4
+ export LMOD_VERSION
+ '[' : '!=' : ']'
+ unalias ml
+ true
++ /usr/bin/tr '[:upper:]' '[:lower:]'
++ echo YES
+ export_module=yes
+ '[' -n '4.2.46(2)-release' -a yes '!=' no ']'
+ export -f module
+ export -f ml
+ unset export_module
+ '[' -n '' ']'
+ '[' 4 -ge 3 ']'
+ '[' -r /usr/share/lmod/lmod/init/lmod_bash_completions ']'
+ '[' -n '' ']'
+ ml Singularity/2.5.2-GCC-5.4.0-2.26
++ /usr/share/lmod/lmod/libexec/ml_cmd Singularity/2.5.2-GCC-5.4.0-2.26
+ eval module load ''\''Singularity/2.5.2-GCC-5.4.0-2.26'\'''
++ module load Singularity/2.5.2-GCC-5.4.0-2.26
+++ /usr/share/lmod/lmod/libexec/lmod bash load Singularity/2.5.2-GCC-5.4.0-2.26
++ eval 'LMOD_DEFAULT_MODULEPATH="/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core";' export 'LMOD_DEFAULT_MODULEPATH;' 'M
+++ LMOD_DEFAULT_MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export LMOD_DEFAULT_MODULEPATH
+++ MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export MODULEPATH
+++ CPATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/include:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/include:/apps/software/binutils/2.26-GCCcore-5.4.0/include:/apps/software
+++ export CPATH
+++ EBDEVELBINUTILS=/apps/software/binutils/2.26-GCCcore-5.4.0/easybuild/binutils-2.26-GCCcore-5.4.0-easybuild-devel
+++ export EBDEVELBINUTILS
+++ EBDEVELGCC=/apps/software/GCC/5.4.0-2.26/easybuild/GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELGCC
+++ EBDEVELGCCCORE=/apps/software/GCCcore/5.4.0/easybuild/GCCcore-5.4.0-easybuild-devel
+++ export EBDEVELGCCCORE
+++ EBDEVELLIBARCHIVE=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/easybuild/libarchive-3.3.1-GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELLIBARCHIVE
+++ EBDEVELSINGULARITY=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/easybuild/Singularity-2.5.2-GCC-5.4.0-2.26-easybuild-devel
+++ export EBDEVELSINGULARITY
+++ EBROOTBINUTILS=/apps/software/binutils/2.26-GCCcore-5.4.0
+++ export EBROOTBINUTILS
+++ EBROOTGCC=/apps/software/GCCcore/5.4.0
+++ export EBROOTGCC
+++ EBROOTGCCCORE=/apps/software/GCCcore/5.4.0
+++ export EBROOTGCCCORE
+++ EBROOTLIBARCHIVE=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26
+++ export EBROOTLIBARCHIVE
+++ EBROOTSINGULARITY=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26
+++ export EBROOTSINGULARITY
+++ EBVERSIONBINUTILS=2.26
+++ export EBVERSIONBINUTILS
+++ EBVERSIONGCC=5.4.0
+++ export EBVERSIONGCC
+++ EBVERSIONGCCCORE=5.4.0
+++ export EBVERSIONGCCCORE
+++ EBVERSIONLIBARCHIVE=3.3.1
+++ export EBVERSIONLIBARCHIVE
+++ EBVERSIONSINGULARITY=2.5.2
+++ export EBVERSIONSINGULARITY
+++ LD_LIBRARY_PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/lib:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib:/apps/software/binutils/2.26-GCCcore-5.4.0/lib:/apps/software/G
+++ export LD_LIBRARY_PATH
+++ LIBRARY_PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/lib:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib:/apps/software/binutils/2.26-GCCcore-5.4.0/lib:/apps/software/GCCc
+++ export LIBRARY_PATH
+++ LMOD_DEFAULT_MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export LMOD_DEFAULT_MODULEPATH
+++ LOADEDMODULES=GCCcore/5.4.0:binutils/2.26-GCCcore-5.4.0:GCC/5.4.0-2.26:libarchive/3.3.1-GCC-5.4.0-2.26:Singularity/2.5.2-GCC-5.4.0-2.26
+++ export LOADEDMODULES
+++ MANPATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/share/man:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/share/man:/apps/software/binutils/2.26-GCCcore-5.4.0/share/man:/apps/
+++ export MANPATH
+++ MODULEPATH=/apps/modules/all:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
+++ export MODULEPATH
+++ PATH=/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binutils/2.26-GCCcore-5.4.0/bin:/apps/software/GCCcore/5.4.
+++ export PATH
+++ PKG_CONFIG_PATH=/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/lib/pkgconfig
+++ export PKG_CONFIG_PATH
+++ _LMFILES_=/apps/modules/all/GCCcore/5.4.0:/apps/modules/all/binutils/2.26-GCCcore-5.4.0:/apps/modules/all/GCC/5.4.0-2.26:/apps/modules/all/libarchive/3.3.1-GCC-5.4.0-2.26.lua:/apps/m
+++ export _LMFILES_
+++ _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJhY3RpdmVTaXplIl09NSxiYXNlTXBhdGhBPXsiL2FwcHMvbW9kdWxlcy9hbGwiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4IiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlI
+++ export _ModuleTable001_
+++ _ModuleTable002_=Y3RpdmUiLH0sR0NDY29yZT17WyJGTiJdPSIvYXBwcy9tb2R1bGVzL2FsbC9HQ0Njb3JlLzUuNC4wIixbImRlZmF1bHQiXT0wLFsiZnVsbE5hbWUiXT0iR0NDY29yZS81LjQuMCIsWyJsb2FkT3JkZXIiXT0xLHByb3BUP
+++ export _ModuleTable002_
+++ _ModuleTable003_=aWxzPXtbIkZOIl09Ii9hcHBzL21vZHVsZXMvYWxsL2JpbnV0aWxzLzIuMjYtR0NDY29yZS01LjQuMCIsWyJkZWZhdWx0Il09MCxbImZ1bGxOYW1lIl09ImJpbnV0aWxzLzIuMjYtR0NDY29yZS01LjQuMCIsWyJsb2FkT
+++ export _ModuleTable003_
+++ _ModuleTable004_=aXZlIix9LH0sbXBhdGhBPXsiL2FwcHMvbW9kdWxlcy9hbGwiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4IiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlIiwiL3Vzci9zaGFyZS9sbW9kL2xtb2QvbW9kd
+++ export _ModuleTable004_
+++ _ModuleTable_Sz_=4
+++ export _ModuleTable_Sz_
++ '[' 0 = 0 ']'
+++ : -s sh
++ eval
+ export PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binu
+ PATH=/home/a1222423/.nextflow/assets/nf-core/eager/bin:/apps/software/Singularity/2.5.2-GCC-5.4.0-2.26/bin:/apps/software/libarchive/3.3.1-GCC-5.4.0-2.26/bin:/apps/software/binutils/2.
+ singularity exec /fast/users/a1222423/eager/work/singularity/nf-core-eager.img /bin/bash -c 'pwd; cd /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f; pwd; ls -al; /bi
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
/fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
total 52
drwxrwxr-x 2 a1222423 a1222423  4096 Nov  7 05:09 .
drwxrwxr-x 3 a1222423 a1222423  4096 Nov  5 13:40 ..
-rw-rw-r-- 1 a1222423 a1222423     0 Nov  5 13:40 .command.begin
-rw-rw-r-- 1 a1222423 a1222423   234 Nov  5 13:40 .command.err
-rw-rw-r-- 1 a1222423 a1222423  1353 Nov  5 13:40 .command.log
-rw-rw-r-- 1 a1222423 a1222423     0 Nov  5 13:40 .command.out
-rw-rw-r-- 1 a1222423 a1222423  2591 Nov  7 05:06 .command.run
-rw-rw-r-- 1 a1222423 a1222423   787 Nov  5 13:40 .command.sh
-rw-rw-r-- 1 a1222423 a1222423  3495 Nov  5 13:40 .command.stub
-rw-rw-r-- 1 a1222423 a1222423     3 Nov  5 13:40 .exitcode
-rw-rw-r-- 1 a1222423 a1222423 15383 Nov  7 05:09 slurm-12963871.out
-rw-rw-r-- 1 a1222423 a1222423   414 Nov  7 05:08 test-bowen.sh

The output above showed the binding worked well.

However, nextflow still have the binding issue

[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ echo $SINGULARITY_BINDPATH                                                                                                                 /fast/users/a1222423/eager/work,/data
[a1222423@l01 12a213d46743cf1ccdd3e71e23f23f]$ nextflow run nf-core/eager --pairedEnd --reads "/data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz" --trim_bam 3 --max_time 12.h --max_cpus 4 --max_memory 32G --snpcapture false --udg true --udg_type Half --bwamem --genome CanFam3.1 --saveReference true  -profile phoenix -r dev -name $(date +%Y%m%d%H%M)
N E X T F L O W  ~  version 18.10.1
Launching `nf-core/eager` [201811081350] - revision: a827830491 [dev]
NOTE: Your local project version looks outdated - a different revision is available in the remote repository [a2b0c7c40a]
=========================================
 nf-core/eager v2.1.0dev
=========================================
Pipeline Name  : nf-core/eager
Pipeline Version: 2.1.0dev
Run Name       : 201811081350
Reads          : /data/acad/ysouilmi/dingo/modern_WGS_data/NGSD2_r2_{1,2}.fastq.gz
Fasta Ref      : false
Data Type      : Paired-End
Max Memory     : 32G
Max CPUs       : 4
Max Time       : 12.h
Output dir     : ./results
Working dir    : /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work
Container Engine: singularity
Container      : shub://nf-core/eager
Current home   : /home/a1222423
Current user   : a1222423
Current path   : /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f
Script dir     : /home/a1222423/.nextflow/assets/nf-core/eager
Config Profile : phoenix
=========================================
[warm up] executor > slurm
[d4/ee3b47] Submitted process > get_software_versions
[37/d40954] Submitted process > fastqc (NGSD2_r2)
[24/a5fd8c] Submitted process > adapter_removal (NGSD2_r2)
[f8/caaf83] Submitted process > output_documentation
ERROR ~ Error executing process > 'get_software_versions'

Caused by:
  Process `get_software_versions` terminated with an error exit status (127)

Command executed:

  echo 2.1.0dev &> v_pipeline.txt
  echo 18.10.1 &> v_nextflow.txt
  fastqc --version &> v_fastqc.txt 2>&1 || true
  multiqc --version &> v_multiqc.txt 2>&1 || true
  bwa &> v_bwa.txt 2>&1 || true
  samtools --version &> v_samtools.txt 2>&1 || true
  AdapterRemoval -version  &> v_adapterremoval.txt 2>&1 || true
  picard MarkDuplicates --version &> v_markduplicates.txt  2>&1 || true
  dedup -v &> v_dedup.txt 2>&1 || true
  preseq &> v_preseq.txt 2>&1 || true
  gatk BaseRecalibrator --version 2>&1 | grep Version: > v_gatk.txt 2>&1 || true
  vcf2genome &> v_vcf2genome.txt 2>&1 || true  
  fastp --version &> v_fastp.txt 2>&1 || true  
  bam --version &> v_bamutil.txt 2>&1 || true  
  qualimap --version &> v_qualimap.txt 2>&1 || true

  scrape_software_versions.py &> software_versions_mqc.yaml

Command exit status:
  127

Command output:
  (empty)

Command error:
  /bin/bash: line 0: cd: /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work/d4/ee3b4742983a9d9387e27bb9b49312: No such file or directory
  /bin/bash: .command.stub: No such file or directory

Work dir:
  /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/work/d4/ee3b4742983a9d9387e27bb9b49312

Tip: when you have fixed the problem you can continue the execution appending to the nextflow command line the option `-resume`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
[37/d40954] NOTE: Process `fastqc (NGSD2_r2)` terminated with an error exit status (127) -- Error is ignored
[nf-core/eager] Pipeline Complete

I also ran another singularity image for a Computational Fluid Dynamics (CFD) application named OpenFOAM under /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f with @yassineS account.

The working directory is /fast/users/a1222423/eager/work/8b/12a213d46743cf1ccdd3e71e23f23f/pitzDaily and both input and output are under it.

It worked well.

All above indicates the issue is on nextflow side rather than singularity configuration, possibly the part integration with singularity or nextflow configuration (nextflow might redefine or not propagate $SINGULARITY_BINDPATH (?)), unless some bugs inside singularity itself

Cheers,

Bowen

Edited history:

Edited format and typos

apeltzer · 2018-11-08T09:13:30Z

Aah, I see!

I added the envWhitelist=SINGULARITY_BINDPATH option in the phoenix profile for this pipeline.
@yassineS Could you test this using the -r dev -profile phoenix branch?

That could resolve this issue then, just make sure you have set your SINGULARITY_BINDPATH before running the pipeline then.

uoabowen · 2018-11-20T03:30:42Z

Hi @apeltzer , I bring bad and good news

I tested it with the new profile, it still had the same issue.
So I set bind path in singularity.conf to let it binds paths automatically, seems everything works fine.

I searched nextflow's document, and found below under images section on https://www.nextflow.io/docs/latest/singularity.html#images

If your Singularity installation support the user bind control feature, enable the Nextflow support for that by defining the singularity.autoMounts = true setting in the Nextflow configuration file.

Seems add singularity.autoMounts = true in profile or nextflow.config (not sure adding to which is correct or better) may resolve the issue when using the user bind control feature with singularity

apeltzer · 2018-11-20T10:04:50Z

Hi @uoabowen !

I added the required configuration to the acad-phoenix profile, so using the -profile phoenix should do the trick now!

yassineS · 2018-11-28T04:32:16Z

That's great!

apeltzer mentioned this issue Oct 31, 2018

Conda profile FastQC crash #67

Closed

apeltzer added a commit that referenced this issue Nov 4, 2018

Add config for acad phoenix #68

a6bc454

apeltzer added the needs upstream fix Needs a fix in the upstream tool project label Nov 7, 2018

apeltzer closed this as completed Nov 28, 2018

apeltzer mentioned this issue Dec 10, 2018

Singularity Run issue #95

Closed

d4straub mentioned this issue Sep 20, 2021

singularity: No such file or directory nf-core/mag#234

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Singularity missing? #68

Singularity missing? #68

apeltzer commented Oct 31, 2018

apeltzer commented Oct 31, 2018

yassineS commented Oct 31, 2018

apeltzer commented Oct 31, 2018

yassineS commented Oct 31, 2018

apeltzer commented Nov 2, 2018

yassineS commented Nov 4, 2018 via email

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018 via email

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018 via email

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018

apeltzer commented Nov 4, 2018

apeltzer commented Nov 6, 2018

uoabowen commented Nov 8, 2018 •

edited

Loading

apeltzer commented Nov 8, 2018

uoabowen commented Nov 20, 2018

apeltzer commented Nov 20, 2018

yassineS commented Nov 28, 2018

Singularity missing? #68

Singularity missing? #68

Comments

apeltzer commented Oct 31, 2018

apeltzer commented Oct 31, 2018

yassineS commented Oct 31, 2018

apeltzer commented Oct 31, 2018

yassineS commented Oct 31, 2018

apeltzer commented Nov 2, 2018

yassineS commented Nov 4, 2018 via email

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018 via email

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018 via email

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018

apeltzer commented Nov 4, 2018

yassineS commented Nov 4, 2018

apeltzer commented Nov 4, 2018

apeltzer commented Nov 6, 2018

uoabowen commented Nov 8, 2018 • edited Loading

apeltzer commented Nov 8, 2018

uoabowen commented Nov 20, 2018

apeltzer commented Nov 20, 2018

yassineS commented Nov 28, 2018

uoabowen commented Nov 8, 2018 •

edited

Loading