Allow nextflow to handle http protocol for Singularity images

[apeltzer]

Bug report

1.) Trying to run new nf-core/rnaseq pipeline:

• nextflow run nf-core/rnaseq -r dev -profile bi,test,cluster
• Failure:

bash
FATAL:   container creation failed: mount /proc/self/fd/10->/opt/singularity/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/10: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted`, size: 331 (max: 255)
Error executing process > 'RNASEQ:CAT_FASTQ (RAP1_UNINDUCED_R2)'
Caused by:
 Process `RNASEQ:CAT_FASTQ (RAP1_UNINDUCED_R2)` terminated with an error exit status (255)
Command executed:
 cat SRR6357074_1.fastq.gz SRR6357075_1.fastq.gz > RAP1_UNINDUCED_R2.merged.fastq.gz
Command exit status:
 255
Command output:
 (empty)
Command error:
 FATAL:   container creation failed: mount /proc/self/fd/10->/opt/singularity/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/10: kernel reported a bad superblock for squashfs image partition, possible causes are that your kernel doesn't support the compression algorithm or the image is corrupted

2.) Trying this with Docker - success :-)
3.) Singularity Version 3.4.2el7, works fine with all nf-core (+ internal pipelines), Nextflow V 20.07.1 - can check 20.10 but others have similar issues?
4.) Checking singularity command to download/work with image (with example command):
singularity exec -B /scratch/nextflow/work/peltzera -B /home/peltzera/nf-core/rnaseq/bin -B "$PWD" https\://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img python --version
- works as intended
5.) wget https\://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img; singularity exec biocontainers_v1.2.0_cv1.img python --version
- works as intended

Comments:

• Issue is only persistent with Singularity, Docker runs fine.
• External download of container works fine, e.g. using wget/curl. Can also externally download containers and pipeline runs then.

Steps to reproduce the problem

Harshil is posting a minimal example soon ;-) Also, can reproduce by just running the dev version of nf-core/rnaseq on any system with singularity to reproduce. First DSLv2 pipeline in nf-core, so we don't have another example unfortunately.

Program output

See above

Environment

• Nextflow version: 20.07.1
• Java version: 1.8.0_172
• Operating system: EL7
• Bash version: 4.2.46(2)

[drpatelh]

Hi @pditommaso below is a minimal example for you to be able to reproduce the error

The main.nf script just contains a mimimal FASTQC process that prints its version and so doesn't require any inputs:

#!/usr/bin/env nextflow

nextflow.enable.dsl = 2

process FASTQC {

    conda (params.enable_conda ? "bioconda::fastqc=0.11.9" : null)
    if (workflow.containerEngine == 'singularity' && !params.pull_docker_container) {
        container "https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0"
    } else {
        container "quay.io/biocontainers/fastqc:0.11.9--0"
    }

    output:
    path  "*.version.txt", emit: version

    script:
    """
    fastqc --version | sed -e "s/FastQC v//g" > fastqc.version.txt
    """
}

workflow {
    FASTQC ( )
}

The nextflow.config file contains a few -profile definitions we can use for testing:

params {
  enable_conda          = false
  pull_docker_container = false

}

profiles {
  conda       { params.enable_conda = true }
  docker      {
    docker.enabled = true
  }
  singularity {
    singularity.enabled = true
    singularity.autoMounts = true
  }
}

And we can execute this script using various -profile as listed below:

## Command below is the only one that fails
nextflow run main.nf -profile singularity

## Works
nextflow run main.nf -profile singularity --pull_docker_container

## Works
nextflow run main.nf -profile docker          

## Works
nextflow run main.nf -profile conda         

As @apeltzer mentioned it seems like there is an issue with downloading Singularity images directly via https: and it has been failing reproducibly for numerous users. Maybe this isn't supported in core Nextflow yet? Are there any workarounds?

We need this to work because it would be much easier downloading Singularity images directly if they exist rather than converting from Docker images.

With DSL2, this will become more important because we will need numerous containers in order to run the pipeline.

Thank you!!

[pditommaso]

"https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0" its not a recognized singularity image url

[drpatelh]

Thanks. So the download of the image works as expected if you click on the link or via wget. Is there a workaround for this? What would the URL have to look like in order to be recognised and work natively with Nextflow?

[pditommaso]

I mean does singularity pull https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0 or singularity exec https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0 works?

[drpatelh]

Not for me...

$ singularity pull https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0
FATAL:   While pulling from image from http(s): Get https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0: net/http: TLS handshake timeout

But I just had a 💡 moment and tried another registry that hosts the containers and it works. Although, the registry itself seems to be more of an unofficial one for Biocontainers:

$ singularity pull https://containers.biocontainers.pro/s3/SingImgsRepo/fastqc/v0.11.9_cv7/fastqc_v0.11.9_cv7.img
 281.96 MiB / ? [------------------------------------=--------------------------------------------------------------------------------------] 2956575.03% 13.85 MiB/s 20s

$ singularity run https://containers.biocontainers.pro/s3/SingImgsRepo/fastqc/v0.11.9_cv7/fastqc_v0.11.9_cv7.img
$ fastqc --version
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
FastQC v0.11.9

[apeltzer]

For me it does work just fine - both exec as well as the pull:

singularity pull https\://containers.biocontainers.pro/s3/SingImgsRepo/biocontainers/v1.2.0_cv1/biocontainers_v1.2.0_cv1.img
singularity pull https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0

[pditommaso]

Ok, nice, tho I think this is a new Singularity feature. NF at this time does not allow handling http protocol for Singularity.

[pditommaso]

Patched. I'll include in the next release.

[drpatelh]

Superstar 😍