Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

running quickstart tutorial fails: squashfs-tools not available #102

Open
stephenturner opened this issue Jul 11, 2018 · 16 comments
Open

running quickstart tutorial fails: squashfs-tools not available #102

stephenturner opened this issue Jul 11, 2018 · 16 comments

Comments

@stephenturner
Copy link

stephenturner commented Jul 11, 2018 •
edited

Trying to run the read filtering workflow at https://dahak-metagenomics.github.io/dahak/quickstart/#read-filtering

After setting up the environment as noted in #99 and issuing the following command (with --use-singularity):

snakemake --use-singularity -p \
        --configfile=config/custom_readfilt_workflow.json \
        read_filtering_pretrim_workflow

I get:

Building DAG of jobs...
Pulling singularity image docker://quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2.
WorkflowError:
Failed to pull singularity image from docker://quay.io/biocontainers/fastqc:0.11.7--pl5.22.0_2:
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
ERROR: You must install squashfs-tools to build images
ABORT: Aborting with RETVAL=255
ERROR: pulling container failed!

$ singularity --version
2.4.2-dist

Please provide further installation/setup instructions.
@brooksph
Copy link
Contributor

Thanks @stephenturner. This seems to be related to apptainer/singularity#1078. Does installing squashfs-tools resolve the issue?

@charlesreid1
Copy link
Member

Just skimmed through that issue, it's not clear how to install squashfs-tools...

@stephenturner is this on Ravenna/HPC or your local machine?

@brooksph
Copy link
Contributor

brooksph commented Jul 11, 2018 •
edited

Gotcha, on ubuntu sudo apt-get install squashfs-tools. Not sure if it's possible via conda.

@stephenturner
Copy link
Author

stephenturner commented Jul 11, 2018 via email

@stephenturner
Copy link
Author

stephenturner commented Jul 11, 2018 via email

@charlesreid1
Copy link
Member

This issue was raised in the singularity repo here, problem was with RedHat. It sounds like a problem with Linux and Singularity not playing nicely together.

I am currently working on reproducing this error on an AWS node running CentOS....

@kternus
Copy link
Collaborator

kternus commented Jul 16, 2018

We've previously had trouble getting that FastQC docker to run on Red Hat, and it may be a Fedora compatibility issue. Is it possible for @stephenturner to skip that step and test the next step in the workflow to narrow down whether it's an issue with the FastQC container or something else?

@stephenturner
Copy link
Author

RHEL/Fedora/CentOS should all be the same, essentially. The assembly quickstart (https://dahak-metagenomics.github.io/dahak/quickstart/#assembly) has the same issue as noted above. After creating the json file, and running:

snakemake --use-singularity -p --configfile=config/custom_assembly_workflow.json assembly_workflow_all

Get the same NEWUSER/namespace problem:

Building DAG of jobs...
Pulling singularity image docker://quay.io/biocontainers/spades:3.11.1--py27_zlib1.2.8_0.
Pulling singularity image docker://quay.io/biocontainers/megahit:1.1.2--py35_0.
Pulling singularity image docker://quay.io/biocontainers/trimmomatic:0.36--5.
Using shell: /usr/bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        4       assembly_megahit
        4       assembly_metaspades
        1       assembly_workflow_all
        1       download_read_adapters
        1       download_reads
        4       quality_trimming
        15

Job 15: --- Downloading adapter file.

wget -O data/TruSeq2-PE.fa http://dib-training.ucdavis.edu.s3.amazonaws.com/mRNAseq-semi-2015-03-04/TruSeq2-PE.fa
--2018-07-16 10:59:22--  http://dib-training.ucdavis.edu.s3.amazonaws.com/mRNAseq-semi-2015-03-04/TruSeq2-PE.fa
Resolving dib-training.ucdavis.edu.s3.amazonaws.com (dib-training.ucdavis.edu.s3.amazonaws.com)... 54.231.236.43
Connecting to dib-training.ucdavis.edu.s3.amazonaws.com (dib-training.ucdavis.edu.s3.amazonaws.com)|54.231.236.43|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 539 [binary/octet-stream]
Saving to: ‘data/TruSeq2-PE.fa’

100%[====================================================================>] 539         --.-K/s   in 0s

2018-07-16 10:59:22 (63.5 MB/s) - ‘data/TruSeq2-PE.fa’ saved [539/539]

Finished job 15.
1 of 15 steps (7%) done

Job 10: --- Quality trimming read data.

trimmomatic PE /data/SRR606249_subset10_1_reads.fq.gz /data/SRR606249_subset10_2_reads.fq.gz /data/SRR606249_subset10_1.trim30.fq.gz /data/SRR606249_subset10_1.trim30_se /data/SRR606249_subset10_2.trim30.fq.gz /data/SRR606249_subset10_2.trim30_se ILLUMINACLIP:/data/TruSeq2-PE.fa:2:40:15 LEADING:30 TRAILING:30 SLIDINGWINDOW:4:30 MINLEN:25
Activating singularity image /path/dahak/workflows/.snakemake/singularity/7a3439791f532ae9738cd41931466141.simg
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255
    Error in rule quality_trimming:
        jobid: 10
        output: data/SRR606249_subset10_1.trim30.fq.gz, data/SRR606249_subset10_1.trim30_se, data/SRR606249_subset10_2.trim30.fq.gz, data/SRR606249_subset10_2.trim30_se
        log: data/trimmomatic_pe_SRR606249_subset10_trim30.log

RuleException:
CalledProcessError in line 405 of /path/dahak/workflows/read_filtering/Snakefile:
Command 'singularity exec --home /path/dahak/workflows  /path/dahak/workflows/.snakemake/singularity/7a3439791f532ae9738cd41931466141.simg bash -c " set -euo pipefail;  trimmomatic PE /data/SRR606249_subset10_1_reads.fq.gz /data/SRR606249_subset10_2_reads.fq.gz /data/SRR606249_subset10_1.trim30.fq.gz /data/SRR606249_subset10_1.trim30_se /data/SRR606249_subset10_2.trim30.fq.gz /data/SRR606249_subset10_2.trim30_se ILLUMINACLIP:/data/TruSeq2-PE.fa:2:40:15 LEADING:30 TRAILING:30 SLIDINGWINDOW:4:30 MINLEN:25 "' returned non-zero exit status 255.
  File "/path/dahak/workflows/read_filtering/Snakefile", line 405, in __rule_quality_trimming
  File "/path/miniconda3/envs/dahak/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/dahak/workflows/.snakemake/log/2018-07-16T105901.667076.snakemake.log

@kternus
Copy link
Collaborator

kternus commented Jul 16, 2018

Thanks for sharing that! I was going to ask @nalbright try different workflow steps on Red Hat, but it looks like that wouldn't be helpful based on your experience here. We can setup an Ubuntu OS for testing, and I'll have her focus on that for now.

@ctb
Copy link
Contributor

ctb commented Jul 17, 2018

Hi @kternus, yes, this is a generic problem with RedHat and Singularity.

@cgrahlm
Copy link

cgrahlm commented Jul 18, 2018

@stephenturner
I was able to get this to run in a CentOS 7 VM. Are you able to run singularity on a test image like so:
singularity --debug run shub://GodloveD/lolcow
I googled around some more and there could be a few problems regarding the setuid bit and installing on clusters with RHEL.

This has some resources we may been to reference:
https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/W1lFqknaBLg

@stephenturner
Copy link
Author

I wasn't able to get singularity --debug run shub://GodloveD/lolcow to run. Same issue. I'm going to compile from source and install as root, see if that helps.

@stephenturner
Copy link
Author

So, I did that, and I'm getting past the NEWUSER error but hitting a wall with

DEBUG   [U=543223,P=21023] singularity_priv_escalate()               Temporarily escalating privileges (U=543223)
DEBUG   [U=0,P=21023]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=21023]      singularity_image_bind()                  Setting loop device flags
DEBUG   [U=0,P=21023]      singularity_priv_drop()                   Dropping privileges to UID=543223, GID=100 (5 supplementary GIDs)
DEBUG   [U=0,P=21023]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=543223,P=21023] singularity_priv_drop()                   Confirming we have correct UID/GID
VERBOSE [U=543223,P=21023] singularity_image_bind()                  Using loop device: /dev/loop0
VERBOSE [U=543223,P=21023] singularity_image_squashfs_mount()        Mounting squashfs image: /dev/loop0 -> /usr/local/var/singularity/mnt/container
ERROR   [U=543223,P=21023] singularity_image_squashfs_mount()        Failed to mount squashfs image in (read only): No such device
ABORT   [U=543223,P=21023] singularity_image_squashfs_mount()        Retval = 255

I googled around a bit and landed on a few issues on the singularity repo, but nothing I could grok.

I do have squashfs-tools installed.

@stephenturner
Copy link
Author

Should also add

$ singularity --version
2.5.2-dist

@cgrahlm
Copy link

cgrahlm commented Jul 18, 2018 via email

@stephenturner
Copy link
Author

stephenturner commented Jul 18, 2018 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@ctb @charlesreid1 @stephenturner @kternus @brooksph @cgrahlm