container creation failed: no loop device available

[sparktx-chao-di]

I've been testing the nf-core/rnaseq pipeline on my EC2 instance without issue, but recently after an instance reboot the same pipeline failed with error message " FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available".

I've searched around and asked for help internally. Here are some info that may be helpful:

• When I reboot the EC2 instance, I can see there are 14 loop devices available by ls /dev/loop*, but when I run the Nextflow pipeline, somehow, it automatically generates more loop devices and the total number became 256 when I do ls /dev/loop*, and the newly generated loop devices all have the permission of "brw-r----- 1" other than the preexisting ones with "brw-rw---- 1". So it seems I don't have the write permission for these loop devices as a non-root user. I wonder if there is any configuration in Singularity that can change these?

• I ran with the same error when I ran Snakemake pipelines with Singularity for some rules, so the issue is not from Nextflow itself. The command is like "snakemake -s Snakefile.py --use-singularity --singularity-args "-B /mnt/bfx-ops/:/mnt/bfx-ops/". One thing strange is that every time I ran the command one job is actually running with the container with the expected output, but all the other jobs running in parallel failed with the same FATAL error.

• My system is "Linux ip-10-50-11-146 5.15.0-1035-aws build(deps): bump github.com/fatih/color from 1.10.0 to 1.11.0 #39~20.04.1-Ubuntu SMP Wed Apr 19 15:34:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux". My EC2 instance has 64 cores. Our system admin claimed there were no changes or updates on the EC2 instance,

• The command I'm running is"
  nextflow run nf-core/rnaseq --input samplesheet_local.csv --outdir output --gtf genes.gtf --fasta genome.fa -profile singularity
  "

-My Singularity has been working well on its own. I can run the following with no issues.
$ singularity pull --name hello.simg shub://vsoch/hello-world
$ singularity run hello.simg

Thank you for your help!

[sparktx-chao-di]

Sorry to mention that my Singularity version is v3.11.0

[sparktx-chao-di]

If I do "singularity shell hello.simg", I was able to get the container running or say get into the container. But if I open a new shell terminal and do "singularity shell hello.simg" again, I would get the same error "FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available".
So I guess there is something wrong with my system config that doesn't allow me to run multiple Singularity containers in parallel, although my EC2 instance has 64 cores available.

[dtrudg]

A recent change in the Ubuntu kernel affected loop device creation with the method previously employed by SingularityCE.

A workaround was implemented in SIngularityCE v3.11.2.

If you upgrade to a newer version, things should work as expected.

References:

• https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2013086
• No loop devices available for multiple singularity shells in specific kernel #1499

[sparktx-chao-di]

Thanks a lot! That solves my problem!