error: can't mount image /proc/self/fd/3: failed to mount squashfs filesystem: invalid argument

[Dipakam]

Version of Singularity: 3.3.0

What version of Singularity are you using? Run:

$ singularity version

3.3.0

Expected behavior

Trying to run hello-world.simg. I first built it using

$ singularity build hello-world.simg shub://vsoch/hello-world

then I tried to run it using

$ singularity shell hello-world.simg

Actual behavior

FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/3: failed to mount squashfs filesystem: invalid argument
I tried with verbose on and it is giving ->

VERBOSE: HOME = /home/deepak
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Execute stage 1
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 1000
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Spawn RPC server
VERBOSE: Execute master process
VERBOSE: Serve RPC requests
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /home/deepak/singularity/builddir: to the container
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
FATAL: container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/3: failed to mount squashfs filesystem: invalid argument

Also with the debug option it gives the following output :

DEBUG [U=1000,P=23937] createConfDir() /home/deepak/.singularity already exits. Not creating.
DEBUG [U=1000,P=23937] getCacheBasedir() environment variable SINGULARITY_CACHEDIR not set, using default image cache
DEBUG [U=1000,P=23937] updateCacheSubdir() Caching directory set to /home/deepak/.singularity/cache/library
DEBUG [U=1000,P=23937] updateCacheSubdir() Caching directory set to /home/deepak/.singularity/cache/oci-tmp
DEBUG [U=1000,P=23937] updateCacheSubdir() Caching directory set to /home/deepak/.singularity/cache/oci
DEBUG [U=1000,P=23937] updateCacheSubdir() Caching directory set to /home/deepak/.singularity/cache/net
DEBUG [U=1000,P=23937] updateCacheSubdir() Caching directory set to /home/deepak/.singularity/cache/shub
DEBUG [U=1000,P=23937] updateCacheSubdir() Caching directory set to /home/deepak/.singularity/cache/oras
DEBUG [U=1000,P=23937] execStarter() Use starter binary /usr/local/libexec/singularity/bin/starter-suid
VERBOSE [U=1000,P=23937] SetContainerEnv() HOME = /home/deepak
VERBOSE [U=0,P=23937] print() Set messagelevel to: 5
VERBOSE [U=0,P=23937] init() Starter initialization
DEBUG [U=0,P=23937] get_pipe_exec_fd() PIPE_EXEC_FD value: 7
VERBOSE [U=0,P=23937] is_suid() Check if we are running as setuid
VERBOSE [U=0,P=23937] priv_drop() Drop root privileges
DEBUG [U=1000,P=23937] init() Read engine configuration
DEBUG [U=1000,P=23937] init() Wait completion of stage1
VERBOSE [U=1000,P=23946] priv_drop() Drop root privileges permanently
DEBUG [U=1000,P=23946] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=1000,P=23946] init() Spawn stage 1
VERBOSE [U=1000,P=23946] startup() Execute stage 1
DEBUG [U=1000,P=23946] StageOne() Entering stage 1
DEBUG [U=1000,P=23946] Init() Entering image format intializer
DEBUG [U=1000,P=23946] Init() Check for image format sif
VERBOSE [U=1000,P=23937] wait_child() stage 1 exited with status 0
DEBUG [U=1000,P=23937] cleanup_fd() Close file descriptor 4
DEBUG [U=1000,P=23937] init() Set child signal mask
DEBUG [U=1000,P=23937] init() Create socketpair for master communication channel
DEBUG [U=1000,P=23937] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=23937] priv_escalate() Get root privileges
VERBOSE [U=0,P=23937] priv_escalate() Change filesystem uid to 1000
VERBOSE [U=0,P=23937] init() Spawn master process
DEBUG [U=0,P=23954] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=23954] create_namespace() Create mount namespace
VERBOSE [U=0,P=23937] enter_namespace() Entering in mount namespace
DEBUG [U=0,P=23937] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=0,P=23954] create_namespace() Create mount namespace
DEBUG [U=0,P=23955] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=0,P=23955] init() Spawn RPC server
VERBOSE [U=0,P=23955] startup() Serve RPC requests
VERBOSE [U=1000,P=23937] startup() Execute master process
DEBUG [U=1000,P=23937] checkOverlay() Overlay seems supported and allowed by kernel
DEBUG [U=1000,P=23937] setupSessionLayout() Attempting to use overlayfs (enable overlay = try)
DEBUG [U=1000,P=23937] setupOverlayLayout() Creating overlay SESSIONDIR layout
DEBUG [U=1000,P=23937] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=1000,P=23937] addRootfsMount() Mounting block [squashfs] image: /home/deepak/singularity/builddir/hello-world.simg
DEBUG [U=1000,P=23937] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=1000,P=23937] addKernelMount() Adding proc to mount list
VERBOSE [U=1000,P=23937] addKernelMount() Default mount: /proc:/proc
DEBUG [U=1000,P=23937] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=1000,P=23937] addKernelMount() Adding sysfs to mount list
VERBOSE [U=1000,P=23937] addKernelMount() Default mount: /sys:/sys
DEBUG [U=1000,P=23937] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=1000,P=23937] addDevMount() Adding dev to mount list
VERBOSE [U=1000,P=23937] addDevMount() Default mount: /dev:/dev
DEBUG [U=1000,P=23937] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=1000,P=23937] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=23937] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=1000,P=23937] addHomeStagingDir() Staging home directory (/home/deepak) at /usr/local/var/singularity/mnt/session/home/deepak
DEBUG [U=1000,P=23937] addHomeMount() Adding home directory mount [/usr/local/var/singularity/mnt/session/home/deepak:/home/deepak] to list using layer: overlay
DEBUG [U=1000,P=23937] isLayerEnabled() Using Layer system: overlay
DEBUG [U=1000,P=23937] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=23937] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=1000,P=23937] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=1000,P=23937] addScratchMount() Not mounting scratch directory: Not requested
VERBOSE [U=1000,P=23937] addCwdMount() Default mount: /home/deepak/singularity/builddir: to the container
DEBUG [U=1000,P=23937] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=1000,P=23937] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=23937] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=1000,P=23937] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=1000,P=23937] create() Mount all
DEBUG [U=1000,P=23937] mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session
DEBUG [U=1000,P=23937] mountImage() Mounting loop device /dev/loop14 to /usr/local/var/singularity/mnt/session/rootfs
DEBUG [U=1000,P=23937] CleanupContainer() Cleanup container
FATAL [U=1000,P=23937] Master() container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: can't mount image /proc/self/fd/3: failed to mount squashfs filesystem: invalid argument

Steps to reproduce this behavior

$ singularity build hello-world.simg shub://vsoch/hello-world
$ singularity shell hello-world.simg

What OS/distro are you running

$ cat /etc/os-release
-> 
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

How did you install Singularity

I used this -> https://sylabs.io/guides/3.3/user-guide/installation.html
The part before RPM.

[dtrudg]

Please try with Singularity 3.5.3 or the 3.6.0-r.c5 release candidate. There have been bug fixes made to mount options since 3.3.0 was released.

[nevesLiliane]

Hi, I'm having the same issue but with singularity 3.7.3
The error is

FATAL:   container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

Version of Apptainer

What version of Apptainer (or Singularity) are you using? Run

singularity version 3.7.3 (or singularity --version).

Expected behavior

$ singularity run shub://GodloveD/lolcow
INFO:    Use cached image
 _________________________________________
/ A gift of a flower will soon be made to \
\ you.                                    /
 -----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Actual behavior

What actually happened? Why was it incorrect?

singularity run shub://GodloveD/lolcow
INFO:    Use cached image
FATAL:   container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

I'm running multiple images, the first image is a RDBMS, if I run it I cannot execute other containers, it gives the error above, for any other image I try to execute after executing the RDBMS.

Steps to reproduce this behavior

How can others reproduce this issue/problem?

I don't know, I have been using singularity for more than 2 years and this is the first time it happens, using this same RDBMS, I haven't changed anything recently, this worked yesterday, and still works on other machines.
I have tested and found out that I can execute every image I have, but if I execute the RDBMS, the "no loop devices" message comes to the containers that I try to execute next. I have no idea how to solve that. Please help

What OS/distro are you running

$ cat /etc/os-release

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

How did you install Apptainer

Write here how you installed Apptainer (or Singularity). Eg. RPM, source.
I used apt-get to install singularity

[nttg8100]

It is quite strange but I think there is an error by mem. The error is similar but it reported on the nextflow pipeline with singularity profile. If someone who met this one could try:
I moved the folder that contain my pipeline to another location then it showed as above. I try to clone a new pipeline or rewrite without copy it then it worked well. Just do nothing

[dcat52]

@nttg8100 Thanks for your comment, definitely a mem error. I just encountered this in Apptainer v1.1.4
I just encountered this when I accidentally left a shell on an old container. I tried rebuilding this container several times and could not launch it. I closed the old container shell and was able to launch a new shell without a rebuild.

[julieaorjuela]

I don't know if it could help you but i resolved it using https://code-aster.org/forum2/viewtopic.php?id=26901

[honggangwang1979]

same error with 3.8.7, anyone has solution?

[DrDaveD]

This repository is closed and that version is obsolete. Please try it with a recent version of either apptainer or singularity-ce and report any problems along with full environment detaills in either https://github.com/apptainer/apptainer or https://github.com/sylabs/singularity.