running singularity within singularity - ruhroh!

[vsoch]

Version of Singularity:

2.4.2-master.g91881f7

When Singularity is installed within another container, e.g.,

Bootstrap: docker
From: continuumio/miniconda3

# sudo singularity build sregistry.simg Singularity


#######################################
# Global
#######################################

%runscript
    exec /opt/conda/bin/sregistry "$@"


#######################################
# Google Cloud Storage
#######################################

%appenv google-storage
    SREGISTRY_CLIENT=google-storage
    export SREGISTRY_CLIENT
%apprun google-storage
    exec /opt/conda/bin/sregistry "$@"



#######################################
# Google Cloud Drive
#######################################

%appenv google-drive
    SREGISTRY_CLIENT=google-drive
    export SREGISTRY_CLIENT
%apprun google-drive
    exec /opt/conda/bin/sregistry "$@"



#######################################
# Singularity Hub
#######################################

%appenv hub
    SREGISTRY_CLIENT=hub
    export SREGISTRY_CLIENT
%apprun hub
    exec /opt/conda/bin/sregistry "$@"


#######################################
# Singularity Registry
#######################################

%appenv registry
    SREGISTRY_CLIENT=registry
    export SREGISTRY_CLIENT
%apprun registry
    exec /opt/conda/bin/sregistry "$@"


%environment
    PATH=/usr/local/bin:$PATH
    export PATH


%labels
    maintainer vsochat@stanford.edu

%post
    apt-get update && apt-get install -y git build-essential \
                   libtool \
                   squashfs-tools \
                   autotools-dev \
                   automake \
                   autoconf \
                   uuid-dev \
                   libssl-dev

    /opt/conda/bin/pip install dateutils

    # Install Singularity
    cd /opt && git clone https://www.github.com/singularityware/singularity.git && cd singularity
    ./autogen.sh && ./configure --prefix=/usr/local && make && make install

    # Install SRegistry Global
    cd /opt && git clone https://www.github.com/singularityhub/sregistry-cli
    cd sregistry-cli
    /opt/conda/bin/pip install setuptools

    # This installs all "install extras"
    /opt/conda/bin/pip install -e .
    /opt/conda/bin/pip install -e .[google-drive]
    /opt/conda/bin/pip install -e .[google-storage]

A command issued to inspect a container results in the following error (I am doing this command shelled in without being root), and the same behavior results when issuing externally:

singularity --quiet inspect -e -d -l -r -hf -t --json sregistry-cli
ERROR  : Singularity is not running with appropriate privileges!
ERROR  : Check installation path is not mounted with 'nosuid', and/or consult manual.
ABORT  : Retval = 255

The command on the host works ok.

singularity --quiet inspect -e -d -l -r -hf -t --json sregistry-cli'

I thought it might be an issue with an installation on host and container, so I tried the command with the environment contained:

singularity shell --cleanenv sregistry-cl

and

singularity exec --cleanenv sregistry-cli singularity inspect -e -d -l -r -l -hf -t --json sregistry-cli
ERROR  : Singularity is not running with appropriate privileges!
ERROR  : Check installation path is not mounted with 'nosuid', and/or consult manual.
ABORT  : Retval = 255

Expected behavior

singularity exec sregistry-cli singularity inspect --json sregistry-cli
{
    "data": {
        "attributes": {
            "labels": {
                "org.label-schema.usage.singularity.deffile.bootstrap": "docker",
                "MAINTAINER": "vsochat@stanford.edu",
                "org.label-schema.usage.singularity.deffile": "Singularity",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.usage.singularity.deffile.from": "continuumio/miniconda3",
                "org.label-schema.build-date": "Mon,_08_Jan_2018_12:54:52_-0800",
                "org.label-schema.usage.singularity.version": "2.4.1-bleuchien-apprunfix.g85c133d",
                "org.label-schema.build-size": "861MB"
            }
        },
        "type": "container"
    }
}

A minimum working example to reproduce:

Singularity.error

Bootstrap: docker
From: continuumio/miniconda3

# sudo singularity build error.simg Singularity

%environment
    PATH=/usr/local/bin:$PATH
    export PATH

%post
    apt-get update && apt-get install -y git build-essential \
                   libtool \
                   squashfs-tools \
                   autotools-dev \
                   automake \
                   autoconf \
                   uuid-dev \
                   libssl-dev

    /opt/conda/bin/pip install dateutils

    # Install Singularity
    cd /opt && git clone https://www.github.com/singularityware/singularity.git && cd singularity
    ./autogen.sh && ./configure --prefix=/usr/local && make && make install

singularity pull --name test.simg shub://vsoch/hello-world
sudo singularity build error.simg Singularity.error 
singularity run error.simg singularity inspect test.simg
ERROR  : Singularity is not running with appropriate privileges!
ERROR  : Check installation path is not mounted with 'nosuid', and/or consult manual.
ABORT  : Retval = 255

Thanks for the help! And all the fish :)

[DrDaveD]

As I mentioned in singularityhub/sregistry-cli#30, I believe that singularity deliberately disallows running any setuid programs inside of a container for security reasons.

[vsoch]

So this means that Singularity can never be run inside Singularity? What is the suggested fix / strategy for an application that needs a (non sudo) singularity command within a Singularity container? I had thought this worked before.

[DrDaveD]

I think it might work if the outer singularity is run by root. I haven't verified that, but I know it works with docker --privileged.

[vsoch]

So there is no solution for running this entirely in user space?

[vsoch]

The workaround I think is to mount the singularity from the host in the container, but @shahzebsiddiqui had issue with this. @shahzebsiddiqui what are your concerns?

[yarikoptic]

ok, good to know that I can't do that (was about to try) which is not the end of the world but disallows a few usecases

[vsoch]

Any update on this issue? I'm testing with sudo, and mounting the /usr/local base where Singularity is installed. My version is a tad old, perhaps too old to have the ability to set specific permissions?

$ sudo singularity shell --bind /usr/local cromwell.simg 
Singularity: Invoking an interactive shell within container...

Singularity cromwell.simg:~> singularity run shub://vsoch/hello-world
Progress |===================================| 100.0% 
ERROR  : Failed to set processus capabilities
ABORT  : Retval = 255

Here is the version

singularity --version
2.5.0-feature-squashbuild-secbuild-2.5.0.gabe9005

Is there docs for how to set process capabilities? I was looking here and I think something is in the works, but it's not documented anywhere. Thanks for the update!

[oguitart]

Hello,

I'm wondering if you find a solution for this issue. I have tried to use a singularity image inside another singularity image but I get the same error:

Singularity sandbox-sing:~/singularity> singularity exec sandbox-deep/ ls
ERROR : Singularity is not running with appropriate privileges!
ERROR : Check installation path is not mounted with 'nosuid', and/or consult manual.
ABORT : Retval = 255

Is not possible at all?
Thanks,

oriol

[DrDaveD]

Singularity does not allow running setuid programs within containers. If your kernel allows unprivileged user namespaces, singularity can be run unprivileged and then nesting of singularity works. I have found only one of them can include a -p option for pid namespaces.

[dtrudg]

As mentioned in the last comment, we explicitly prohibit running setuid things in containers as part of the general approach of Singularity. Nesting with user-namespace is the option as pointed out by DrDaveD