singularity doesn't take advantage of unprivileged mount namespaces when enabled on RHEL7.4

[DrDaveD]

Version of Singularity:

2.3.1

Expected behavior

singularity would work when "allow setuid = no" and "enable overlay = no". without using setuid-root, when namespace.unpriv_enable=1 on the kernel boot command line on RHEL7.4.

Actual behavior

Running "singularity -v exec --containall ~/centos7.img bash" ends with

VERBOSE: Running NON-SUID program workflow
VERBOSE: Invoking the user namespace
ERROR  : Could not write info to setgroups: Permission denied
ABORT  : Retval = 255

Steps to reproduce behavior

Install RHEL7.4 on a VM (I got a 30 day evaluation license). Edit /etc/sysconfig/grub to add "namespace.unpriv_enable=1" to the end of GRUB_CMDLINE_LINUX and run "grub2-mkconfig -o /boot/grub2/grub.cfg" to update the boot parameter. Reboot. Then run the above command.

For reference, search for "unpriv" in the Redhat 7.4 release notes kernel section. It says "issuing a call to the clone() function with the flag CLONE_NEWNS as an unprivileged user no longer returns an error and allows the operation." I looked through the singularity source code and I do not see it using that function.

[DrDaveD]

Sorry, the 2.3.1 error message is really

ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument

The message I put in the initial report about not being able to write to setgroups was from a version that I modified to comment out the exit when it got this failure invoking the NEWUSER namespace (I had momentarily forgotten I had done that).

[bauerm97]

@DrDaveD Can you run this command to see if your kernel is supporting user namespaces? CONFIG_USER_NS needs to be enabled in order to support user namespace support. Could you verify that for me by running the following command?

grep CONFIG_USER_NS /boot/config-$(uname -r)

[bauerm97]

@DrDaveD hmm, so it appears that RHEL 7.4 might only be able to enable the user namespace by using clone(), as opposed to unshare(). We ran into a similar issue with #777 where CLONE_NEWPID wasn't possible to use with unshare(). Maybe it is possible for us to invoke all of the requested namespaces via a single call to clone(), the same as we do with CLONE_NEWPID.

[DrDaveD]

$ grep CONFIG_USER_NS /boot/config-$(uname -r)
CONFIG_USER_NS=y

[DrDaveD]

@bauerm97 Yes, very good, when I switch to the development branch I see the #803 fix that's now using clone(). A very similar fix is likely needed indeed.

@bbockelm they're still hoping for feedback from you on #803.

[DrDaveD]

Hang on, a colleague showed me that this can be made to work by adding the sysctl setting "user.max_user_namespaces = 15000". It apparently does not require the clone() function after all.

Also I had to switch to using an unpacked image directory rather than an image file, otherwise it says

ERROR  : Could not create /dev/loop0: Permission denied

[bauerm97]

@DrDaveD unfortunately I don't think we can avoid that when running with user namespaces. AFAIK it's impossible to work with loop devices unless you have privileges in the host namespace.

[DrDaveD]

The lack of access to the loop device is not a problem for us; we always read unpacked images out of cvmfs. I just wanted to note it for the record. I'm going to go ahead and close this issue.

[yunchih]

@DrDaveD I got the same permission error you had when running with non-root user:

$ singularity shell vsoch-hello-world-master.simg            
ERROR  : Could not create /dev/loop1: Permission denied
ABORT  : Retval = 255

How did you work around this?
Thank you ( :

[DrDaveD]

@yunchih, as I said "I had to switch to using an unpacked image directory rather than an image file". Mounting image files is a privileged operation; it only works with setuid.

[yunchih]

@DrDaveD Thanks for the reply. Would you mind sharing how you unpack a singularity image file into an image directory and how you invoke the singularity command with that directory? And I'd like to know which binary is required to be setuid. (the mount command?)

It's explicitly explained in the build doc what type of build requires root privilege, but it is not explained in the run doc that it requires root.

Many thanks.

[DrDaveD]

@yunchih There are various ways to get an unpacked image directory, but one way is with the singularity image.export command. To invoke a singularity command with that directory, simply pass the directory path to singularity in the place of an image file. The name of the setuid binaries varies between releases of singularity, but it's automatically set up when you install the package. With singularity-2.4 I believe it is action-suid that's needed to mount an image file to run or to enable overlayfs, and to do bind mounts in the namespace if user namespaces is not supported by the kernel.

[caot]

@DrDaveD can you have a demo that shows how you made it work and paste the console log here. That will save others quite much time. Thanks.

[caot]

It works as a root user, however it's not acceptable in cluster. This information could be helpful to others.

[root@node1001 demo]# singularity pull --name hello-world.simg shub://vsoch/hello-world
Progress |===================================| 100.0% 
Done. Container is at: /home/tangc/pkgs/singularity-2.6.0/demo/hello-world.simg
[root@mgmtnode demo]# singularity shell hello-world.simg
Singularity: Invoking an interactive shell within container...

Singularity hello-world.simg:~> id
uid=0(root) gid=0(root) groups=0(root),1(daemon),2(bin),3(sys),4(adm),6(disk),10(uucp)
Singularity hello-world.simg:~> 

[DrDaveD]

@caot Which issue are you referring to? The one with the /dev/loopN permission failure? There's more than one problem referred to in this issue, and you don't show the failure you're getting without being a root user. What's your host operating system?

[caot]

@DrDaveD I had the following issue as that mentioned above that is the same as link Failed invoking the NEWUSER namespace runtime: Invalid argument

It's a el6 cluster, we expect it to work for none-root users.

ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

[DrDaveD]

Oh, sorry, it's not going to happen. el6 does not support unprivileged user namespaces.