Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed invoking the NEWUSER namespace runtime: Invalid argument #415

Closed
fwip opened this issue Dec 14, 2016 · 31 comments
Closed

Failed invoking the NEWUSER namespace runtime: Invalid argument #415

fwip opened this issue Dec 14, 2016 · 31 comments
Assignees
@jscook2345

Comments

@fwip
Copy link

fwip commented Dec 14, 2016

I'm trying to test out Singularity on our cluster - I am not an administrator of the cluster, and I don't have root.

I created a few tar images on my laptop (Mac OSX, via vagrant), and sftp'ed them over to the cluster.

On one of the cluster nodes, I compiled and installed singularity with:

./autogen.sh
./configure --with-userns --with-slurm --prefix=$HOME
make && make install

So far so good. However, when I try to run the image, I get the error mentioned in the title:

$ singularity  shell   ~/tmp/tensorflow.tar
Opening tar archive, stand by...
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

Running it with --debug results in the following output:

$ singularity --debug shell   ~/tmp/tensorflow.tar
enabling debugging
ending argument loop
Exec'ing: /home/nelsonjs/libexec/singularity/cli/shell.exec /home/nelsonjs/tmp/tensorflow.tar
Opening tar archive, stand by...
VERBOSE [U=677,P=91936]    message.c:52:message_init()                : Set messagelevel to: 5
DEBUG   [U=677,P=91936]    privilege.c:73:singularity_priv_init()     : Called singularity_priv_init(void)
DEBUG   [U=677,P=91936]    privilege.c:138:singularity_priv_init()    : Returning singularity_priv_init(void)
VERBOSE [U=677,P=91936]    privilege.c:264:singularity_priv_drop()    : Could not restore EUID to 0: Operation not permitted (errno=1).
DEBUG   [U=677,P=91936]    privilege.c:267:singularity_priv_drop()    : Dropping privileges to UID=677, GID=506 (1 supplementary GIDs)
DEBUG   [U=677,P=91936]    privilege.c:269:singularity_priv_drop()    : Restoring supplementary groups
DEBUG   [U=677,P=91936]    privilege.c:286:singularity_priv_drop()    : Confirming we have correct UID/GID
VERBOSE [U=677,P=91936]    sexec.c:72:main()                          : Running NON-SUID program workflow
DEBUG   [U=677,P=91936]    sexec.c:74:main()                          : Checking program has appropriate permissions
VERBOSE [U=677,P=91936]    config_parser.c:112:singularity_config_parse(): Initialize configuration file: /home/nelsonjs/etc/singularity/singularity.conf
DEBUG   [U=677,P=91936]    config_parser.c:124:singularity_config_parse(): Starting parse of configuration file /home/nelsonjs/etc/singularity/singularity.conf
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key allow setuid = 'no'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key allow pid ns = 'no'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key allow user image = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key protected image mode = 'none'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key enable overlay = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key config passwd = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key config group = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key config resolv_conf = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount proc = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount sys = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount dev = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount home = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount tmp = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount hostfs = 'no'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key bind path = '/etc/hosts'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key user bind control = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key mount slave = 'yes'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key container dir = '/var/lib/singularity/mnt'
VERBOSE [U=677,P=91936]    config_parser.c:175:singularity_config_parse(): Got config key singularity user = 'singularity'
DEBUG   [U=677,P=91936]    config_parser.c:183:singularity_config_parse(): Finished parsing configuration file '/home/nelsonjs/etc/singularity/singularity.conf'
VERBOSE [U=677,P=91936]    sexec.c:82:main()                          : Checking that we are allowed to run as SUID
DEBUG   [U=677,P=91936]    config_parser.c:293:_singularity_config_get_bool_chaCalled singularity_config_get_bool(allow setuid, yes)
DEBUG   [U=677,P=91936]    config_parser.c:238:_singularity_config_get_value_imReturning configuration value allow setuid='no'
DEBUG   [U=677,P=91936]    config_parser.c:304:_singularity_config_get_bool_chaReturn singularity_config_get_bool(allow setuid, yes) = 0
VERBOSE [U=677,P=91936]    sexec.c:107:main()                         : Not invoking SUID mode: disallowed by the system administrator
DEBUG   [U=677,P=91936]    util/util.c:104:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_IMAGE'
VERBOSE [U=677,P=91936]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_IMAGE'
DEBUG   [U=677,P=91936]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_IMAGE
DEBUG   [U=677,P=91936]    util/util.c:60:envar()                     : Checking environment variable length (<= 4096): SINGULARITY_IMAGE
DEBUG   [U=677,P=91936]    util/util.c:66:envar()                     : Checking environment variable has allowed characters: SINGULARITY_IMAGE
VERBOSE [U=677,P=91936]    util/util.c:89:envar()                     : Obtained input from environment 'SINGULARITY_IMAGE' = '/tmp/singularity-rundir.ilKVyC62/tensorflow.tar'
VERBOSE [U=677,P=91936]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_COMMAND'
DEBUG   [U=677,P=91936]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_COMMAND
DEBUG   [U=677,P=91936]    util/util.c:60:envar()                     : Checking environment variable length (<= 10): SINGULARITY_COMMAND
DEBUG   [U=677,P=91936]    util/util.c:66:envar()                     : Checking environment variable has allowed characters: SINGULARITY_COMMAND
VERBOSE [U=677,P=91936]    util/util.c:89:envar()                     : Obtained input from environment 'SINGULARITY_COMMAND' = 'shell'
DEBUG   [U=677,P=91936]    action.c:54:singularity_action_init()      : Checking on action to run
DEBUG   [U=677,P=91936]    action.c:62:singularity_action_init()      : Setting action to: shell
DEBUG   [U=677,P=91936]    action.c:94:singularity_action_init()      : Getting current working directory path string
DEBUG   [U=677,P=91936]    rootfs.c:71:singularity_rootfs_init()      : Checking on container source type
DEBUG   [U=677,P=91936]    rootfs.c:79:singularity_rootfs_init()      : Figuring out where to mount Singularity container
DEBUG   [U=677,P=91936]    config_parser.c:238:_singularity_config_get_value_imReturning configuration value container dir='/var/lib/singularity/mnt'
VERBOSE [U=677,P=91936]    rootfs.c:82:singularity_rootfs_init()      : Set image mount path to: /var/lib/singularity/mnt
DEBUG   [U=677,P=91936]    dir.c:44:rootfs_dir_init()                 : Inializing container rootfs dir subsystem
DEBUG   [U=677,P=91936]    util/util.c:94:envar_defined()             : Checking if environment variable is defined: SINGULARITY_WRITABLE
VERBOSE [U=677,P=91936]    util/util.c:96:envar_defined()             : Environment variable is undefined: SINGULARITY_WRITABLE
DEBUG   [U=677,P=91936]    sessiondir.c:60:singularity_sessiondir_init(): Checking Singularity configuration for 'sessiondir prefix'
DEBUG   [U=677,P=91936]    util/util.c:104:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_SESSIONDIR'
VERBOSE [U=677,P=91936]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_SESSIONDIR'
DEBUG   [U=677,P=91936]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_SESSIONDIR
VERBOSE [U=677,P=91936]    util/util.c:56:envar()                     : Environment variable is NULL: SINGULARITY_SESSIONDIR
DEBUG   [U=677,P=91936]    config_parser.c:226:_singularity_config_get_value_imNo configuration entry found for 'sessiondir prefix'; returning default value '/tmp/.singularity-session-'
DEBUG   [U=677,P=91936]    sessiondir.c:75:singularity_sessiondir_init(): Set sessiondir to: /tmp/.singularity-session-677.64770.30406
DEBUG   [U=677,P=91936]    util/file.c:263:s_mkpath()                 : Creating directory: /tmp/.singularity-session-677.64770.30406
DEBUG   [U=677,P=91936]    sessiondir.c:91:singularity_sessiondir_init(): Opening sessiondir file descriptor
DEBUG   [U=677,P=91936]    sessiondir.c:97:singularity_sessiondir_init(): Setting shared flock() on session directory
DEBUG   [U=677,P=91936]    util/util.c:94:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOSESSIONCLEANUP
VERBOSE [U=677,P=91936]    util/util.c:96:envar_defined()             : Environment variable is undefined: SINGULARITY_NOSESSIONCLEANUP
DEBUG   [U=677,P=91936]    util/util.c:94:envar_defined()             : Checking if environment variable is defined: SINGULARITY_NOCLEANUP
VERBOSE [U=677,P=91936]    util/util.c:96:envar_defined()             : Environment variable is undefined: SINGULARITY_NOCLEANUP
DEBUG   [U=677,P=91936]    fork.c:77:prepare_fork()                   : Creating parent/child coordination pipes.
VERBOSE [U=677,P=91936]    fork.c:153:singularity_fork()              : Forking child process
VERBOSE [U=677,P=91936]    fork.c:171:singularity_fork()              : Hello from parent process
DEBUG   [U=677,P=91936]    fork.c:190:singularity_fork()              : Assigning sigaction()s
DEBUG   [U=677,P=91936]    fork.c:221:singularity_fork()              : Creating generic signal pipes
DEBUG   [U=677,P=91936]    fork.c:229:singularity_fork()              : Creating sigchld signal pipes
VERBOSE [U=677,P=91993]    fork.c:157:singularity_fork()              : Hello from child process
DEBUG   [U=677,P=91936]    fork.c:260:singularity_fork()              : Waiting on signal from watchdog
DEBUG   [U=677,P=91993]    fork.c:160:singularity_fork()              : Closing watchdog write pipe
DEBUG   [U=677,P=91993]    fork.c:167:singularity_fork()              : Child process is returning control to process thread
DEBUG   [U=677,P=91993]    ns.c:45:singularity_ns_unshare()           : Unsharing all namespaces
DEBUG   [U=677,P=91993]    user.c:47:singularity_ns_user_configured() : Checking if user namespaces are configured.
DEBUG   [U=677,P=91993]    config_parser.c:293:_singularity_config_get_bool_chaCalled singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=677,P=91993]    config_parser.c:226:_singularity_config_get_value_imNo configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=677,P=91993]    config_parser.c:299:_singularity_config_get_bool_chaReturn singularity_config_get_bool(allow user ns, yes) = 1
DEBUG   [U=677,P=91993]    user.c:91:singularity_ns_user_unshare()    : Attempting to virtualize the USER namespace
ERROR   [U=677,P=91993]    user.c:93:singularity_ns_user_unshare()    : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT   [U=677,P=91993]    user.c:94:singularity_ns_user_unshare()    : Retval = 255
DEBUG   [U=677,P=91936]    fork.c:287:singularity_fork()              : Parent process is exiting
DEBUG   [U=677,P=91936]    util/util.c:104:envar_path()               : Checking environment variable is valid path: 'SINGULARITY_RUNDIR'
VERBOSE [U=677,P=91936]    util/util.c:52:envar()                     : Checking input from environment: 'SINGULARITY_RUNDIR'
DEBUG   [U=677,P=91936]    util/util.c:54:envar()                     : Checking environment variable is defined: SINGULARITY_RUNDIR
DEBUG   [U=677,P=91936]    util/util.c:60:envar()                     : Checking environment variable length (<= 4096): SINGULARITY_RUNDIR
DEBUG   [U=677,P=91936]    util/util.c:66:envar()                     : Checking environment variable has allowed characters: SINGULARITY_RUNDIR
VERBOSE [U=677,P=91936]    util/util.c:89:envar()                     : Obtained input from environment 'SINGULARITY_RUNDIR' = '/tmp/singularity-rundir.ilKVyC62'
DEBUG   [U=677,P=91936]    sessiondir.c:111:singularity_sessiondir_init(): Cleanup thread waiting on child...
DEBUG   [U=677,P=91936]    sessiondir.c:116:singularity_sessiondir_init(): Checking to see if we are the last process running in this sessiondir
VERBOSE [U=677,P=91936]    sessiondir.c:118:singularity_sessiondir_init(): Cleaning sessiondir: /tmp/.singularity-session-677.64770.30406
DEBUG   [U=677,P=91936]    util/file.c:285:s_rmdir()                  : Removing directory: /tmp/.singularity-session-677.64770.30406
VERBOSE [U=677,P=91936]    sessiondir.c:126:singularity_sessiondir_init(): Cleaning run directory: /tmp/singularity-rundir.ilKVyC62
DEBUG   [U=677,P=91936]    util/file.c:285:s_rmdir()                  : Removing directory: /tmp/singularity-rundir.ilKVyC62
$

I've tried a variety of settings in ~/etc/singularity/singularity.conf based on http://singularity.lbl.gov/docs-config, but I haven't hit upon a config that works. I started with the default config, of course.

I believe that the kernel does support user namespaces, but I could be wrong. Here's some info in case it's useful:

$ cat /boot/config-$(uname -a | awk '{print $3}') | grep CONFIG_USER_NS
CONFIG_USER_NS=y
$ uname -a
Linux [hostname] 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/centos-release
CentOS Linux release 7.2.1511 (Core)
@gmkurtzer
Copy link
Contributor

RedHat's implementation of user namespaces is very misleading.... While the kernel reports to support it, and the user space appears to be present, it is considered by Red Hat to be a "technology preview" and thus can only be enabled via a kernel boot argument (and even then, I'm not sure how functional it truly is).

To get proper support on the system, you will need to ask your system administrator to install Singularity to the system as root so it can leverage a set UID code path which does not require the user namespace.

Hope that helps!

@fwip
Copy link
Author

fwip commented Dec 14, 2016

Ah, thank you for your help!

@gmkurtzer
Copy link
Contributor

My pleasure. I closed the ticket, but if you have any additional questions feel free to reopen it, create another or join our Slack or Google Group.

Thanks!

@pelson
Copy link

pelson commented Jan 24, 2017

Worked for me too - FWIW I simply sudo make install to a prefix that was sitting in my (non-root) homespace.

Thanks @gmkurtzer.

@stillill
Copy link

stillill commented Aug 8, 2017 •
edited

I'm having the same problem on my university's HPC cluster even though the Admin installed Singularity 2.3.1 as root. I'm accessing Singularity via modules.

$ singularity exec ubuntu.img ls
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

Here's some system info in case it's helpful:

$ uname --all
Linux [hostname] 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

If anyone has any other ideas let me know. I'd like to get this working on my university's HPC cluster if at all possible.

@DrDaveD
Copy link
Collaborator

DrDaveD commented Aug 15, 2017

@deepdawg, try running singularity with the -vvv option to get debug information on what it is doing. Also make sure that allow setuid = yes in /etc/singularity/singularity.conf (which is the default).

@blakedewey
Copy link

blakedewey commented Sep 26, 2017 •
edited

@DrDaveD I am having similar problems to @deepdawg on my university HPC cluster (also accessed using lmod). I do believe that singularity was installed as root but I got the following debug output:

Increasing verbosity level (5)
Ending argument loop
Singularity version: 2.3-master.gadf5259
Exec'ing: /cm/shared/apps/singularity/libexec/singularity/cli/shell.exec
Evaluating args: '-c -B /scratch -B /data --home /home-1/bdewey3@jhu.edu:/home/bdewey3@jhu.edu work/software/singularity_images/keras208_cuda80_cudnn6_medimg.img'
VERBOSE [U=2027,P=36583]   message_init()                            Set messagelevel to: 5
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Initialize configuration file: /cm/shared/apps/singularity/etc/singularity/singularity.conf
DEBUG   [U=2027,P=36583]   singularity_config_parse()                Starting parse of configuration file /cm/shared/apps/singularity/etc/singularity/singularity.conf
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key bind path = '/cm/shared'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key bind path = '/work-zfs/marccadmin'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key enable overlay = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=2027,P=36583]   singularity_config_parse()                Got config key sessiondir max size = '16'
DEBUG   [U=2027,P=36583]   singularity_config_parse()                Finished parsing configuration file '/cm/shared/apps/singularity/etc/singularity/singularity.conf'
VERBOSE [U=2027,P=36583]   singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'CONTAIN' = '1'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(CONTAIN, 1) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/cm/shared/apps/singularity/libexec'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(libexecdir, /cm/shared/apps/singularity/libexec) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'COMMAND' = 'shell'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'VERSION' = '2.3-master.gadf5259'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(version, 2.3-master.gadf5259) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/cm/shared/apps/singularity/var'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(localstatedir, /cm/shared/apps/singularity/var) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'HOME' = '/home-1/bdewey3@jhu.edu:/home/bdewey3@jhu.edu'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(HOME, /home-1/bdewey3@jhu.edu:/home/bdewey3@jhu.edu) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'DIR' = '/cm/shared/apps/singularity'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(DIR, /cm/shared/apps/singularity) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'IMAGES' = '/scratch/groups/jprince1/software/singularity_images'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(IMAGES, /scratch/groups/jprince1/software/singularity_images) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/cm/shared/apps/singularity/etc'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /cm/shared/apps/singularity/etc) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'BINDPATH' = '/data,/scratch,'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(BINDPATH, /data,/scratch,) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'BINDIR' = '/cm/shared/apps/singularity/bin'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(bindir, /cm/shared/apps/singularity/bin) = 0
VERBOSE [U=2027,P=36583]   singularity_registry_set()                Adding value to registry: 'IMAGE' = 'work/software/singularity_images/keras208_cuda80_cudnn6_medimg.img'
DEBUG   [U=2027,P=36583]   singularity_registry_set()                Returning singularity_registry_set(IMAGE, work/software/singularity_images/keras208_cuda80_cudnn6_medimg.img) = 0
DEBUG   [U=2027,P=36583]   singularity_registry_get()                Returning value from registry: 'HOME' = '/home-1/bdewey3@jhu.edu:/home/bdewey3@jhu.edu'
DEBUG   [U=2027,P=36583]   singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=2027,P=36583]   singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=2027,P=36583]   singularity_priv_init()                   Initializing user info
DEBUG   [U=2027,P=36583]   singularity_priv_init()                   Set the calling user's username to: bdewey3@jhu.edu
DEBUG   [U=2027,P=36583]   singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=2027,P=36583]   singularity_priv_init()                   Obtaining home directory
VERBOSE [U=2027,P=36583]   singularity_priv_init()                   Set home (via SINGULARITY_HOME) to: /home/bdewey3@jhu.edu
VERBOSE [U=2027,P=36583]   singularity_priv_init()                   Set the home directory (via SINGULARITY_HOME) to: /home-1/bdewey3@jhu.edu
VERBOSE [U=2027,P=36583]   singularity_suid_init()                   Running NON-SUID program workflow
DEBUG   [U=2027,P=36583]   singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=2027,P=36583]   singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=2027,P=36583]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=2027,P=36583]   singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=2027,P=36583]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG   [U=2027,P=36583]   singularity_priv_userns()                 Attempting to virtualize the USER namespace
ERROR   [U=2027,P=36583]   singularity_priv_userns()                 Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT   [U=2027,P=36583]   singularity_priv_userns()                 Retval = 255

@smoe smoe mentioned this issue Mar 26, 2018
Closed
@ODiogoSilva ODiogoSilva mentioned this issue Jul 3, 2018
Closed
@charlesreid1 charlesreid1 mentioned this issue Jul 12, 2018
Open
@caot
Copy link

caot commented Aug 16, 2018

@stillill did you figure out the issue?

@mforde84
Copy link

im still having the same issues. singularity was install to root level system paths, though my users are still getting namespace errors

$ uname -r
2.6.32-573.12.1.el6.x86_64

$ singularity -vvv shell hello-world.simg
Increasing verbosity level (4)
Singularity version: 2.5.2-dist
Exec'ing: /usr/local/libexec/singularity/cli/shell.exec
Evaluating args: 'hello-world.simg'
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /usr/local/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Initializing Singularity Registry
VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec'
VERBOSE: Adding value to registry: 'COMMAND' = 'shell'
VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4'
VERBOSE: Adding value to registry: 'VERSION' = '2.5.2-dist'
VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var'
VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc'
VERBOSE: Adding value to registry: 'BINDIR' = '/usr/local/bin'
VERBOSE: Adding value to registry: 'IMAGE' = 'hello-world.simg'
VERBOSE: Set home (via getpwuid()) to: /home/a.cri.mforde
VERBOSE: Running NON-SUID program workflow
VERBOSE: Invoking the user namespace
ERROR : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT : Retval = 255

@caot
Copy link

caot commented Aug 17, 2018 •
edited

singularity couldn't parse the follows. Thank you @mforde84

$ singularity shell hello-world.simg -vvv

@DrDaveD
Copy link
Collaborator

DrDaveD commented Aug 17, 2018

@mforde84 Indeed unprivileged user namespaces are not supported on el6. They're supported on el7 only as a technology preview. I can give you a pointer to instructions for enabling it there if you want to try it on el7.

@caot yes the -vvv has to come immediately after the singularity command.

@caot caot mentioned this issue Aug 17, 2018
Closed
@caot
Copy link

caot commented Aug 17, 2018 •
edited

@mforde84

The follows works for root user, however it's not acceptable in cluster.

[none-root@node-1001 demo]$ singularity shell hello-world.simg
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

[none-root@node-1001 demo]$ su
[root@node-1001 demo]# singularity shell hello-world.simg
Singularity: Invoking an interactive shell within container...

Singularity hello-world.simg:~>

@mforde84
Copy link

mforde84 commented Aug 17, 2018 •
edited

Ok, this is really confusing, why does this work?

$ sudo /usr/local/bin/singularity create container-centos7-test.img
...
$ sudo /usr/local/bin/singularity bootstrap container-centos7-test.img centos.def
...
$ /usr/local/bin/singularity shell container-centos7-test.img
Singularity: Invoking an interactive shell within container...

Singularity.container-centos7-test.img>

@mforde84
Copy link

Im sorry, I don't understand what you're trying to say.

@caot
Copy link

caot commented Aug 17, 2018

@mforde84 It could be helpful https://youtu.be/29NLgM9fnh4?t=437

@mforde84
Copy link

mforde84 commented Aug 17, 2018 •
edited

I'm not clear how this is relevant to my question. I'm being told that I can only run containers with suid due to my kernel headers, yet I'm still able to generate a container and run it as an unprivileged user. Some clarification on why this works, yet other containers don't would be helpful. That or how I can generate containers with different kernel versions to support later versions of glibc would be helpful.

@DrDaveD
Copy link
Collaborator

DrDaveD commented Aug 17, 2018

@mforde84 Assuming you're still running on el6, if you are successfully invoking a container as an unprivileged user, you must now be using a singularity with setuid enabled. The setuid bit is not on the singularity executable itself, it is on a helper executable ending with "-suid" in /usr/local/libexec/singularity/bin. -vvv should tell you whether it is using the NON-SUID workflow or not; the previous one you posted said it was.

@mforde84
Copy link

mforde84 commented Aug 17, 2018 •
edited

Yep,

VERBOSE: Checking for sexec-suid at /usr/local/libexec/singularity/sexec-suid
VERBOSE: Invoking SUID sexec: /usr/local/libexec/singularity/sexec-suid

Is there a way I can force this behavior for dockers I pull or build from someone elses repo/tags? Just for functionality sake? For instance, the hello world container (from above) is running from non-suid, can I throw a flag to the command forcing the suid path? Or can I completely disable user namespace execution with a compile / configure option?

@DrDaveD
Copy link
Collaborator

DrDaveD commented Aug 17, 2018

The image file run should make no difference to that, as far as I know. Are you sure you didn't change something? Try switching back and forth between the images with the same singularity installation.

If you're still seeing sexec-suid you must have an old version. For a while it has been called action-suid. I'm quite sure that all versions that use sexec have security vulnerabilities. Please upgrade.

@mforde84
Copy link

mforde84 commented Aug 17, 2018 •
edited

Yea, I'm running 2.2.1 I believe. I can upgrade. Still testing to make sure this works as intended.

So just a clarification, should configuration and the initial make be performed by root user? e.g,

sudo su -
./configure
make
make install

Thats the only real differences I can see across my test cases. Maybe version difference as well. The installation that allows set uid was compiled as follows:

su - nonrootuser
./configure
make
sudo make install

@DrDaveD
Copy link
Collaborator

DrDaveD commented Aug 17, 2018

Yes the second one is the right way to do it. Also ./autogetn.sh

If you're using el6 I advise getting it from EPEL. I support the rpm there. 2.6.0 is in epel-testing and will be in epel next week.

@mforde84
Copy link

mforde84 commented Aug 17, 2018 •
edited

Great. Thanks. One additional question, and I can generate another ticket if you prefer. But one thing I'm running into issues with other peoples containers is a mount permission error:

eg.,
ERROR : Failed to mount image in (read only): Invalid argument
ABORT : Retval = 255

If I understand correctly, it's, by default, trying to mount to loop block device which a read only fs. Do you have suggestions on how to mount to a shared path with read/write/exec mount flags, say within a lmod sourced build directory?

@DrDaveD
Copy link
Collaborator

DrDaveD commented Aug 17, 2018

Please create a new issue with all the details on how to reproduce. What you've given isn't sufficient for me to think of anything helpful.

@lstagner
Copy link

Hi, I am running into the same error when running on my local cluster.

$uname --all
Linux irisa.cluster 2.6.32-754.10.1.el6.x86_64 #1 SMP Tue Jan 15 17:07:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$singularity --version
2.6.1-HEAD.9103f01

$singularity -d shell soft.simg
Exec'ing: /fusion/usc/opt/singularity/libexec/singularity/cli/shell.exec
Evaluating args: 'soft'
VERBOSE [U=6336,P=3039]    message_init()                            Set messagelevel to: 5
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Initialize configuration file: /fusion/usc/opt/singularity/etc/singularity/singularity.conf
DEBUG   [U=6336,P=3039]    singularity_config_parse()                Starting parse of configuration file /fusion/usc/opt/singularity/etc/singularity/singularity.conf
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount devpts = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key enable overlay = 'try'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key sessiondir max size = '16'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key allow container squashfs = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key allow container extfs = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key allow container dir = 'yes'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key memory fs type = 'tmpfs'
VERBOSE [U=6336,P=3039]    singularity_config_parse()                Got config key always use nv = 'no  '
DEBUG   [U=6336,P=3039]    singularity_config_parse()                Finished parsing configuration file '/fusion/usc/opt/singularity/etc/singularity/singularity.conf'
DEBUG   [U=6336,P=3039]    singularity_config_get_value_impl()       Returning configuration value always use nv='no  '
VERBOSE [U=6336,P=3036]    message_init()                            Set messagelevel to: 5
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Initialize configuration file: /fusion/usc/opt/singularity/etc/singularity/singularity.conf
DEBUG   [U=6336,P=3036]    singularity_config_parse()                Starting parse of configuration file /fusion/usc/opt/singularity/etc/singularity/singularity.conf
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount devpts = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key enable overlay = 'try'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key sessiondir max size = '16'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key allow container squashfs = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key allow container extfs = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key allow container dir = 'yes'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key memory fs type = 'tmpfs'
VERBOSE [U=6336,P=3036]    singularity_config_parse()                Got config key always use nv = 'no  '
DEBUG   [U=6336,P=3036]    singularity_config_parse()                Finished parsing configuration file '/fusion/usc/opt/singularity/etc/singularity/singularity.conf'
VERBOSE [U=6336,P=3036]    singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/fusion/usc/opt/singularity/libexec'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(libexecdir, /fusion/usc/opt/singularity/libexec) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'COMMAND' = 'shell'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'VERSION' = '2.6.1-HEAD.9103f01'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(version, 2.6.1-HEAD.9103f01) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/fusion/usc/opt/singularity/var'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(localstatedir, /fusion/usc/opt/singularity/var) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/fusion/usc/opt/singularity/etc'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /fusion/usc/opt/singularity/etc) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'BINDIR' = '/fusion/usc/opt/singularity/bin'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(bindir, /fusion/usc/opt/singularity/bin) = 0
VERBOSE [U=6336,P=3036]    singularity_registry_set()                Adding value to registry: 'IMAGE' = 'soft'
DEBUG   [U=6336,P=3036]    singularity_registry_set()                Returning singularity_registry_set(IMAGE, soft) = 0
DEBUG   [U=6336,P=3036]    singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=6336,P=3036]    singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=6336,P=3036]    singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=6336,P=3036]    singularity_priv_init()                   Initializing user info
DEBUG   [U=6336,P=3036]    singularity_priv_init()                   Set the calling user's username to: stagnerl
DEBUG   [U=6336,P=3036]    singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=6336,P=3036]    singularity_priv_init()                   Obtaining home directory
VERBOSE [U=6336,P=3036]    singularity_priv_init()                   Set home (via getpwuid()) to: /home/stagnerl
VERBOSE [U=6336,P=3036]    singularity_suid_init()                   Running NON-SUID program workflow
DEBUG   [U=6336,P=3036]    singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=6336,P=3036]    singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=6336,P=3036]    singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=6336,P=3036]    singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=6336,P=3036]    singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG   [U=6336,P=3036]    singularity_priv_userns()                 Attempting to virtualize the USER namespace
ERROR   [U=6336,P=3036]    singularity_priv_userns()                 Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT   [U=6336,P=3036]    singularity_priv_userns()                 Retval = 255

The administrator compiled singularity from source using the following commands (he didn't want to pull it EPEL and singularity 3.0 has too many dependencies)

./autogen.sh
./configure --prefix=/usr/local --sysconfdir=/etc
make
sudo make install

only changing the configure paths. These commands should work or are there some configure flags missing? What needs to be done to get things working?

@DrDaveD
Copy link
Collaborator

DrDaveD commented Feb 14, 2019

The key verbose message is "Running NON-SUID program workflow". Does /fusion/usc/opt/singularity/libexec/singularity/bin/action-suid exist, and does it have setuid root permissions? If so maybe /fusion doesn't allow executing setuid binaries.

@lstagner
Copy link

action-suid does exist but it does not have setuid permissions

-rwxr-xr-x 1 admin usc 263992 Feb 13 13:31 action
-rwxr-xr-x 1 admin usc 606390 Feb 13 13:31 action-suid
...

The directory where singularity is installed has permissions drwxrwsx-x

So what is the fix? Its it just a matter of changing the permissions of the files in libexec/singularity/bin with chmod or does singularity need to be installed in a different place?

@DrDaveD
Copy link
Collaborator

DrDaveD commented Feb 14, 2019

Something messed with the ownership of the files, and that probably cleared the setuid bit. Yes the *suid files in that directory need to be owned by root and chmod u+s.

@lstagner
Copy link

Thanks that fixed it. Also for future readers the singularity.conf file also needs to owned by root.

@luyang93
Copy link

@mforde84 Indeed unprivileged user namespaces are not supported on el6. They're supported on el7 only as a technology preview. I can give you a pointer to instructions for enabling it there if you want to try it on el7.

@caot yes the -vvv has to come immediately after the singularity command.

I came out the same problem, how to enable user namespaces unser el7.
I install singularity using conda.

$ singularity -vvv -d shell hello-world.simg 
Increasing verbosity level (4)
Enabling debugging
Ending argument loop
Singularity version: 2.6.0-master.72a2295
Exec'ing: /PATH/TO/CONDA/conda_env/onecellpipe/libexec/singularity/cli/shell.exec
Evaluating args: 'hello-world.simg'
VERBOSE [U=1011,P=28675]   message_init()                            Set messagelevel to: 5
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Initialize configuration file: /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
DEBUG   [U=1011,P=28675]   singularity_config_parse()                Starting parse of configuration file /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount devpts = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key enable overlay = 'try'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key sessiondir max size = '16'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key allow container squashfs = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key allow container extfs = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key allow container dir = 'yes'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key memory fs type = 'tmpfs'
VERBOSE [U=1011,P=28675]   singularity_config_parse()                Got config key always use nv = 'no  '
DEBUG   [U=1011,P=28675]   singularity_config_parse()                Finished parsing configuration file '/PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf'
DEBUG   [U=1011,P=28675]   singularity_config_get_value_impl()       Returning configuration value always use nv='no  '
VERBOSE [U=1011,P=28671]   message_init()                            Set messagelevel to: 5
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Initialize configuration file: /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
DEBUG   [U=1011,P=28671]   singularity_config_parse()                Starting parse of configuration file /PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount devpts = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key enable overlay = 'try'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key sessiondir max size = '16'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key allow container squashfs = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key allow container extfs = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key allow container dir = 'yes'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key memory fs type = 'tmpfs'
VERBOSE [U=1011,P=28671]   singularity_config_parse()                Got config key always use nv = 'no  '
DEBUG   [U=1011,P=28671]   singularity_config_parse()                Finished parsing configuration file '/PATH/TO/CONDA/conda_env/onecellpipe/etc/singularity/singularity.conf'
VERBOSE [U=1011,P=28671]   singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/libexec'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(libexecdir, /PATH/TO/CONDA/conda_env/onecellpipe/libexec) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'COMMAND' = 'shell'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'VERSION' = '2.6.0-master.72a2295'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(version, 2.6.0-master.72a2295) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/var'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(localstatedir, /PATH/TO/CONDA/conda_env/onecellpipe/var) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/etc'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /PATH/TO/CONDA/conda_env/onecellpipe/etc) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'BINDIR' = '/PATH/TO/CONDA/conda_env/onecellpipe/bin'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(bindir, /PATH/TO/CONDA/conda_env/onecellpipe/bin) = 0
VERBOSE [U=1011,P=28671]   singularity_registry_set()                Adding value to registry: 'IMAGE' = 'hello-world.simg'
DEBUG   [U=1011,P=28671]   singularity_registry_set()                Returning singularity_registry_set(IMAGE, hello-world.simg) = 0
DEBUG   [U=1011,P=28671]   singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=1011,P=28671]   singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=1011,P=28671]   singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=1011,P=28671]   singularity_priv_init()                   Initializing user info
DEBUG   [U=1011,P=28671]   singularity_priv_init()                   Set the calling user's username to: luyang
DEBUG   [U=1011,P=28671]   singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=1011,P=28671]   singularity_priv_init()                   Obtaining home directory
VERBOSE [U=1011,P=28671]   singularity_priv_init()                   Set home (via getpwuid()) to: /home/luyang
VERBOSE [U=1011,P=28671]   singularity_suid_init()                   Running NON-SUID program workflow
DEBUG   [U=1011,P=28671]   singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=1011,P=28671]   singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=1011,P=28671]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=1011,P=28671]   singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=1011,P=28671]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
DEBUG   [U=1011,P=28671]   singularity_priv_userns()                 Attempting to virtualize the USER namespace
ERROR   [U=1011,P=28671]   singularity_priv_userns()                 Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT   [U=1011,P=28671]   singularity_priv_userns()                 Retval = 255

@jscook2345
Copy link
Contributor

@luyang93

Can you move your comment to a new issue and link to this one since it's over two years old and closed?

Additionally, can you include the output of:

cat /proc/sys/user/max_user_namespaces

Thanks!

@jscook2345 jscook2345 self-assigned this Jul 24, 2019
@DrDaveD
Copy link
Collaborator

DrDaveD commented Jul 25, 2019

For the record, el7.6 supports user namespaces without being a technology preview. It just needs to be enabled, for example with

echo "user.max_user_namespaces = 15000" > /etc/sysctl.d/90-max_user_namespaces.conf
sysctl -p /etc/sysctl.d/90-max_user_namespaces.conf

@brianjohnhaas brianjohnhaas mentioned this issue Nov 9, 2021
Open
This was referenced Apr 5, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jscook2345 jscook2345

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests