Reprozip wrapping singularity executions

[gkiar]

It would be fantastic (for me, anyways), if Reprzip could wrap Singularity executions. Discussed in VIDA-NYU/reprozip#294 is an issue in which singularity requires suid to be run, but process tracing does not enable suid as it is a security risk.

Do you have a proposed solution, or potential modification that may enable this type of integration to work? Apologies if I've missed an issue addressing this directly. Thanks so much for your help!

cc: @remram44

[cclerget]

@gkiar I don't know exactly how reprozip works, if it capture content directly from memory process to create files/directory/symlinks, or if it records path and copy all files after/during execution. For the second case you will need to run tracer inside container, otherwise you won't be able to access container files because Singularity use namespaces. As far as I know Reprozip requires a tight integration with Singularity.

I see two options :

• install reprozip inside container and trace process inside container
• by using user namespace singularity exec -u /tmp/image (with user namespace you can only use directory as container image), but same remark, reprozip need to resolve paths to container image, if there is an option to prepend a path that could work otherwise reprozip will fail to create archive

[gkiar]

Hi @cclerget, thanks so much for your response! I agree with your first option, and that's something we've considered. I've not thought about the second option, but will investigate to see if I can make something work. Before that, could you please clarify what you mean by "use directory as container image"? How would I configure such a directory/environment? Thanks again!

[cclerget]

@gkiar Singularity use two format for container, sandbox directory and single file image (squashfs, ext3). You can create a sandbox directory image like this :

$singularity build --sandbox /tmp/ubuntu docker://ubuntu

Sandbox directories don't requires privileges to use them, that's why sandbox directory can work with user namespaces

[gkiar]

Great, that's a terrific tip @cclerget! Is there a way to easily convert between these two formats? I.e., can I singularity pull shub://myname/myimage:version and then do something like singularity convert2sandbox ...? If I could get a sandbox directory from an image file I download, I think this problem would be totally solved. Thanks!!

[cclerget]

@gkiar Yes, you can do that with build command like this:
singularity build --sandbox /tmp/converted /tmp/image-from-shub.img

[gkiar]

Fantastic, I'll give this a shot and let you know. Thank you, @cclerget !!

[gkiar]

Hi @cclerget - I have given your suggestion a try, and have run into an issue. Below is my command and log:

$ singularity -vvv exec -u ${PWD}/myimage-dir
Increasing verbosity level (4)
Singularity version: 2.5.1-dist
Exec'ing: /opt/software/singularity-2.5.1/libexec/singularity/cli/exec.exec
Evaluating args: '-u /home/gkiar/myimage-dir'
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /opt/software/singularity-2.5.1/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Initializing Singularity Registry
VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/opt/software/singularity-2.5.1/libexec'
VERBOSE: Adding value to registry: 'COMMAND' = 'exec'
VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4'
VERBOSE: Adding value to registry: 'NOSUID' = '1'
VERBOSE: Adding value to registry: 'VERSION' = '2.5.1-dist'
VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/opt/software/singularity-2.5.1/var'
VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/opt/software/singularity-2.5.1/etc'
VERBOSE: Adding value to registry: 'BINDIR' = '/opt/software/singularity-2.5.1/bin'
VERBOSE: Adding value to registry: 'IMAGE' = '/home/gkiar/myimage-dir'
VERBOSE: Set home (via getpwuid()) to: /home/gkiar
VERBOSE: Running NON-SUID program workflow
VERBOSE: Invoking the user namespace
ERROR  : Failed invoking the NEWUSER namespace runtime: Invalid argument
ABORT  : Retval = 255

Thank you!

[cclerget]

Hey @gkiar, what is the distribution you use ?

[gkiar]

Version of singularity? 2.5.1-dist

Otherwise, I'm not exactly sure the distribution of what you're referring to. Thanks!

[cclerget]

I mean linux distribution like Ubuntu, CentOS ... and the version too

[gkiar]

Ah, sorry about that! CentOS Linux release 7.4.1708 (Core)

[cclerget]

Ok, the message Failed invoking the NEWUSER namespace runtime: Invalid argument appears because user namespaces are disabled, you can enable user namespaces with sysctl user.max_user_namespaces=10000 and see if it works

[gkiar]

Hi @cclerget - this is a shared machine which I don't have sudo access on, is there a way to get around setting the sysctl flag?

[cclerget]

@gkiar No solution unfortunately, I think your option is to install reprozip in container

[gkiar]

Thank you @cclerget - this was a very helpful process. I consider this resolved from your perspective. :) Cheers!