Can't run image from dockerhub related to read-only image despite --writable flag

[josh-marshall-jax]

Version of Singularity:

$ singularity version
3.2.1-1.1.el7

Expected behavior

Build and run an image from Dockerhub similarly to docker

Actual behavior

$ sudo singularity run --writable -e -C docker://rabbitmq:3
WARNING: can't determine current working directory: stat .: permission denied
WARNING: can't determine current working directory
WARNING: While attempting to set up SIFOverlay: no SIF writable overlay partition found
chown: changing ownership of '/var/lib/rabbitmq': Read-only file system

Steps to reproduce this behavior

singularity run --writable -e -C docker://rabbitmq:3

What OS/distro are you running

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How did you install Singularity

yum install singularity

[dtrudg]

Apologies - the CLI help string for --writable isn't explicit that using it requires a writable overlay partition in the container SIF, which is not something that is implicitly created when using singularity run docker://xxxxx.

If you do not need the changes in the container to be persistent (which they cannot be when running directly from docker://) then you can use the --writable-tmpfs flag

dave@piran~> sudo singularity run --writable-tmpfs -e -C docker://rabbitmq:3
2019-10-01 17:50:13.762 [info] <0.8.0> Feature flags: list of feature flags found:
2019-10-01 17:50:13.762 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2019-10-01 17:50:13.783 [info] <0.228.0> 
 Starting RabbitMQ 3.7.18 on Erlang 22.1.1
 Copyright (C) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/

  ##  ##
  ##  ##      RabbitMQ 3.7.18. Copyright (C) 2007-2019 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>

              Starting broker...
2019-10-01 17:50:13.783 [info] <0.228.0> 
 node           : rabbit@piran
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : 3c167+0S+Pu9IgBpJTLnWg==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@piran
2019-10-01 17:50:13.792 [info] <0.228.0> Running boot step pre_boot defined by app rabbit
...

There is more documentation around persisting changes, via persistent overlays, here: https://sylabs.io/guides/3.4/user-guide/persistent_overlays.html

[josh-marshall-jax]

I was thinking that it was something like that, just something almost documented. I'll try that and see what it does.

[dtrudg]

See also: https://github.com/sylabs/singularity/issues/3220 - this definitely needs some doc attention, so I'll open an issue here: https://github.com/sylabs/singularity-userdocs/issues/248

[josh-marshall-jax]

Still not working:

$ singularity run --writable-tmpfs docker://rabbitmq:3
touch: cannot touch '/etc/rabbitmq/rabbitmq.conf': Permission denied
$ sudo singularity run --writable-tmpfs docker://rabbitmq:3
WARNING: can't determine current working directory: stat .: permission denied
WARNING: can't determine current working directory
touch: cannot touch '/etc/rabbitmq/rabbitmq.conf': Permission denied

[dtrudg]

WARNING: can't determine current working directory: stat .: permission denied
WARNING: can't determine current working directory
touch: cannot touch '/etc/rabbitmq/rabbitmq.conf': Permission denied

Are you running the sudo singularity in a location that root does not have permissions to access, e.g. a root-squashed filesystem?

I believe there may have been some tweaks to CWD handling in container startup in releases since 3.2 - 3.4.1 is the current release, if you are able to try using that.

[josh-marshall-jax]

I don't control the version used on the deployed machines, so I'm sticking with 3.2. I'll try running this in /tmp soon.

[josh-marshall-jax]

Running it logged into the root user user /root worked with the singularity scripts from https://github.com/sylabs/examples/blob/master/message-broker/rabbitmq/README.md .

Doing the same from the dockerhub image gives the following:

singularity run --writable-tmpfs docker://rabbitmq:3
touch: cannot touch '/etc/rabbitmq/rabbitmq.conf': Permission denied

[jmstover]

If you're using --writable-tmpfs ... then it still falls under standard permissions. Since you don't seem to be using sudo to run it, your user doesn't have write permission to the /etc/rabbitmq directory.

If you look at the examples use case, we have a command in the definition file that does:

chmod -R 777 /var/lib/rabbitmq /etc/rabbitmq /var/log/rabbitmq /tmp/rabbitmq-ssl

So, we're explicitly opening up some directory permissions, and also bind mounting host directories into locations where the process in the image will write to.

These writable flags come down to:
--writable -- Use with a sandbox image, to make changes to that sandbox
--writable-tmpfs -- Creates a tmpfs overlay when running, but permissions still will control on if you can write to an area or not. If you don't have write permission on the underlying file system, you can't write to the tmpfs overlay.

Note: I think that if you have an old ext based image, you can use that and --writable will allow that image to be modified. But, there were a lot of consistency issues, and images becoming corrupt when they were being written to.

[josh-marshall-jax]

I was running everything as the root user directly, so it should have permissions on everything except for the /home directories which are squashed. What ended up working was an upgrade to 3.4 since the machine had just not been updated in awhile, and as root running RABBITMQ_DEFAULT_USER=guest RABBITMQ_DEFAULT_PASS=guest singularity run --writable-tmpfs docker://rabbitmq:3

I'd be nice to have a way for this use case to run unprivileged. I'm not sure what that would involve, or if it is even feasible.

[jmstover]

I'd be nice to have a way for this use case to run unprivileged

It would be a couple step process ... but as a normal user you can try:

singularity build --sandbox rabbitmq_3 docker://rabbitmq:3

At this point, the user should own everything in the rabbitmq_3 directory. There could be some permission issues where there isn't write permission on some directories, or similar, though. Then, once you have the sandbox built, try running like:

RABBITMQ_DEFAULT_USER=guest RABBITMQ_DEFAULT_PASS=guest singularity run --writable rabbitmq_3/

The other was is to build a SIF file, so your definition you'll build from would start with:

Bootstrap: docker
From: rabbitmq:3

In the %post section, open the permissions up on any directories a normal user would need to be able to write to. And then you can bind mound in directories from the host to write to, use --writable-tmpfs, or both. But you can't use --writable on a SIF file because of squashfs.

[josh-marshall-jax]

I'm running into issues with phantom instances which cause new instances to fail or files not be removed. May be a new ticket?

$ singularity instance list
INSTANCE NAME    PID      IMAGE
rabbitmq         26871    /home/marshjo/rabbitmq.sif
$ kill -9 26871
-bash: kill: (26871) - No such process
$ singularity instance list
INSTANCE NAME    PID      IMAGE
rabbitmq         26871    /home/marshjo/rabbitmq.sif
$ rm -rf rabbitmq_3/
rm: cannot remove ‘rabbitmq_3/lib/x86_64-linux-gnu/.nfs00000002093f4e8200000003’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/lib/x86_64-linux-gnu/.nfs0000000202fc747b00000002’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/lib/x86_64-linux-gnu/.nfs00000002093f4e7900000001’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/etc/resolv.conf’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/etc/passwd’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/etc/group’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/etc/hosts’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/var/tmp’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/tmp’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/dev’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/home’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/usr/local/lib/erlang/erts-10.5.1/bin/.nfs00000002018d6dea00000004’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/sys’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/proc’: Device or resource busy
rm: cannot remove ‘rabbitmq_3/.singularity.d/actions’: Device or resource busy

[dtrudg]

@josh-marshall-jax - if you are still having issues with phantom instances on the latest version of Singularity (3.5.2) please do open a new ticket. There were fixes to handling of instance files through the 3.x versions of Singularity. Thanks.