Environment Var For --allow-run-as-root

[ax3l]

Background information

What version of Open MPI are you using?

openmpi 3.0.0

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

from source, via the spack package manager inside a Docker container.

Please describe the system on which you are running

• Operating system/version: Ubuntu 16.04
• Computer hardware: x86
• Network type: Ethernet

---

Details of the problem

I would like to control the --allow-run-as-root flag via an environment variable (or config file) as well.

Running as root in Docker containers is fine and partitioning by exporting an environment variable (e.g. autotools equivalent is FORCE_UNSAFE_CONFIGURE=1) would make it way easier to interact with mpirun/mpiexec for deeply integrated workflows (where directly editing the final mpi call might not always be possible).

[jsquyres]

I hear what you're saying, but I'm not sure it's the right approach. We put the --allow-run-as-root option there only with very, very deep reservations. We strongly advise against using it. HPC / MPI jobs should not need to be run as root.

Specifically: --allow-run-as-root is already bad enough. I really, really do not want to provide another trapdoor to already-unsafe functionality; that's just going in the wrong direction. Is there a better reason to do it other than "all the other kids in Dockerland do it"?

If Docker requires running MPI jobs as root, it seems like a better solution would be to fix that (i.e., enable Docker jobs to not run as root).

Just my $0.02; sorry to be a grumpy old man...

[ax3l]

Thank you for the detailed answer. I understand your concerns and I fully share your thoughts on regular systems.

Nevertheless, there is an important difference to be aware of when running something in a Docker container. Running containers are fully isolated (tm), which means it is not being started as hostsytem root (but from a user with user rights). Only internally, inside the container, the (container-)root user is active - which really is fine and save. The only thing that can happen is, that the "active copy" of an image, which when it is running is called container, self-destructs. But this does not affect the image it came from (which can be restarted just from that state) nor does it harm the host system.

"all the other kids in Dockerland do it"

Yes, it's a fundamental crazyness in Docker to be inside as a new user (root) and assumes something like full isolation really exists. Other solutions such as Singularity spin this way better, keeping the external user transparently as the in-container process owner and adding a transparent layer on top of the FS. Of course, I can also add a local user just to satisfy MPI inside docker (which is still another user), but this is adding no safety net (since it is already there) and is just a hard to maintain work-around.

Long story short: for Docker --allow-run-as-root is really needed, it's very save to use it in Docker since it can not destroy things that are not "checkpointed" in images.

[bwbarrett]

@ax3l, when we've looked at best practices for running MPI in Docker containers (under Amazon ECS or AWS Batch), we didn't run into any real issues using a "normal" user to run MPI applications, and were much happier with the overall posture of the system. Yes, you're right that it's the container and not the host system running a root process, but there's still plenty of reason to be careful. For example, you do create some natural blast radius for filesystems mounted into your container (like an NFS/Lustre system with everyone's data on it).

[ax3l]

using a "normal" user to run MPI applications

I did that as well as a work-around in the past but this complicates, e.g. the import of such Docker images into other solutions such as Singularity. Also, see the argument below about "who creates images" for a HPC workflow (the user).

[we] were much happier with the overall posture of the system

Can you please elaborate a little on that? In what sense?

In an ideal world, a (containerized) process should never have more rights (including FS access) than the user that executed it, independent of what userid it presents internally. In that sense, one would not mount a HPC file system directly inside docker but on the host system and then forward it with the same rights a user has.

filesystems mounted into your container

I would not do that anyway (see above) but let FS mounts always be managed by the actual host system. Then only "docker mount" a user-visible view on a FS. One also needs to think about container workflows, where users should be allowed & flexible enough to adjust their containers and then could do anything to the FS if they control the direct mount / lustre clients / ... We would not want that.

[jsquyres]

I'm not sure I understand your reply -- you seem to be saying that the container should be running with limited rights on the host. I was not under the impression that that was (currently) possible. E.g., if you are root in the container, you can be root on the underlying filesystem.

Regardless, yes, we know that the --allow-run-as-root option is awkward. It's intended to be, if for no other reason than the emphasize that MPI jobs should never be run as root.

Meaning / to be blunt: "it's awkward" is not a strong enough reason for us to implement another trapdoor to already-unsafe behavior. Is there something you can't do -- in any way -- without having --allow-run-as-root functionality being able to be invoked via an env variable and/or config file?

[rhc54]

err...allow-run-as-root is only available on the cmd line - there is no envar or config file option for controlling it. I believe that actually was the request

[jsquyres]

I know. I'm asking if there's something that can't be done (via any mechanism) unless that functionality is invoked via an env var or config file.

[ax3l]

Yes, running simple mpi programs and test suites that just run mpiexec will not run in containers out of the box without sed-ing through them.

Isn't that bad enough? I think a person that sets a central config file to allow run as root or even an environment variable should be aware of what he/she is doing, we don't need to make it awkward.

[zach-nervana]

I just ran into this issue as well. Requiring non-root in docker can be very inconvenient, since many existing dockerfiles and recipes assume a root user. Running as a non-root user often requires extra work. I understand that having some default safety features can be nice, but this particular method for implementing it is resulting in everyone using docker to either mix infrastructure details into their code, or modify the rest of their infrastructure which uses (or shares dockerfile images with code which runs) mpi to allow running as non-root.

[jsquyres]

@ax3l To be clear: that was not my question. I asked if there was something that can't be done (via any mechanism) unless that functionality is provided via an env var or config file.

sed-ing through a Dockerfile qualifies as "any mechanism".

I'm trying to ask if there is a technical reason to require an environment variable or config file for some reason. The "I don't want to modify my Dockerfile" motivation is already established -- but that's not a technical reason. As I said above, we do provide an [intentionally] awkward way to get this functionality. Is there a technical reason to require that we provide this functionality via env var and/or config file option (other than "I don't want to modify my Dockerfile")?

Additionally: is anyone leaning on Docker to fix this scenario? Forcing everything to run as root is not good practice, and is the real root of the problem (see what I did there? 😉).

One thing I'm not clear on: even if we provide an env var or config file option for this functionality, you have to add that to the container somehow (i.e., modify/add this to the Dockerfile), right? Am I missing something?

[zach-nervana]

There isn't anything that can't currently be done inside of docker with the current code (by any mechanism).

To be more clear about why we see this is an issue, having to modify a dockerfile at all is not the problem. The problem is that it requires making changes to components completely unrelated to mpi as running as root is a standard practice in docker. For example, if you already have a recipe for installing or configuring a package for use as root used throughout your infrastructure, you will no longer be able to include that recipe in your dockerfile without conditioning that recipe on whether or not mpi will also be used. Then you would also need to maintain a root and a non-root recipe for every component. Alternatively, you could modify every component in your system to support non-root, just to use mpi. This additional coupling between mpi 'config' and every other component of the system is the problem. Setting an environment variable on the other hand is not going to have any unintended consequences with the rest of the system.

[jsquyres]

@zach-nervana Just to be clear: you can run Open MPI jobs as root, you simply have to specify --allow-run-as-root on the mpirun command line. We strongly recommend not running as root, but you can certainly do so if you wish. So I'm not sure I follow the "Open MPI requires modifying every recipe in the system" argument...?

Setting an environment variable on the other hand is not going to have any unintended consequences with the rest of the system.

Isn't the same thing true with adding --allow-run-as-root?

[zach-nervana]

This is true, but as you've mentioned you don't really want your code to always have that flag set as it may be unsafe for many cases. The decision to run as root is a deployment level concern and so would be most naturally be set inside of a docker container/through an environment variable instead of baking it into your code, which might be deployed by different people in different contexts.