Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Jupyter images cannot be run in a secured multi tenant Docker environment. #188
Although you have tried to set up the Docker images to be able to run as a specific non
In such an environment, as well as enforcing that images cannot be run as
Different user IDs are used to reduce the risk that were someone to break out of the Docker container, even though they wouldn't be
Even with what you have tried to do with ensuring that a non
To work in such an environment a number of changes are required to the Jupyter images. I am going to describe the required changes here and explain why they are needed.
I am happy to provide a pull request containing all the changes, but would only do that if there is an indication that you would be willing to make the changes. If for some reason you are going to baulk on making the changes, then some other solution would need to be found. If no solution can be found that you are happy with, then the Jupyter Notebooks will not be able to be run in such environments and so you would miss out on potential users. Given that such environments are going to become more prevalent in enterprise cloud deployments where security is very important, it is a large space you would be missing out on.
Integer User ID vs Named User
The first problem as mentioned above is the use of:
when closing out the
In a hosting environment which blocks the running of Docker images as
To remedy this issue, the
where 1000 is the actual uid for the
Group That User Belongs To
The next issue is the group that the
This is relying on the default rules in place for the system of adding any user created to the
This is problematic for the case where a hosting environment overrides what uid the container is run as. This is best explained by showing what happens.
As you can see, when no uid is specified, that specified by
If however the uid is overridden using the
This is occurring because the uid being used doesn't correspond to an actual user account defined within the
Hosting environments which use such uid overrides when running a container will actually use very high numbers for the uid, which are going to be well beyond what a
The potential for a solution exists for this in the
Unfortunately this isn't the case as even with this the result will be:
That is, whenever using the
This might be worked around by the hosting environment inspecting the Docker image to query the group and so calling
but such hosting environments that I know of do not do this and so it cannot be relied upon.
Because one cannot depend on
This is not to add
The only concern with this is that you currently advertise in the documentation for each image:
Although using the group
The only other thing one can do is leave
Ownership of Directories and Files
Whether or not you add
This will result in the user owning everything being
Consider now the two scenarios above.
For where a uid override was now done for a uid with no user account, the gid would be that of the
Overriding the HOME Directory
A final thing that needs to be done is to force set the value of the
If this is not done
Using a Docker Shim Layer
To demonstrate the viability of this being able to work one can create a derived Docker image which goes back and fixes up at least the directory and file permissions, plus ownership, the
You can test it works by running with the default user plus an explicit uid passed via the
You will with this shim still have the situation of the running gid being different between
Also note that you wouldn't want to run the changes in group ownership and permissions in a separate step to avoid addition of extra layers. They should be done to the extent necessary as part of every command when Python and/or the packages are installed. Unfortunately Docker doesn't provide a way of specifying a
Reasons for Seeking Changes
As to why am pursuing it, it is because I work for Red Hat and we have our new version of OpenShift which uses Docker and Kubernetes. Because it targets the enterprise space and also multi tenant online hosting, security is very important. The default security mechanisms in Docker are not sufficient to ensure a completely secure environment in multi tenant environments, thus why the extra security provisions around assigned uid's.
I would really love to be able to demonstrate Jupyter Notebooks working using the official Jupyter images rather than having to create my own, or use a small shim Docker image to fix them up so they work. Such solutions are not practical and can't be used by real customers as they would be unsupported in any way.
In addition to at least getting the standard Jupyter images be usable in OpenShift, I would also like to provide additional changes for inclusion which Source to Image (S2I) enable the images. What S2I does is provide a way of being able to use an image as a builder, which combines files from a users Git repository with the base image to create a final deployable image. In the case of Jupyter images, this would provide a one step solution for users to combine notebooks they have with a Jupyter image for deployment without them needing to know how to write a
The S2I system is a standalone project and can be used by users themselves to create new Docker images, but it also has builtin support within OpenShift. This means it becomes a trivial matter for a user to deploy the Jupyter images and have their notebooks automatically copied into the image when run.
The changes required to S2I enable the images are trivial. Being the addition of 5
If you want to go a step further, the S2I scripts could also be extended to allow a file to be provided with a list of additional Python packages to be installed automatically as part of the deploy. In other words, similar to how a
As far as timeline for working to come up with a solution that works for both of us, I am hoping to have something by the time of PyCon US so I can demonstrate it and talk to people interested in Jupyter deployment in the cloud to see how else we can help as far as providing an environment to run it, including helping developing solutions for provision many Jupyter instances for a classroom like teaching environment.
So let me know if you would like to try and find a solution to these issues and I can work on the pull request I mentioned if you don't wish to work it out yourself.
If you want to dig more into these problems, there are few things you can check out.
The first is my repository where I have all the Docker shims to go back and fix the images, plus various templates to help with deployment of Jupyter to OpenShift. This repository can be found at:
I have also blogged extensively about these issues when looking at older IPython images at:
@GrahamDumpleton Thank you for the extensive writeup. I am certainly interested in pursuing how to make the images easier to secure with your assistance.
After reading your notes through once, I have a few questions off the top of my head and might have a few more as I re-read what you've written. Mostly, I'd like to understand if and how what you propose will affect the experience of users pulling these images and running them in a single-user environment (probably the most popular use of these images today).
If we can strike a balance between how easily fly-by users can run a container on a single host, and how easily an enterprise can run containers in a secure multitenant environment, then by all means we should do that here. If not, then refining the scope and purpose of this repo becomes a question to the broader Jupyter team, and that scope should help us decide how to proceed (e.g. maybe this repo becomes the destination for end-users to get images to run as one-offs and another repo becomes the spot for maintaining images built for cloud providers).
Just to add a bit to what @parente has already said with an eye to how this might be done.
Related. I would also want to be very sure that
Also, related. If we do need to make these sorts of changes to our
Finally, any code like this must come with clear documentation. What has been written in this issue description is a good start.
This is a little worrisome to me as well.
Thanks for the detailed post!
I think the proposed groups/filesystem permissions changes should be fine, as long as the user can install packages, which is our main requirement, there.
I think the single-user case is the first priority for these images, and tmpnb/jupyterhub-type cases second. It is not our intention for these to be the way to use Jupyter in docker. These are a way, and our way, but they shouldn't be expanded to cover all possible cases. I'm not sure, yet, whether the different pieces of this proposal fall in scope or not. I'm pretty sure that maintaining enterprise-ready-multi-tenant-security is not something that we can commit to, though, unless you are proposing to take on that responsibility yourself.
The desire is that it shouldn't affect at all the typical 'docker run jupyter/minimal-notebook' experience.
In response to @parente points:
(1) For a file system mount from the Docker host, then user/group that it is running as in the container would apply. This is actually a good argument for not changing the default group from
(2) From my experience I don't trust sticky bits on group in Docker, ie., g+s. I know that for some reason if you try and change the pip cache directory to g+s from the
This makes it impossible to then install any further packages.
Wherever it is done, the outcome needs to be that group is
(3) A hosting service wouldn't be using the
That gid is not inherited from
I have started a discussion in OpenShift about whether OpenShift should itself inherit any group from
(4) I don't know of any security implications of running as gid of 0 and security folks would have looked at that quite closely in OpenShift and given it an okay. Personally the only issue I have seen, is that some Debian packages left some documentation files they installed with group write access for
In response to other follow ups, the only thing I see being raised that would require some care as to how to handle it, is the case where people use the images as a base image to then install further packages. Or even where the install packages from the running notebook and then checkpoint the image. As they would be doing it as
The issue in this case of derived images is what if they tried to take that derived image and use it with a hosting service that used
I will need to think through some more on that specific case and what issues may arise. I will investigate some more what the issue is with g+s on the pip cache directory. I have seen the same issue once before with a directory under HOME which was only o+rwx and used to store some nssdb file. It is something to do with Docker layers. In the prior case I saw it, the problems with bad permissions went away when Docker squash was used to remove layers, but running that is not a solution. They did solve it in some other way, so need to ask what they did.
So let me go sort out the sticky bit issues for the pip cache directory and whether can work out how g+s can be used and whether having that helps in any way.
So they actually solved the prior nssdb problem and strange permissions by recreating the directory which caused the problems.
This didn't involve g+s but was triggered in a different way. I will try recreating the pip cache directory with permissions which have it as group writable to start with. If pip decides it doesn't like that and reverts the permissions that will be a problem. That will be harder for me to test though as have to add pre-create in base image
Thanks @GrahamDumpleton for your attention to this topic. I am not a security expert, but I have spent (wasted) a fair amount of time struggling with setting up non-root users within Docker containers, and still have the containers be usable. Has anyone on your team experimented with Docker user namespaces?
I ask because dealing with all this non-root user business within the container seems like a ton of accidental complexity. It would be so much easier if one could just be
This has been a good discussion back and forth. For a way ahead, there are at least components out of here that can be broken up into smaller PRs. One of these, that I was a bit surprised we didn't already have, is setting
@jtyberg My understanding is that right now Docker user namespaces is not sufficient, or cannot be applied in a practical way, for a multi tenant system. Part of the issue is:
So although it can remap uid ranges such that you could allow
This means that containers for different users cannot use a different range of uid's at the host level.
Thus if someone can break out of the container, even though not
To solve this problem properly such that different containers could be mapped to different uid ranges for file system storage at least, I am told that it would require kernel changes and even if that was something that kernel developers would be prepared to do, that would likely be a long time coming.
One promising thing to mention is that I actually got a favourable response on OpenShift being changed to honour the group specified in
The other technical point of relevance with user namespaces is simply that they're still a relatively new security mechanism at the kernel level, and much harder to systematically audit for potential escape hatches (i.e. using root access in one container to gain root access either on the host or inside a different container), as compared to enforcing a blanket ban on the use of the root UID inside tenant containers.
However, if OpenShift can be updated to honour the group ID in USER statements, that does sound promising. Am I right in thinking that the remaining change requests in that case would be to:
If so, then it would make sense to submit at least those changes, while the discussion about using the default root group vs the Dockerfile specified group can continue on the OpenShift side.
Having OpenShift honour the group may take a while to achieve as looks like that will entail a change to Kubernetes.
You don't per chance know whether it is possible to modify the system wide behaviour that root group is used for gid when process started with uid not in passwd file? If could change it to use
In respect of pip cache permission issues, why is the cache not being disabled in the first place?
When building Docker images, one would normally use
And finally found the comment I thought I had seen before which acknowledges the permissions issue on the pip cache directory is an AUFS issue, one which apparently still hasn't got a resolution.
This means any permissions must always be fixed up in same layer, or for the pip cache, pre create it with more open permissions before any pip install is run.
Or as I note above, disable the pip cache anyway as there is no good reason to have it that I have ever seen for a Docker image.
If you explicitly use
@rgbkrk We are currently relying on HOME to be set by virtue of running as jovyan in the image. In any of the stacks, if you do:
in a Python notebook, you'll see
I think setting HOME in the Docker environment means that when you're running any commands as root, you're still pointing to the jovyan home directory. Not sure that's desirable.
@GrahamDumpleton Thank you for addressing my questions.
I was wondering about the implications of membership to root group if a user breaks out of the container.
I think we can disable the pip cache during the Docker build like you say. I'm more interested in if sticky bits can be used on
I foresee difficulties explaining the find+chmod requirement to every new contributor to the stacks. I also see us forgetting about the requirement during code review and causing regressions unless there are some automated tests put in place.
I think @rgbkrk is right that the next step is to start opening small PRs that address some of the easiest issues first (e.g., disable pip cache during the docker build, set ENV HOME). We can build up to the final solution in increments, and sanity check the impact of each change to the single-user experience along the way.
added a commit
May 30, 2016
added a commit
May 30, 2016
1-year check-in. A few PRs last year addressed low-hanging fruit. I see https://github.com/GrahamDumpleton/openshift3-jupyter-stacks exists with Dockerfiles that touch up permissions starting
@GrahamDumpleton I'm going to close this issue as inactive. If you think what you have in your openshift3-jupyter-stacks repo can merge here without impacting non-OpenShift use, feel free to submit PRs.
I have not created any PRs against your repos because there needed to be a decision made up front on one key issue. I wasn't going to do a lot of work creating a PR against all the different images before the decision was made on the strategy as would be quite a bit of wasted effort on making changes and testing it given wouldn't likely be accepted at this point.
The specific issue which needs to be looked at further is that of changing the group for files, directories and the running process to be the group
So far in the discussion this was a sticking point.
Fixing up these permissions in a derived Docker image as was being done in
I will review where suggested changes were at as far as what has already been done and create a list of what is outstanding. At this point though my general recommendation have been giving people is to ignore the Jupyter Project images.