Support running podman containers inside unprivileged (docker) container #4131
Comments
openshift-ci-robot
added
the
kind/feature
|
This is possible with UML but extremely slow |
|
Thanks for the link @AkihiroSuda, however it doesn't seem like it supports running containers, in my testing. I'm not sure if I'm doing something wrong, or maybe it doesn't support running inside |
|
UML should be able to run containers. (both podman-in-docker and docker-in-podman) Something seems wrong with either Podman or sysctl. |
|
Ok thanks I will debug it a bit more! |
|
@giuseppe Would this be possible with rootless containers running with fuse-overlay? We would need setuid and setgid to handle setting up a namespace. We could try this out with podman in podman. I think the issue with running podman in Docker is the tighter seccomp controls. Docker seccomp.json mistakenly blocks all mount syscalls, even though non privileged mount syscall is allowed for procfs, tmpfs, bind, fuse and sysfs, I believe. |
|
I think UID/GID mapping will also be an issue. You'll either need the storage flag to ignore chown errors, or a separate newuidmap/newgidmap setup within the container - and I suspect you won't have the privileges to run them in a rootless container. |
|
podman in LXD seems to work fine (unprivileged LXD container created with -c security.nesting=true, same option as for running Docker in LXD). Here is a debug log for rootless if it is any help for developing this feature. |
|
@AkihiroSuda @giuseppe Ideas? |
does the container work fine if you use |
|
I reproduced the LXD issue. slirp4netns sandbox doesn't seem to work on LXD with slirp4netns --configure --mtu=65520 --disable-host-loopback $(cat /tmp/pid) --enable-sandbox tap0
WARNING: Support for sandboxing is experimental
sent tapfd=5 for tap0
received tapfd=5
Starting slirp
* MTU: 65520
* Network: 10.0.2.0
* Netmask: 255.255.255.0
* Gateway: 10.0.2.2
* DNS: 10.0.2.3
* Recommended IP: 10.0.2.100
cannot mount tmpfs on /tmp
create_sandbox failed
do_slirp is exiting
do_slirp failed
parent failed
yes |
|
strace: |
|
This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days. |
|
I believe Dan is working on this. We may need a few capabilities that
aren't available in an unprivileged container - SETUID was mentioned
…On Sun, Nov 17, 2019, 19:09 github-actions[bot] ***@***.***> wrote:
This issue had no activity for 30 days. In the absence of activity or the
"do-not-close" label, the issue will be automatically closed within 7 days.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#4131>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3AOCBSL5AA4HOQZYPENQDQUHMKLANCNFSM4I3D3OMA>
.
|
|
Yes I have a working prototype of this, now, will publish a blog on it shortly. I think we could get some additional support into containers.conf to make this easier to do. |
|
Just to be clear, the issue specifically mentions a use case:
Is this supported by the prototype? |
|
How is this possible? With seccomp=unconfined apparmor=unconfined? |
|
Currently have to disable SELinux since by default it blocks a few commands I have a modified seccomp.json file also. BTW I have been sending out updates on podman.io mailing list. Since I don't use apparmor, I would figure it is similar to SELinux. Main SELinux issues were on mounting file systems. |
|
This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days. |
|
Still being worked on |
|
It has been back burnered especially until after the break. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
This is the main PR to get this working. |
|
@rhatdan @AkihiroSuda Hi. I wanted to use Podman to build images in my Gitlab CI pipeline using Kubernetes executor and I have set a restricted PSP (Using exactly this: https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) where no capabilities are added and root access is disabled. Since the build is running in kubernetes directly, there is no docker involved, its directly running within unpriviliged containerd with no root access in Kubernetes. I get the same error as mentioned: And this is how the sample pipeline looks: image: "quay.io/podman/stable"
buildah:
tags:
- development
- ops
variables:
STORAGE_DRIVER: "vfs"
BUILDAH_FORMAT: "docker"
IMAGE_TAG: $CI_REGISTRY_IMAGE:edge
script:
- podman version
- whoami
- echo "Logging into $CI_REGISTRY"
- podman login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
- podman build -t ${IMAGE_TAG} .
- podman images
- podman push ${IMAGE_TAG}
- podman logout $CI_REGISTRYMay I know how I can get this to work? Would it not work with the current PSP? Should I be changing something else? Thanks. |
|
I have been reading this thread, but kind of lack to see how I can run podman inside a docker container. From what I understand this is resolved in this PR #4698. Would be great if someone can point out the exact requirement to run podman successfully inside a docker container. A docker-compose or cli example is very much appreciated. Currently I run into the following issue $ docker run --rm -it quay.io/podman/stable /bin/bash
$ podman pull alpine
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: operation not permitted |
|
It would definitely need to be --privileged, and have a volume mounted on /var/lib/containers. |
|
So the PR mentioned above doesn't resolve yet the fact that a privileged container is required? E.g. for Github Actions or Gitlab CI. From a security point of view you can't really run a privileged container there. |
|
@marcofranssen, I'm not sure about the PR mentioned, but I have verified that podman now works in an unprivileged docker container with some caveats. One that tripped me up is that the host kernel must support deferred deletion. Here's a simple docker image that lets you check that quickly: Assuming that works, you can run podman in docker like this: Edit: |
|
buildah with chroot mode with vfs works in rootless podman. So building containers no longer requires root anymore thanks to podman/buildah. Naturally you can also run in buildah isolation mode, but it's just chroot and there are some gotchas. |
|
It also does not work with sysbox. $ docker run --rm --runtime=sysbox-runc quay.io/podman/stable podman run hello-world
Trying to pull registry.fedoraproject.org/hello-world...
manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/hello-world...
name unknown: Repo not found
Trying to pull registry.centos.org/hello-world...
manifest unknown: manifest unknown
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689
Copying config sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
Writing manifest to image destination
Storing signatures
time="2020-10-16T21:07:09Z" level=error msg="error unmounting /var/lib/containers/storage/overlay/68c6ae97224fb5a4a2f602ce295a0a09fb2860bdb3a4320338077065dc64bc59/merged: invalid argument"
Error: error mounting storage for container 8bbe8b1eb1e37035079cb309bbec9d507c1a4efda2ade4dbd8ae729a8a029b9b: error creating overlay mount to /var/lib/containers/storage/overlay/68c6ae97224fb5a4a2f602ce295a0a09fb2860bdb3a4320338077065dc64bc59/merged: using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1@ctalledo, @rodnymolina maybe it's because |
|
Podman is pulling down the content. @giuseppe Any ideas? This looks close. |
|
A different error happens. :) ❯ docker run --rm --device=/dev/fuse --runtime=sysbox-runc quay.io/podman/stable podman run hello-world
Trying to pull registry.fedoraproject.org/hello-world...
manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/hello-world...
name unknown: Repo not found
Trying to pull registry.centos.org/hello-world...
manifest unknown: manifest unknown
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689
Copying config sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b
Writing manifest to image destination
Storing signatures
Error: cannot chown /var/lib/containers/storage/overlay/765c5b5697c416f4ed9d1b577141466df3ab4afb60941e1b86b68506a3863cf8/merged to 0:0: chown /var/lib/containers/storage/overlay/765c5b5697c416f4ed9d1b577141466df3ab4afb60941e1b86b68506a3863cf8/merged: operation not permitted |
|
I think seccomp rules from Docker are to tight. Try with --security-opt seccomp=unconfined Or use the seccomp.json file from Podman /usr/share/containers/seccomp.json |
|
@felipecrs, thanks for letting us know, i've filed this one to track Sysbox fix. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Any updates here? |
|
This is a docker issue, not something that can be fixed in Podman. |
|
So, podman run --rm quay.io/podman/stable podman run hello-worldWas supposed to work? |
|
Not quite. Sadly that seems broken now, I will play around with it. |
|
This works (rootless privileged), But this doesn't. (rootfull non-privileged) I believe if the latter is solved it'll work for docker as well. |
|
If anyone has cycles to help us improve Sysbox (a new type of runc), then running Podman inside unprivileged containers (deployed by Docker, Podman, or even K8s) should be doable and not too difficult to implement. Docker + Sysbox containers can already run Docker, systemd, and even K8s inside unprivileged (rootless) containers. And do so without requiring the vfs driver (i.e., using the native overlayfs driver). |
|
@ctalledo is there anything that doesn't work if we just use |
|
@giuseppe, last time i checked podman worked to instantiate outer containers, but there are a few things we need to iron out to claim full support, such as the uid-mapping approach to follow to be able to instantiate inner containers either with podman or docker (not working right now w/o hacks). As Cesar said, we are not that far off, need some more time (or help :-) to have it done. |
github-actions
bot
added
the
locked - please file new issue/PR
and others
/kind feature
Description
Very similar to #4056 but with the exception that the host container is an unprivileged (docker) container.
The specific use case is being able to programmatically create and destroy containers while running inside an unprivileged container, for automated tests in CI environments such as CircleCI and Github actions.
The comments by @mheon (#4056 (comment)) imply this is currently impossible, and may never be possible, but I'd like to explore in more detail the feasibility of this separately from that issue.
The stack overflow discussion https://stackoverflow.com/q/56032747 seems to touch on the same problem, and unfortunately come to the same conclusion, that
--privilegedis required at this time, which makes it impossible to use in CircleCI and Github Actions.Steps to reproduce the issue:
Describe the results you received:
At the moment the error I'm getting looks like this:
Describe the results you expected:
I expected to be able to run a container inside a container.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Additional environment details (AWS, VirtualBox, physical, etc.):
Running on bare metal (laptop)
The text was updated successfully, but these errors were encountered: