can buildah run in a docker container?

[rawlingsj]

I'd like to use buildah inside a an OpenShift / Kubernetes pod. So I'm testing buildah from inside a docker container however buildah bud and buildah run commands fail with:

ERRO[0000] 'overlay' is not supported over overlay
ERRO[0000] 'overlay' is not supported over overlay
ERRO[0000] backing file system is unsupported for this graph driver
backing file system is unsupported for this graph driver

[rhatdan]

I think you should have /var/lib/containers/ volume mounted into the container which should work. Since that storage would not be on a overlay file system then. Right now you are building a container on top of a /var/lib/container/storage inside.

[rawlingsj]

Thanks @rhatdan that fixed it although I had to run the docker container as privileged for it to work. Are there any plans or is it even possible to use buildah in a non privileged docker container?

[rhatdan]

It could potentially run some workloads without requiring too much privilege,
You would need to add capability SYS_ADMIN since it is mounting shares. It should be able to work with SELinux enforcing on it though. @nalind WDYT

[rawlingsj]

My main use case is to build images with fabric8 / OpenShift.io using non privileged pods and also push (using skopeo?) that image to external registries.

[rawlingsj]

FWIW these are the steps I took to test this incase it's helpful to others. I have a Mac so decided to use minishift so I could mount the /var/lib/containers/ dir.

I created a Dockerfile which includes the example from the blog http://www.projectatomic.io/blog/2017/06/introducing-buildah/

eval $(minishift docker-env)
git clone https://github.com/rawlingsj/buildah.git
cd buildah 
docker build -t  test/buildah:dev .
docker run --privileged -v /var/lib/containers/:/var/lib/containers/ -ti test/buildah:dev bash
buildah bud -t hellofromcontainer .
buildah run hellofromcontainer-working-container

[rhatdan]

Could you see if this would work with a couple of tweeks.

docker run --cap-add=SYS_ADMIN -v /var/lib/containers/:/var/lib/containers/:Z -ti test/buildah:dev bash

That if this works it proves that buildah would work with a more confined container, then privileged.

BTW
buildah run != docker run.

It is really equivalent of executing the RUN statement inside of a Dockerfile. It will execute a the sepcified command inside the buildcontainer using runc.

[rawlingsj]

It is really equivalent of executing the RUN statement inside of a Dockerfile. It will execute a the sepcified command inside the buildcontainer using runc.

Ah ok, so we'd still need to be able to perform buildah run.

Could you see if this would work with a couple of tweeks.
docker run --cap-add=SYS_ADMIN -v /var/lib/containers/:/var/lib/containers/:Z -ti test/buildah:dev bash

Sure thing, so bud works and run fails:

buildah run hellofromcontainer-working-container

[root@7e09af59a10a example]# buildah run hellofromcontainer-working-container
container_linux.go:259: starting container process caused "process_linux.go:257: getting pipe fds for pid 74 caused \"readlink /proc/74/fd/0: permission denied\""

[mariusgrigoriu]

The SYS_ADMIN capability is as good as root. http://forums.grsecurity.net/viewtopic.php?f=7&t=2522
Building container images without requiring root would be a benefit to those of us who want to build images in shared CI environments without single-use virtual machines.

[rhatdan]

That article is comparing CAPABILITIES. Not CAPABILITIES while in a container. With things like SELinux further locking the processes down. Namespaces/seccomp would provide some hurdles needed to break through as well.

That being said, I am not sure how easily someone can break out of the containment with SYS_ADMIN. First thing would be to remount the kernel file systems to read/write and see if you could modify the kernel to break out.

@rawlingsj The difficult part here is figuring out what was blocked.

Could you see if you have any AVC's. ausearch -m avc -ts recent

[ae6rt]

I'm interested in building container images within a container, as the original poster is.

I'm using Docker for Mac, and do

On the Mac

$ sudo mkdir -p /private/var/lib/containers

$ sudo chmod -R ugo+rwx /private/var/lib/containers

$ docker run --cap-add=SYS_ADMIN -v /private/var/lib/containers/:/var/lib/containers/:rw,Z -ti fedora bash

Within the container

# dnf install -y buildah

# buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
 72.07 MiB / 72.07 MiB [=======================================================]
ERRO[0009] error pulling image "fedora": Error writing blob: Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

Any ideas on how to fix this?

Thanks.

[rhatdan]

I have no idea, do you have any AVC's from the host?

[ae6rt]

The version of Linux used by Docker for Mac is some Alpine variant that has no SELinux enabled as far as I can tell.

(The buildah docker image in the docker-run commands below is nothing more than fedora with dnf install -y buildah run on it).

But this is interesting: consider basing a new buildah-image on ubuntu, and then another one on alpine. The ubuntu build fails, but not in the same way the fedora build above fails. The alpine build succeeds.

Ubuntu

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah from ubuntu
Getting image source signatures
Copying blob sha256:d5c6f90da05dc7e77d2e5fef63c341ab05ba2a03396ab5ae8f18814a7bbf5265
 43.44 MiB / 45.07 MiB [=====================================================>-]
 45.07 MiB / 45.07 MiB [=======================================================]
ERRO[0006] error pulling image "ubuntu": Error writing blob: Error processing tar file(exit status 1): function not implemented 

Alpine

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah from alpine
Getting image source signatures
Copying blob sha256:88286f41530e93dffd4b964e1db22ce4939fffa4a4c665dab8591fbab03d4926
 0 B / 1.90 MiB [--------------------------------------------------------------]
Copying config sha256:7328f6f8b41890597575cbaadc884e7386ae0acc53b747401ebce5cf0d624560
 0 B / 1.48 KiB [--------------------------------------------------------------]
Writing manifest to image destination
Storing signatures
 1.90 MiB / 1.90 MiB [=========================================================]
alpine-working-container

More on fedora

Circling back to the fedora build, because the failure message mentioned a specific file (/root/.bash_logout), if you set --debug on buildah while building the fedora image, you get this excerpt

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah --debug from fedora
...
DEBU[0001] GET https://registry-1.docker.io/v2/library/fedora/blobs/sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e 
DEBU[0002] Detected compression format gzip             
 0 B / 72.07 MiB [-------------------------------------------------------------]DEBU[0002] Using original blob without modification     
DEBU[0002] Applying tar in /var/lib/containers/storage/overlay/4a78003b84c8ead429ff490634b32ac9cb15c1b5edadfd4c997c97c9a521cc71/diff 
 72.07 MiB / 72.07 MiB [=======================================================]DEBU[0008] error importing layer blob "sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e" as "4a78003b84c8ead429ff490634b32ac9cb15c1b5edadfd4c997c97c9a521cc71": Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

ERRO[0009] error pulling image "fedora": Error writing blob: Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

If you pull that fedora layer manually with

curl -L -O -H"Authorization: Bearer ${token}"   https://registry-1.docker.io/v2/library/fedora/blobs/sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e

and inspect the layer tar file, you see

$ tar tvf sha256\:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
...
dr-xr-x---  0 0      0           0 Jul  5 14:48 root/
-rw-r--r--  0 0      0          18 Feb 11  2017 root/.bash_logout
-rw-r--r--  0 0      0         176 Feb 11  2017 root/.bash_profile
-rw-r--r--  0 0      0         176 Feb 11  2017 root/.bashrc
-rw-r--r--  0 0      0         100 Feb 11  2017 root/.cshrc
-rw-r--r--  0 0      0         129 Feb 11  2017 root/.tcshrc
...

I still don't know what it all means, but more info is better than less :)

[TomSweeneyRedHat]

I'll try the AVC's again. This was on a Fedora rather than a MAC host. After posting my previous comment I realized I'd not chmod the file as @ae6rt had.

[root@58f3375e03c8 /]# buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
72.07 MiB / 72.07 MiB [=======================================================]
ERRO[0012] error pulling image "fedora": Error writing blob: Error processing tar file(exit status 1): error unmounting root: permission denied

Aug 17 18:08:29 localhost.localdomain audit[2284]: AVC avc: denied { unmount } for pid=2284 comm="exe" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:39 localhost.localdomain dockerd-current[1000]: [4.1K blob data]
Aug 17 18:08:39 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:39 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:39 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
:

[ae6rt]

I'm sorry, I should have stated my buildah version: it is

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah version
Version:       0.3
Go Version:    go1.8.1
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    e9748f0
Built:         Thu Jul 20 20:45:03 2017
OS/Arch:       linux/amd64

[giuseppe]

@ipbabble could we merge our two efforts?

My work is here:

projectatomic/atomic-system-containers#162

A system container is still able to run as a Docker container, I just needed to add some more metadata.

[ipbabble]

@giuseppe Eager to chat. I'm not sure what you mean by merge our two efforts. I did write up my use of vfs for the inside container file system build. I also wrote about why, despite being cool that it works, it's also not a very practical solution. (Due to every container potentially pulling down the same image on a single host.)

Let's connect on IRC (freenode #podman) so we can discuss why I might be missing in my effort.

[ipbabble]

I've got a PR atomic-site for a blog post.

1. It should be noted that it covers only the privileged container mode right now.
2. It covers some of the issues we faced with unprivileged.
3. I pulled the unpriv section for a later blog because I saw some regression that I am looking into.
4. @giuseppe 's work has been incorporated into a separate blog because this one was getting too long.

This blog is framed as the Buildah container for Kubernetes.
The system container Buildah example will be for a Buildah system container for Atomic
The unpriv. example will be a sequel to the first one.

I'll post the link when it goes live.
Any questions?

[ipbabble]

projectatomic/atomic-site#529

[ipbabble]

http://www.projectatomic.io/blog/2018/03/building-buildah-container-image-for-kubernetes/

[ashcrow]

@ipbabble great post! I did notice there's a hiccup in the markdown to youtube in the post.

[ipbabble]

Yeah. I'm bummed I missed that. We all missed my misplaced quotes. I'm trying to fix it. @rossturk gave me heads up too. Cheers. William William Henry Senior Consulting Software Engineer Red Hat Colorado, USA.

…

On Thu, Mar 1, 2018 at 11:11 AM, Stephen Milner ***@***.***> wrote: @ipbabble <https://github.com/ipbabble> great post! I did notice there's a hiccup in the markdown to youtube in the post. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#158 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPpgXIKXcG1fz4Ye7fT8YXuTzS1C5QQks5taDnRgaJpZM4ODYiG> .

[ipbabble]

So I'm going to create a new/separate issue about non priv. Buildah container

[rhatdan]

The answer to this question is yes, so I am closing this issue. We have other issues about running with UserNamespace and in un priv containers.

[Conan-Kudo]

Since this issue is getting linked around the internets, I'll leave in here a little note about how we solved this problem for us:

For our GitLab CI setup, we made sure that the /var/lib/containers in the container environment mapped to a different directory on the host system, so that the containers wouldn't be picked up by the container executor. That keeps us from accidentally breaking things in the CI system.