New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can buildah run in a docker container? #158

Closed
rawlingsj opened this Issue Jun 23, 2017 · 134 comments

Comments

Projects
None yet
@rawlingsj
Copy link

rawlingsj commented Jun 23, 2017

I'd like to use buildah inside a an OpenShift / Kubernetes pod. So I'm testing buildah from inside a docker container however buildah bud and buildah run commands fail with:

ERRO[0000] 'overlay' is not supported over overlay
ERRO[0000] 'overlay' is not supported over overlay
ERRO[0000] backing file system is unsupported for this graph driver
backing file system is unsupported for this graph driver
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Jun 23, 2017

I think you should have /var/lib/containers/ volume mounted into the container which should work. Since that storage would not be on a overlay file system then. Right now you are building a container on top of a /var/lib/container/storage inside.

@rawlingsj

This comment has been minimized.

Copy link

rawlingsj commented Jun 23, 2017

Thanks @rhatdan that fixed it although I had to run the docker container as privileged for it to work. Are there any plans or is it even possible to use buildah in a non privileged docker container?

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Jun 23, 2017

It could potentially run some workloads without requiring too much privilege,
You would need to add capability SYS_ADMIN since it is mounting shares. It should be able to work with SELinux enforcing on it though. @nalind WDYT

@rawlingsj

This comment has been minimized.

Copy link

rawlingsj commented Jun 23, 2017

My main use case is to build images with fabric8 / OpenShift.io using non privileged pods and also push (using skopeo?) that image to external registries.

@rawlingsj

This comment has been minimized.

Copy link

rawlingsj commented Jun 23, 2017

FWIW these are the steps I took to test this incase it's helpful to others. I have a Mac so decided to use minishift so I could mount the /var/lib/containers/ dir.

I created a Dockerfile which includes the example from the blog http://www.projectatomic.io/blog/2017/06/introducing-buildah/

eval $(minishift docker-env)
git clone https://github.com/rawlingsj/buildah.git
cd buildah 
docker build -t  test/buildah:dev .
docker run --privileged -v /var/lib/containers/:/var/lib/containers/ -ti test/buildah:dev bash
buildah bud -t hellofromcontainer .
buildah run hellofromcontainer-working-container
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Jun 23, 2017

Could you see if this would work with a couple of tweeks.

docker run --cap-add=SYS_ADMIN -v /var/lib/containers/:/var/lib/containers/:Z -ti test/buildah:dev bash

That if this works it proves that buildah would work with a more confined container, then privileged.

BTW
buildah run != docker run.

It is really equivalent of executing the RUN statement inside of a Dockerfile. It will execute a the sepcified command inside the buildcontainer using runc.

@rawlingsj

This comment has been minimized.

Copy link

rawlingsj commented Jun 23, 2017

It is really equivalent of executing the RUN statement inside of a Dockerfile. It will execute a the sepcified command inside the buildcontainer using runc.

Ah ok, so we'd still need to be able to perform buildah run.

Could you see if this would work with a couple of tweeks.
docker run --cap-add=SYS_ADMIN -v /var/lib/containers/:/var/lib/containers/:Z -ti test/buildah:dev bash

Sure thing, so bud works and run fails:

buildah run hellofromcontainer-working-container

[root@7e09af59a10a example]# buildah run hellofromcontainer-working-container
container_linux.go:259: starting container process caused "process_linux.go:257: getting pipe fds for pid 74 caused \"readlink /proc/74/fd/0: permission denied\""
@mariusgrigoriu

This comment has been minimized.

Copy link

mariusgrigoriu commented Jun 27, 2017

The SYS_ADMIN capability is as good as root. http://forums.grsecurity.net/viewtopic.php?f=7&t=2522
Building container images without requiring root would be a benefit to those of us who want to build images in shared CI environments without single-use virtual machines.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Jun 29, 2017

That article is comparing CAPABILITIES. Not CAPABILITIES while in a container. With things like SELinux further locking the processes down. Namespaces/seccomp would provide some hurdles needed to break through as well.

That being said, I am not sure how easily someone can break out of the containment with SYS_ADMIN. First thing would be to remount the kernel file systems to read/write and see if you could modify the kernel to break out.

@rawlingsj The difficult part here is figuring out what was blocked.

Could you see if you have any AVC's. ausearch -m avc -ts recent

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 17, 2017

I'm interested in building container images within a container, as the original poster is.

I'm using Docker for Mac, and do

On the Mac

$ sudo mkdir -p /private/var/lib/containers

$ sudo chmod -R ugo+rwx /private/var/lib/containers

$ docker run --cap-add=SYS_ADMIN -v /private/var/lib/containers/:/var/lib/containers/:rw,Z -ti fedora bash

Within the container

# dnf install -y buildah

# buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
 72.07 MiB / 72.07 MiB [=======================================================]
ERRO[0009] error pulling image "fedora": Error writing blob: Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

Any ideas on how to fix this?

Thanks.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 17, 2017

I have no idea, do you have any AVC's from the host?

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 17, 2017

The version of Linux used by Docker for Mac is some Alpine variant that has no SELinux enabled as far as I can tell.

(The buildah docker image in the docker-run commands below is nothing more than fedora with dnf install -y buildah run on it).

But this is interesting: consider basing a new buildah-image on ubuntu, and then another one on alpine. The ubuntu build fails, but not in the same way the fedora build above fails. The alpine build succeeds.

Ubuntu

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah from ubuntu
Getting image source signatures
Copying blob sha256:d5c6f90da05dc7e77d2e5fef63c341ab05ba2a03396ab5ae8f18814a7bbf5265
 43.44 MiB / 45.07 MiB [=====================================================>-]
 45.07 MiB / 45.07 MiB [=======================================================]
ERRO[0006] error pulling image "ubuntu": Error writing blob: Error processing tar file(exit status 1): function not implemented 

Alpine

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah from alpine
Getting image source signatures
Copying blob sha256:88286f41530e93dffd4b964e1db22ce4939fffa4a4c665dab8591fbab03d4926
 0 B / 1.90 MiB [--------------------------------------------------------------]
Copying config sha256:7328f6f8b41890597575cbaadc884e7386ae0acc53b747401ebce5cf0d624560
 0 B / 1.48 KiB [--------------------------------------------------------------]
Writing manifest to image destination
Storing signatures
 1.90 MiB / 1.90 MiB [=========================================================]
alpine-working-container

More on fedora

Circling back to the fedora build, because the failure message mentioned a specific file (/root/.bash_logout), if you set --debug on buildah while building the fedora image, you get this excerpt

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah --debug from fedora
...
DEBU[0001] GET https://registry-1.docker.io/v2/library/fedora/blobs/sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e 
DEBU[0002] Detected compression format gzip             
 0 B / 72.07 MiB [-------------------------------------------------------------]DEBU[0002] Using original blob without modification     
DEBU[0002] Applying tar in /var/lib/containers/storage/overlay/4a78003b84c8ead429ff490634b32ac9cb15c1b5edadfd4c997c97c9a521cc71/diff 
 72.07 MiB / 72.07 MiB [=======================================================]DEBU[0008] error importing layer blob "sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e" as "4a78003b84c8ead429ff490634b32ac9cb15c1b5edadfd4c997c97c9a521cc71": Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

ERRO[0009] error pulling image "fedora": Error writing blob: Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

If you pull that fedora layer manually with

curl -L -O -H"Authorization: Bearer ${token}"   https://registry-1.docker.io/v2/library/fedora/blobs/sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e

and inspect the layer tar file, you see

$ tar tvf sha256\:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
...
dr-xr-x---  0 0      0           0 Jul  5 14:48 root/
-rw-r--r--  0 0      0          18 Feb 11  2017 root/.bash_logout
-rw-r--r--  0 0      0         176 Feb 11  2017 root/.bash_profile
-rw-r--r--  0 0      0         176 Feb 11  2017 root/.bashrc
-rw-r--r--  0 0      0         100 Feb 11  2017 root/.cshrc
-rw-r--r--  0 0      0         129 Feb 11  2017 root/.tcshrc
...

I still don't know what it all means, but more info is better than less :)

@TomSweeneyRedHat

This comment has been minimized.

Copy link
Collaborator

TomSweeneyRedHat commented Aug 17, 2017

I'll try the AVC's again. This was on a Fedora rather than a MAC host. After posting my previous comment I realized I'd not chmod the file as @ae6rt had.

[root@58f3375e03c8 /]# buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
72.07 MiB / 72.07 MiB [=======================================================]
ERRO[0012] error pulling image "fedora": Error writing blob: Error processing tar file(exit status 1): error unmounting root: permission denied

Aug 17 18:08:29 localhost.localdomain audit[2284]: AVC avc: denied { unmount } for pid=2284 comm="exe" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:39 localhost.localdomain dockerd-current[1000]: [4.1K blob data]
Aug 17 18:08:39 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:39 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:39 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
Aug 17 18:08:40 localhost.localdomain audit[2278]: AVC avc: denied { unmount } for pid=2278 comm="buildah" scontext=system_u:system_r:container_t:s0:c600,c868 tcontext=system_u:object_r:fs_t:s0 tclass=filesystem permissive=0
:

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 17, 2017

I'm sorry, I should have stated my buildah version: it is

$ docker run --cap-add=SYS_ADMIN -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah version
Version:       0.3
Go Version:    go1.8.1
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    e9748f0
Built:         Thu Jul 20 20:45:03 2017
OS/Arch:       linux/amd64
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

Ok so lets start by putting the machine into permissive mode to gather the AVC's. It would probably be best to run "buildah" containers under a different context or with SELinux Disabled in the container. SELinux will definitely block mounting behaviour.

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

For my part, using Docker for Mac, there is no SELinux:

/ # cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.5.0
PRETTY_NAME="Alpine Linux v3.5"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"
/ # apk info | grep selinux
/ # ls -ld /etc/se*
-rw-r--r--    1 root     root            71 Aug 17 16:00 /etc/securetty
-rw-r--r--    1 root     root         36141 Nov 25  2016 /etc/services
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

So your VM is running alpine?

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

Yes. When a user installs Docker for Mac, you get an Alpine VM as the actual docker host.

https://www.docker.com/docker-mac

@TomSweeneyRedHat

This comment has been minimized.

Copy link
Collaborator

TomSweeneyRedHat commented Aug 18, 2017

FWIW. Running a Fedora VM (not making any changes to it) then doing the commands @ae6rt (Mark) had:

# sudo mkdir -p /private/var/lib/containers
# sudo chmod -R ugo+rwx /private/var/lib/containers
# docker run --cap-add=SYS_ADMIN -v /private/var/lib/containers/:/var/lib/containers/:rw,Z -ti fedora bash

Then in the container  'vi /etc/sysconfig/docker' and change this line:
  OPTIONS='--selinux-enabled --log-driver=journald'
to
  OPTIONS='--log-driver=journald'

[root@653f673b728c /]# dnf install -y buildah
[root@653f673b728c /]# buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
 72.07 MiB / 72.07 MiB [=======================================================]
Copying config sha256:49236bc2f0da4105c84cfb7238b48879efb489a62fe8536934434cf2072a2319
 0 B / 2.29 KiB [--------------------------------------------------------------]
Writing manifest to image destination
Storing signatures
fedora-working-container

No selinux, but otherwise happy sauce.

Mark, TYVM! for telling us about this and for working through this issue with us. Much appreciated!

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

Of course, Tom. Happy to help. There is a tremendous need for a tool like buildah. And with so many Macs in developer hands these days, getting it to work well with Docker for Mac would be a very big deal. Let me know if I can help further. I'm motivated.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

@ae6rt I think this could be seccomp related or CAPABILTIES, I think next step would be to turn off all caps, and see if it works. If yes then we can start guessing at which cap it needs.

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

@rhatdan Here is what I believe you are advising. It also fails:

$ docker run --security-opt="seccomp=unconfined" --cap-add=ALL -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
 72.07 MiB / 72.07 MiB [=======================================================]
ERRO[0130] error pulling image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/fedora:latest": Error writing blob: Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 

More information, gathered from inside the Alpine VM that is the docker host:

/ # docker info
Containers: 597
 Running: 0
 Paused: 0
 Stopped: 597
Images: 121
Server Version: 17.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.36-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: AQ6J:JHUG:2CHZ:A4RA:AGVM:JAIE:ECC4:RG6A:25YV:XJYF:TDRQ:D4ZS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 22
 Goroutines: 46
 System Time: 2017-08-18T18:02:27.865040689Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

Which brings us to seccomp. Run with --privileged, and see if that helps. If that works you can disable seccomp with
docker run -ti security-opts seccomp:unconfined ...
I think.

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

I ran with seccomp unconfined here, but I'm probably misunderstanding something.

Here is what I get when I add --privileged to the mix:

$ docker run --privileged --security-opt="seccomp=unconfined" --cap-add=ALL -v /private/tmp/containers/:/var/lib/containers/:rw,Z -ti buildah buildah from fedora
Getting image source signatures
Copying blob sha256:4db9daa7aafd1ea07f24d2ec893833adc17b5a9c5dde4150cf99a5789b3d322e
 72.07 MiB / 72.07 MiB [=======================================================]
ERRO[0154] error pulling image "containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/fedora:latest": Error writing blob: Error processing tar file(exit status 1): open /root/.bash_logout: permission denied 
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

What happens if you run alpine with out buildah and attempt to write to /root/.bash_logout?

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

Oh, I'm sorry, I don't understand. What should I type, even if pseudo-code?

@ae6rt

This comment has been minimized.

Copy link

ae6rt commented Aug 18, 2017

What I can do, and whether it proves anything seems unlikely, is that getting a shell inside a plain fedora container, then grabbing the offending fedora image layer like I did here, then running tar zxf thelayer successfully in a scratch directory succeeds. There, /root/.bash_logout is written just fine. The fedora container where I get the shell to run these tasks resides on the Alpine Docker for Mac VM.

# ls -al root
total 40
dr-xr-x---  2 root root 4096 Jul  5 21:48 .
drwxr-xr-x 15 root root 4096 Aug 18 20:26 ..
-rw-r--r--  1 root root   18 Feb 11  2017 .bash_logout
-rw-r--r--  1 root root  176 Feb 11  2017 .bash_profile
-rw-r--r--  1 root root  176 Feb 11  2017 .bashrc
-rw-r--r--  1 root root  100 Feb 11  2017 .cshrc
-rw-r--r--  1 root root  129 Feb 11  2017 .tcshrc
-rw-------  1 root root 2955 Jul  5 21:48 anaconda-ks.cfg
-rw-r--r--  1 root root  435 Jul  5 21:48 anaconda-post.log
-rw-------  1 root root 2615 Jul  5 21:48 original-ks.cfg
@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

docker run -ti buildah touch /root/.bash_logout

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Aug 18, 2017

Did you do this on the /var/lib/containers directory? What is the file system in this directory? This could be something where overlay is not allowed to write to the file system.
@rhvgoyal @nalind Any ideas?

@baude

This comment has been minimized.

Copy link
Collaborator

baude commented Feb 14, 2018

fwiw, it will run in a podman container too

@ashcrow

This comment has been minimized.

Copy link

ashcrow commented Feb 14, 2018

@ipbabble I haven't hit any issues. 👍

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Feb 15, 2018

I'm not sure if I'm late to the party but we are actually running buildah (and s/kpod/podman/) in a runc container within ansible container: https://github.com/ansible/ansible-container/pull/790/files#diff-9fa3b8fd302da35fe483d1166416a024R433

We also bind-mount /var/lib/containers/ from host to the runc container and all of this works just fine. The container itself has privileges though.

@giuseppe

This comment has been minimized.

Copy link
Member

giuseppe commented Feb 15, 2018

@TomasTomecek would something like what I am doing in projectatomic/atomic#1186 be helpful to you?

@TomasTomecek

This comment has been minimized.

Copy link
Contributor

TomasTomecek commented Feb 15, 2018

@giuseppe sounds interesting; I'm worried about adding dependency from atomic to ansible container (upstream guys actually want ansible container to be usable on mac and win); on linux it would be super-useful and helped us remove ton of code.

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Feb 15, 2018

Running buildah without requiring privs is a future goal. Nice work people. We need to get updates on Karma to get a good version of podman to demonstrate this tests though.

https://bodhi.fedoraproject.org/updates/podman-0.2-3.git3d0100b.fc27

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 15, 2018

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 15, 2018

@giuseppe

This comment has been minimized.

Copy link
Member

giuseppe commented Feb 15, 2018

@nalind is it fine to use the 'vfs' driver instead of 'overlay'?

I've done a quick try with running buildah in an user namespace, bwrap-oci handles it well, this runs as non-root user:

$ atomic pull --storage ostree docker.io/gscrivano/buildah
$ atomic run --storage ostree docker.io/gscrivano/buildah buildah bud /host/home/gscrivano/container
Extracting to /home/gscrivano/.containers/repo/tmp/atomic-container/14806/rootfs
STEP 1: FROM scratch
STEP 2: COPY foo /
STEP 3: COMMIT containers-storage:[vfs@/var/lib/containers/storage+/var/run/containers/storage:vfs.override_kernel_check=true]@14f360973b1eddabb1d41e79a3340611f05f74998c7cfaee05bef6efb8a4fafa
@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 15, 2018

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Feb 15, 2018

I think only VFS would work on a rootless environment. Since otherwise you would be required to do a mount. But @giuseppe you said you got this to work without requiring root?

@giuseppe

This comment has been minimized.

Copy link
Member

giuseppe commented Feb 15, 2018

@rhatdan using vfs yes. The steps I showed in my previous comment are done with my user uid=1000, and no root is required. The container was managed by bwrap-oci.

It is the same system container image that works for the root user, it requires this change in atomic: projectatomic/atomic#1186 so that we are able to atomic run --storage ostree IMAGE COMMANDS

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Feb 15, 2018

SO is buildah running in a usernamespace then?

@giuseppe

This comment has been minimized.

Copy link
Member

giuseppe commented Feb 15, 2018

Yes, it is

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 15, 2018

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Feb 16, 2018

This is well worth a blog and an explanation on why it is slow. Bottom line this can show we can build a container image without requiring root using Buildah. This is huge.

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 17, 2018

@ashcrow also availabel on quay.io

podman pull quay.io/ipbabble/buildah

write up is done. having reviewed and will post asap.

William

@giuseppe

This comment has been minimized.

Copy link
Member

giuseppe commented Feb 17, 2018

@ipbabble could we merge our two efforts?

My work is here:

projectatomic/atomic-system-containers#162

A system container is still able to run as a Docker container, I just needed to add some more metadata.

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 17, 2018

@giuseppe Eager to chat. I'm not sure what you mean by merge our two efforts. I did write up my use of vfs for the inside container file system build. I also wrote about why, despite being cool that it works, it's also not a very practical solution. (Due to every container potentially pulling down the same image on a single host.)

Let's connect on IRC (freenode #podman) so we can discuss why I might be missing in my effort.

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 28, 2018

I've got a PR atomic-site for a blog post.

  1. It should be noted that it covers only the privileged container mode right now.
  2. It covers some of the issues we faced with unprivileged.
  3. I pulled the unpriv section for a later blog because I saw some regression that I am looking into.
  4. @giuseppe 's work has been incorporated into a separate blog because this one was getting too long.

This blog is framed as the Buildah container for Kubernetes.
The system container Buildah example will be for a Buildah system container for Atomic
The unpriv. example will be a sequel to the first one.

I'll post the link when it goes live.
Any questions?

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Feb 28, 2018

@ipbabble

This comment has been minimized.

@ashcrow

This comment has been minimized.

Copy link

ashcrow commented Mar 1, 2018

@ipbabble great post! I did notice there's a hiccup in the markdown to youtube in the post.

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Mar 1, 2018

@ipbabble

This comment has been minimized.

Copy link
Contributor

ipbabble commented Mar 8, 2018

So I'm going to create a new/separate issue about non priv. Buildah container

@rhatdan

This comment has been minimized.

Copy link
Member

rhatdan commented Mar 12, 2018

The answer to this question is yes, so I am closing this issue. We have other issues about running with UserNamespace and in un priv containers.

@rhatdan rhatdan closed this Mar 12, 2018

nalind pushed a commit that referenced this issue Apr 2, 2018

Plumb through the --stop-timeout signal handling
podman run/create have the ability to set the stop timeout flag.
We need to stop it in the database.

Also Allowing negative time for stop timeout makes no sense, so switching
to timeout of uint, allows user to specify huge timeout values.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>

Closes: #158
Approved by: TomSweeneyRedHat
@Conan-Kudo

This comment has been minimized.

Copy link

Conan-Kudo commented Jun 11, 2018

Since this issue is getting linked around the internets, I'll leave in here a little note about how we solved this problem for us:

For our GitLab CI setup, we made sure that the /var/lib/containers in the container environment mapped to a different directory on the host system, so that the containers wouldn't be picked up by the container executor. That keeps us from accidentally breaking things in the CI system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment