Build container with unprivileged mode in Kubernetes #2554

alexandrsemak · 2020-08-20T01:46:21Z

Description

I am trying use buildah for replace docker in jenkins kubernetes plugin which run agent as kubernetes pod for building container.

I can't run buildah bud with unprivileged mode

buildah --storage-driver vfs \
              bud \
              --format docker \
              -t ${tagName} \
             -f services/${serviceJson.folderName}/Dockerfile services/${serviceJson.folderName}

Error:

level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
Getting image source signatures
Copying blob sha256:6c29051c46d66e19d39b578f69f424269edad28a39dee60b4ff06571e8bbc5c8
Copying blob sha256:adc60475692c9734e4de7e676c289f696c976f39013e5be1c73fa8016f8eb83b
Copying blob sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f
Copying blob sha256:d978813e923f720f72c705dc6765d8cd8ee5714a0ccec7a88c734b4d3d9265b9
Copying blob sha256:b60f122e0c8ca3da6a03fd1bb5873d1a14502a192cfb1b6354ff7280bf3aa245
Copying blob sha256:49581cb51103142941c6dcf88ae0c36c9e30fe63d05512b3be98ac7d8b04c7ef
Copying config sha256:ea39da9a194b483cf359c97d06552f7132fbcf8ced30089b64e5801e5b6aae3a
Writing manifest to image destination
Storing signatures
level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
error creating build container: Error committing the finished image: error adding layer with blob "sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f": ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied
level=error msg="exit status 125"

Steps to reproduce the issue:

Create base image for Jenkins Kubernetes Plugin base on debian_10

ARG JNLP_VERSION=4.3-9
FROM jenkins/jnlp-slave:$JNLP_VERSION

USER root

# Install buildah
RUN echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list \
    && wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/Debian_10/Release.key -O Release.key \
    && apt-key add - < Release.key \
    && apt-get update \
    && apt-get -qq -y install buildah podman \
    && rm -rf /var/lib/apt/lists/*

Spinup jnlp-agent in kubernetes

apiVersion: "v1"
kind: "Pod"
metadata:
  labels:
    jenkins: "slave"
    jenkins/label: "wxu-web"
  name: "wxu-web"
spec:
  containers:
    image: "****/debian_10_jnlp:buildah"
    imagePullPolicy: "Always"
    name: "jnlp"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  hostNetwork: false

Use buildah bud for create container

buildah --storage-driver vfs \
              bud \
              --format docker \
              -t ${tagName} \
             -f services/${serviceJson.folderName}/Dockerfile services/${serviceJson.folderName}

Output of rpm -q buildah or apt list buildah:

(paste your output here)

Output of buildah version:

Version:         1.15.1
Go Version:      go1.14
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.5.1
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

Output of cat /etc/*release:

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian

Output of uname -a:

Linux wxu-web-1.0.2-w3wtk 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[storage]

# Default Storage Driver
driver = ""

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot="/root/.local/share/containers/storage"

# Storage path for rootless users
#
# rootless_storage_path = "$HOME/.local/share/containers/storage"

[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"

additionalimagestores = [
"/var/lib/shared",
]

[storage.options.overlay]

mount_program = "/usr/bin/fuse-overlayfs"

mountopt = "nodev"

[storage.options.thinpool]

[containers]

[network]

[engine]

[engine.runtimes]

The text was updated successfully, but these errors were encountered:

rhatdan · 2020-08-20T12:55:21Z

For now you need to run buildah with a privileged container or at least one running as root. If this is running on a Docker back end, then there could be issues with seccomp preventing the mounts.

You probably should use the buildah image or one based on it. quay.io/buildah/stable

The main points are to run with --isolation=chroot, since you are already inside of a container.

alexandrsemak · 2020-08-20T13:31:48Z

Thank you for answer. I will look in buildah image, as I understood quay.io/buildah/stable has root-less build user?

rhatdan · 2020-08-20T13:45:03Z

Yes, that will only work though, if you pass the /dev/fuse into the container.

alexandrsemak · 2020-08-20T17:15:14Z

@rhatdan so --storage-driver=vfs required privileged mode ? when /dev/fuse and fuse-overlayfs no ?

rhatdan · 2020-08-20T17:51:35Z

storage-driver=vfs works without privilege, but has its owne issues. fuse-overlayfs requires /dev/fuse in order to be able to use overlay within a non privileged container (A container without CAP_SYS_ADMIN)

alexandrsemak · 2020-08-20T18:56:33Z

what the issue with storage-driver=vfs ?

rhatdan · 2020-08-20T19:11:56Z

Each layer copies the entire contents of the sublayer. Can take a huge amount of space and be slow.

alexandrsemak · 2020-08-24T10:57:31Z

Hello @rhatdan I switched to quay.io/buildah/stable:v1.15.0 and run buildah bud with --storage-driver vfs and as userId: 1000(build) but still have Error while applying layer: ApplyLayer exit status 1 stdout: stderr: remount /, flags: 0x44000: permission denied

level=debug msg="Start untar layer"
level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
level=info msg="Warning: pull failed, retrying in 2s ... (3/3)"
level=debug msg="Trying to access \"wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0\""
level=debug msg="Returning credentials from /var/tmp/1000/containers/containers/auth.json"
level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
level=debug msg=" Using \"default-docker\" configuration"
level=debug msg=" No signature storage configuration found for wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0"
level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/"
level=debug msg="Ping https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/ status 401"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com:443/artifactory/api/docker/wcp-twc-web-decoupling-docker-local/v2/token?account=webops%40us.ibm.com&scope=repository%3Awxu-web%3Apull&service=wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com%3A443"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/manifests/node-13.11.0-1.0.0"
level=debug msg="Content-Type from manifest GET is \"application/vnd.docker.distribution.manifest.v2+json\""
level=debug msg="Using blob info cache at /home/build/.local/share/containers/storage/cache/blob-info-cache-v1.boltdb"
level=debug msg="IsRunningImageAllowed for image docker:wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0"
level=debug msg=" Using default policy section"
level=debug msg=" Requirement 0: allowed"
level=debug msg="Overall: allowed"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:ea39da9a194b483cf359c97d06552f7132fbcf8ced30089b64e5801e5b6aae3a"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:ea39da9a194b483cf359c97d06552f7132fbcf8ced30089b64e5801e5b6aae3a"
Getting image source signatures
level=debug msg="Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]"
level=debug msg="... will first try using the original manifest unmodified"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:b60f122e0c8ca3da6a03fd1bb5873d1a14502a192cfb1b6354ff7280bf3aa245"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:b60f122e0c8ca3da6a03fd1bb5873d1a14502a192cfb1b6354ff7280bf3aa245"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:d978813e923f720f72c705dc6765d8cd8ee5714a0ccec7a88c734b4d3d9265b9"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:d978813e923f720f72c705dc6765d8cd8ee5714a0ccec7a88c734b4d3d9265b9"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:6c29051c46d66e19d39b578f69f424269edad28a39dee60b4ff06571e8bbc5c8"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:6c29051c46d66e19d39b578f69f424269edad28a39dee60b4ff06571e8bbc5c8"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:adc60475692c9734e4de7e676c289f696c976f39013e5be1c73fa8016f8eb83b"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:adc60475692c9734e4de7e676c289f696c976f39013e5be1c73fa8016f8eb83b"
level=debug msg="Downloading /v2/wxu-web/blobs/sha256:49581cb51103142941c6dcf88ae0c36c9e30fe63d05512b3be98ac7d8b04c7ef"
level=debug msg="GET https://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/v2/wxu-web/blobs/sha256:49581cb51103142941c6dcf88ae0c36c9e30fe63d05512b3be98ac7d8b04c7ef"
Copying blob sha256:adc60475692c9734e4de7e676c289f696c976f39013e5be1c73fa8016f8eb83b
level=debug msg="Detected compression format gzip"
level=debug msg="Using original blob without modification"
Copying blob sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f
level=debug msg="Detected compression format gzip"
level=debug msg="Using original blob without modification"
Copying blob sha256:49581cb51103142941c6dcf88ae0c36c9e30fe63d05512b3be98ac7d8b04c7ef
level=debug msg="Detected compression format gzip"
level=debug msg="Using original blob without modification"
Copying blob sha256:b60f122e0c8ca3da6a03fd1bb5873d1a14502a192cfb1b6354ff7280bf3aa245
level=debug msg="Detected compression format gzip"
level=debug msg="Using original blob without modification"
Copying blob sha256:d978813e923f720f72c705dc6765d8cd8ee5714a0ccec7a88c734b4d3d9265b9
level=debug msg="Detected compression format gzip"
level=debug msg="Using original blob without modification"
Copying blob sha256:6c29051c46d66e19d39b578f69f424269edad28a39dee60b4ff06571e8bbc5c8
level=debug msg="Detected compression format gzip"
level=debug msg="Using original blob without modification"
Copying config sha256:ea39da9a194b483cf359c97d06552f7132fbcf8ced30089b64e5801e5b6aae3a
level=debug msg="No compression detected"
level=debug msg="Using original blob without modification"
Writing manifest to image destination
Storing signatures
level=debug msg="Start untar layer"
level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
level=debug msg="error copying src image [\"docker://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0\"] to dest image [\"wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0\"] err: Error committing the finished image: error adding layer with blob \"sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f\": ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
level=debug msg="error pulling image \"docker://wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0\": Error committing the finished image: error adding layer with blob \"sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f\": ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
level=debug msg="unable to pull and read image \"wcp-twc-web-decoupling-docker-local.artifactory.swg-devops.com/wxu-web:node-13.11.0-1.0.0\": Error committing the finished image: error adding layer with blob \"sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f\": ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied"
error creating build container: Error committing the finished image: error adding layer with blob "sha256:e6b0cf9c0882fb079c9d35361d12ff4691f916b6d825061247d1bd0b26d7cf3f": ApplyLayer exit status 1 stdout:  stderr: remount /, flags: 0x44000: permission denied
level=error msg="exit status 125"

deftdawg · 2020-11-03T06:40:36Z

@alexandrsemak did you ever find a solution to this? I'm getting ApplyLayer exit status 1 stdout: stderr: remount /, flags: 0x44000: invalid argument which seems related...

Also how did you turn on debugging?

alexandrsemak · 2020-11-10T18:01:32Z

@deftdawg nope, We just setup pod in privileged mode where execute inside buildah bud

PirateBread · 2020-12-08T17:49:54Z

I am testing using Buildah today on a kubernetes container in unprivileged mode. Interested to know if anyone has any tips for this.
I have tried various things suggested but unable to build a simple Dockerfile. I think privileged mode would not be possible as this is a secure corporate environment with lots of workloads on the kubernetes clusters.

rhatdan · 2020-12-08T20:44:14Z

It is impossible without at least CAP_SETUID and CAP_SETGID.

Buildah needs multiple UIDs in order to run. If it pulls an image with more then one UID, then it needs to chown files to this other UID. Most files are probably owned by root, so it needs to be able to write the root files. We can run it within a User Namespace to give us most of the functionality but these containers need to run with CAP_SETUID and CAP_SETGID to setup the user namespace.

We are working in the CRI-O world to define how a container/pod can be launched in a user namespace, where Buildah would just work naturally, but this is not possible yet.

Bottom line, in a fully locked down, UID !=root system, you can not build images. You can not run with multiple UIDs.

rhatdan · 2020-12-08T20:44:52Z

@giuseppe @mrunalp Do we have support yet in CRI-O to specify a pod within a user namespace?

giuseppe · 2020-12-09T16:49:23Z

@giuseppe @mrunalp Do we have support yet in CRI-O to specify a pod within a user namespace?

yes, we have support for it upstream now to create a pod user namespace through an annotation

rhatdan · 2021-02-10T21:32:26Z

@umohnani8 Another example of building containers within a container.

gregoryboue · 2021-03-03T22:30:02Z

Hi @rhatdan is there any feature planned for this ?

rhatdan · 2021-03-03T22:41:23Z

Not sure what you mean by feature, we plan on writing some articles of different ways you an run buildah and podman inside of containers.
We will be convering run a build without privs inside of kubernetes but running as root, and we will cover running a build within a user namespace launched by kubernetes.

rhatdan · 2021-03-03T22:41:54Z

Bottom line Buildah is going to need either a user namespace or to run as root with a few linux capabilities.

gregoryboue · 2021-03-04T08:43:59Z

@rhatdan Thanks, I thought that it was not possible to have a kubernetes pod with unprivileged buildah container currently ?

But from your answer i understand that it's possible, we just need an how to ?

gregoryboue · 2021-03-05T10:55:13Z

@rhatdan is it possible in few lines to explain us how to do it ?

rhatdan · 2021-03-05T16:20:25Z

@umohnani8 Do you have time to play with this next week?

bodhi-one · 2021-03-06T18:44:03Z

My team is very interested in a solution to this (running as a non-root user). Have tried many settings and options for buildah. Best we can do is the need to sudo chmod 777 /dev/fuse once inside the container and all is well. I'll create a new support case/request/bug for what we are seeing.

gregoryboue · 2021-03-11T13:18:06Z

@rhatdan @umohnani8 To explain why this issue is very important for us I just want to share my professional context.
In my company, we provide CI/CD environments for more than thousand project.

Our 'build agents' are ephemeral kubernetes pod instanciated just for CI/CD pipelines.

For more than 3 years (first version was 0.12) we choose buildah as image builder solution to avoid DinD. It's working but it forced us to make secure the CI/CD context because of privileged pod needed.
This constraint is very painful for us and it have impact in a lot of features that we want to do for projects

gregoryboue · 2021-03-23T22:12:33Z

@rhatdan any news ?

rhatdan · 2021-03-24T20:39:27Z

Nope nothing yet.

gregoryboue · 2021-04-01T14:41:24Z

@rhatdan ok thanks I will be waiting :-)

rhatdan added the kind/feature label Oct 7, 2020

twolffpiggott mentioned this issue Jan 11, 2021

mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied #2899

Closed

rhatdan added the buildah-in-container label Feb 10, 2021

chmouel mentioned this issue Apr 2, 2021

buildah: add a new 0.3 version removing privileged tektoncd/catalog#593

Open

9 of 9 tasks complete

containers / buildah

Build container with unprivileged mode in Kubernetes #2554

Build container with unprivileged mode in Kubernetes #2554

alexandrsemak commented Aug 20, 2020 •

edited

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 20, 2020

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 20, 2020

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 20, 2020

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 24, 2020

deftdawg commented Nov 3, 2020 •

edited

alexandrsemak commented Nov 10, 2020

PirateBread commented Dec 8, 2020

rhatdan commented Dec 8, 2020

rhatdan commented Dec 8, 2020

giuseppe commented Dec 9, 2020

rhatdan commented Feb 10, 2021

gregoryboue commented Mar 3, 2021

rhatdan commented Mar 3, 2021

rhatdan commented Mar 3, 2021

gregoryboue commented Mar 4, 2021

gregoryboue commented Mar 5, 2021

rhatdan commented Mar 5, 2021

bodhi-one commented Mar 6, 2021

gregoryboue commented Mar 11, 2021

gregoryboue commented Mar 23, 2021

rhatdan commented Mar 24, 2021

gregoryboue commented Apr 1, 2021

containers / buildah

Build container with unprivileged mode in Kubernetes #2554

Build container with unprivileged mode in Kubernetes #2554

Comments

alexandrsemak commented Aug 20, 2020 • edited

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 20, 2020

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 20, 2020

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 20, 2020

rhatdan commented Aug 20, 2020

alexandrsemak commented Aug 24, 2020

deftdawg commented Nov 3, 2020 • edited

alexandrsemak commented Nov 10, 2020

PirateBread commented Dec 8, 2020

rhatdan commented Dec 8, 2020

rhatdan commented Dec 8, 2020

giuseppe commented Dec 9, 2020

rhatdan commented Feb 10, 2021

gregoryboue commented Mar 3, 2021

rhatdan commented Mar 3, 2021

rhatdan commented Mar 3, 2021

gregoryboue commented Mar 4, 2021

gregoryboue commented Mar 5, 2021

rhatdan commented Mar 5, 2021

bodhi-one commented Mar 6, 2021

gregoryboue commented Mar 11, 2021

gregoryboue commented Mar 23, 2021

rhatdan commented Mar 24, 2021

gregoryboue commented Apr 1, 2021

alexandrsemak commented Aug 20, 2020 •

edited

deftdawg commented Nov 3, 2020 •

edited