podman-in-docker: building image fails on cgroupv2-enabled systems #12559

stac47 · 2021-12-09T17:51:22Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Building image with podman from a docker container fails on recent OS (ubuntu 21.10) but works on old systems (Red Hat Enterprise Linux Server release 7.9 (Maipo)). It is probably linked to the fact cgroupv2 is enabled on not.

Steps to reproduce the issue:

Using the podman image described in the article https://www.redhat.com/sysadmin/podman-inside-container, build a basic image as described in the following oneliner:

% docker run -i --privileged quay.io/podman/stable podman build - <<EOF
FROM fedora
RUN echo hello
EOF
STEP 1/2: FROM fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying config sha256:3059bef432ebb91a6a51d8f5cf20b033041dbddb3cab79628c1eb3412cbde0ae
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN echo hello
error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo hello]: writing file `/sys/fs/cgroup/buildah-buildah957585528/cgroup.procs`: Operation not supported
: exit status 1
Error: error building at STEP "RUN echo hello": error while running runtime: exit status 1
zsh: exit 125   docker run -i --privileged quay.io/podman/stable podman build - <<<''

Describe the results you received:

The image build fails when on the RUN directive:

error running container: error from /usr/bin/crun creating container for [/bin/sh -c echo hello]: writing file `/sys/fs/cgroup/buildah-buildah957585528/cgroup.procs`: Operation not supported

Describe the results you expected:

Running the same on Red Hat Enterprise Linux Server release 7.9 (Maipo)

% docker run -i --privileged quay.io/podman/stable podman build - <<EOF
heredoc> FROM fedora
heredoc> RUN echo hello
heredoc> EOF
Unable to find image 'quay.io/podman/stable:latest' locally
latest: Pulling from podman/stable
4545346f2a49: Already exists
b59d2d8f6c51: Pull complete
6f9c9697d233: Pull complete
64c213ae2a2d: Pull complete
efc31865fe95: Pull complete
f59ce0390c36: Pull complete
6176dbbe111e: Pull complete
6f4545bc34ef: Pull complete
Digest: sha256:c9758557a650a5b71f9aa440c3ba853bfb248f63cb583d60f9014ae408fd5d6f
Status: Downloaded newer image for quay.io/podman/stable:latest
STEP 1/2: FROM fedora
Resolved "fedora" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.fedoraproject.org/fedora:latest...
Getting image source signatures
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying blob sha256:4545346f2a492b62d5a82682efe19b0e8e7583d5c19f75a74c81d62ec536c32d
Copying config sha256:3059bef432ebb91a6a51d8f5cf20b033041dbddb3cab79628c1eb3412cbde0ae
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN echo hello
hello
COMMIT
--> c326f21e77a
c326f21e77a1e2ba6bcdf88fce26875a14d92a42bce40f3b432d0987febe5482

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

% docker run -i --privileged quay.io/podman/stable podman version
Version:      3.4.2
API Version:  3.4.2
Go Version:   go1.16.8
Built:        Fri Nov 12 20:25:37 2021
OS/Arch:      linux/amd64

I also tried with podman 3.4.3 coming when we run dnf update --refresh --enablerepo=updates-testing podman in fedora:latest.

Output of podman info --debug:

% docker run -i --privileged quay.io/podman/stable podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "35"
  eventLogger: file
  hostname: aa3ea5642c31
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.13.0-21-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1977335808
  memTotal: 16779907072
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 577h 16m 44.04s (Approximately 24.04 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636748737
  BuiltTime: Fri Nov 12 20:25:37 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

For comparison with the system on which it works:

% docker run -i --privileged quay.io/podman/stable podman info --debug
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - memory
  - devices
  - freezer
  - net_cls
  - blkio
  - perf_event
  - hugetlb
  - pids
  - net_prio
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 8
  distribution:
    distribution: fedora
    variant: container
    version: "35"
  eventLogger: file
  hostname: bc9e523cb51b
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 3.10.0-1160.45.1.el7.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1610510336
  memTotal: 16806350848
  ociRuntime:
    name: crun
    package: crun-1.3-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.3
      commit: 8e5757a4e68590326dafe8a8b1b4a584b10a1370
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 705789952
  swapTotal: 1073737728
  uptime: 441h 47m 29.01s (Approximately 18.38 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 1636748737
  BuiltTime: Fri Nov 12 20:25:37 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.2

Package info (e.g. output of rpm -q podman or apt list podman):

% docker run -i --privileged quay.io/podman/stable rpm -q podman
podman-3.4.2-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

vrothberg · 2021-12-13T12:15:15Z

@giuseppe PTAL

giuseppe · 2021-12-13T13:16:07Z

I think it might happen because the Podman process is at the cgroup root inside the parent container, so the OCI fails to create a new cgroup and move a process there. I am not sure if it is something Podman needs to deal with though.

To confirm what I said above, could you please try running these commands first in the Docker container before running Podman?

# mkdir /sys/fs/cgroup/init
# cat /sys/fs/cgroup/cgroup.procs | tee /sys/fs/cgroup/init/cgroup.procs

vrothberg · 2021-12-15T12:34:11Z

@giuseppe I can confirm that this is resolving the issue.

stac47 · 2021-12-16T08:30:28Z

Unfortunately, I cannot confirm it works. I am stuck here:

% docker run -it --privileged quay.io/podman/stable bash
[root@59dcb74c5a73 /]#  mkdir /sys/fs/cgroup/init
[root@59dcb74c5a73 /]# cat /sys/fs/cgroup/cgroup.procs
1
22
[root@59dcb74c5a73 /]# cat /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs
cat: write error: Invalid argument
[root@59dcb74c5a73 /]#  cat /sys/fs/cgroup/cgroup.procs | tee /sys/fs/cgroup/init/cgroup.procs
1
24
25
tee: /sys/fs/cgroup/init/cgroup.procs: Invalid argument

I don't know why I cannot write /sys/fs/cgroup/init/cgroup.procs

[root@59dcb74c5a73 /]# mount | grep cgroup
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

Tell me if I missed something.

github-actions · 2022-01-16T00:05:03Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2022-01-18T10:32:29Z

@giuseppe @stac47 @vrothberg What should we do with this issue, does not look like something Podman can fix. But @stac47 question was never answered. Since this seems like more of a discussion at this point I am transferring.

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 9, 2021

vrothberg assigned giuseppe Dec 15, 2021

github-actions bot added the stale-issue label Jan 16, 2022

containers locked and limited conversation to collaborators Jan 18, 2022

rhatdan converted this issue into discussion #12898 Jan 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

podman-in-docker: building image fails on cgroupv2-enabled systems #12559

podman-in-docker: building image fails on cgroupv2-enabled systems #12559

stac47 commented Dec 9, 2021 •

edited

vrothberg commented Dec 13, 2021

giuseppe commented Dec 13, 2021 •

edited

vrothberg commented Dec 15, 2021

stac47 commented Dec 16, 2021

github-actions bot commented Jan 16, 2022

rhatdan commented Jan 18, 2022

This issue was moved to a discussion.

This issue was moved to a discussion.

podman-in-docker: building image fails on cgroupv2-enabled systems #12559

podman-in-docker: building image fails on cgroupv2-enabled systems #12559

Comments

stac47 commented Dec 9, 2021 • edited

vrothberg commented Dec 13, 2021

giuseppe commented Dec 13, 2021 • edited

vrothberg commented Dec 15, 2021

stac47 commented Dec 16, 2021

github-actions bot commented Jan 16, 2022

rhatdan commented Jan 18, 2022

This issue was moved to a discussion.

stac47 commented Dec 9, 2021 •

edited

giuseppe commented Dec 13, 2021 •

edited