Error starting toolbox with runc and Cgroups v2

[garrett]

I cannot run any toolbox containers in Fedora 31 (alpha) Silveblue.

I'm not sure if this is a problem with toolbox, podman, a mismatch between the toolbox & podman, a system configuration issue in Fedora, or something Silverblue specific.

Versions

• Fedora Silverblue 31.20190902.n.0 (2019-09-02T08:29:58Z)
• toolbox-0.0.12-2.fc31.noarch
• podman-1.5.1-2.16.dev.gitce64c14.fc31.x86_64

Existing container

$ toolbox -v enter --container fedora-toolbox-30

toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user garrett
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.5.1-dev is unchanged
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-30
toolbox: checking if container fedora-toolbox-30 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-30
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-30
Error: unable to start container "fedora-toolbox-30": error reading container (probably exited) json message: EOF
toolbox: failed to start container fedora-toolbox-30

New container

$ toolbox create -c test2 -r 31

Created container: test2
Enter with: toolbox enter --container test2

$ toolbox -v enter --container test2

toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user garrett
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.5.1-dev is unchanged
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is test2
toolbox: checking if container test2 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container test2
toolbox: /etc/profile.d/toolbox.sh already mounted in container test2
Error: unable to start container "test2": time="2019-09-03T10:47:15+02:00" level=warning msg="signal: killed"
time="2019-09-03T10:47:15+02:00" level=warning msg="no such directory for freezer.state"
time="2019-09-03T10:47:15+02:00" level=warning msg="no such directory for freezer.state"
time="2019-09-03T10:47:15+02:00" level=error msg="container_linux.go:346: starting container process caused \"process_linux.go:297: applying cgroup configuration for process caused \\\"mountpoint for cgroup not found\\\"\"\n"
container_linux.go:346: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"mountpoint for cgroup not found\"": OCI runtime error
toolbox: failed to start container test2

[HarryMichal]

I would say that the problem is caused by Podman. I'm not sure exactly what it is but I'll try to look at it.

[debarshiray]

This is likely due to the introduction of Cgroups v2 in Fedora 31.

[debarshiray]

@garrett was this a clean new Fedora 31 Silverblue install? Or did you upgrade?

I am asking because it might be due to configuration file issues in /etc.

[debarshiray]

Versions

* Fedora Silverblue 31.20190902.n.0 (2019-09-02T08:29:58Z)

* toolbox-0.0.12-2.fc31.noarch

* podman-1.5.1-2.16.dev.gitce64c14.fc31.x86_64

podman-1.5.1-2.16.dev.gitce64c14.fc31 was still using runc as the OCI container runtime, and it doesn't work with Cgroups v2 which is what Fedora 31 uses.

podman-1.5.1-2.17.dev.gitce64c14.fc31 is the build that switched to a different runtime, crun. Using that build would at least get you a bit further.

Let's close this. Please do open a new issue if the new build leads to another failure.

[garrett]

Thanks for looking into this!

I have podman-1.5.1-2.17.dev.gitce64c14.fc31 on my Silverblue 31 testing machine now and it still doesn't work. podman info shows that it's using runc, not crun. Meanwhile, for podman info as root, I do see crun. As a result of this, I thought it could be a local config file.

I looked at the auto-generated ~/.config/containers/libpod.conf and it's configured to use runc. Moving the file out of the way (for it to be recreated) and running podman info, I see it is now using crun instead:

host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: 118fcdfca36d706f766bad2663b11bd2c41bf2e7'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 5966553088
  MemTotal: 16599789568
  OCIRuntime:
    package: crun-0.8-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun 0.8
      spec: 1.0.0
      +SYSTEMD +SELINUX +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 8363438080
  SwapTotal: 8363438080
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: foo.lan
  kernel: 5.3.0-0.rc6.git0.1.fc31.x86_64
  os: linux
  rootless: true
  uptime: 17h 29m 44.09s (Approximately 0.71 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/garrett/.config/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/garrett/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /run/user/1000
  VolumePath: /var/home/garrett/.local/share/containers/storage/volumes

However, toolbox still fails to enter a container:

$ toolbox -v enter

toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user garrett
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.5.1-dev is old
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-31
toolbox: checking if container fedora-toolbox-31 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-31
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31
Error: unable to start container "fedora-toolbox-31": error reading container (probably exited) json message: EOF
toolbox: failed to start container fedora-toolbox-31

[debarshiray]

Ok, at least it's a different error now:

Error: unable to start container "fedora-toolbox-31": error reading container (probably exited) json message: EOF

Did you re-create the container from scratch after resolving the runc versus crun issue?

What does this say:

$ podman --log-level debug start fedora-toolbox-31

(We should probably open a new issue for this.)

[returntrip]

Filed issue with libpod: containers/podman#4024

[garrett]

@debarshiray: I did! Yeah, it probably should be another error. I'll try again tomorrow with the latest Silverblue Fedora 31 beta.

Thanks for your effort on this! (I know it has been more of a podman issue, so extra thanks!)

[debarshiray]

Thanks for your effort on this!

My pleasure. :)