container with systemd don't start #1308

rogeriojlle · 2023-09-12T01:35:10Z

Issue Description

After updating my operating system, all containers starting with /usr/bin/systemd stopped working

Steps to reproduce the issue

install systemd package inside a container, then commit
start new image with /usr/bin/systemd or /sbin/init
upgrade Fedora ( host )

Describe the results you received

podman start zapzap

Error: OCI runtime error: unable to start container "52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db": crun: chmod `run/shm`: Operation not supported

Describe the results you expected

start the container

podman info output

uname -a
Linux desktop 6.6.0-0.rc0.20230908gita48fa7efaf11.10.fc40.x86_64 containers/podman#1 SMP PREEMPT_DYNAMIC Fri Sep  8 15:57:23 UTC 2023 x86_64 GNU/Linux

[root@desktop containers]# podman info
host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-3.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 94.52
    systemPercent: 1.82
    userPercent: 3.67
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2044
  hostname: desktop
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.6.0-0.rc0.20230908gita48fa7efaf11.10.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1116749824
  memTotal: 16276656128
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-3.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: crun-1.9-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9
      commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230908.g05627dc-1.fc40.x86_64
    version: |
      pasta 0^20230908.g05627dc-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc40.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8588881920
  swapTotal: 8589930496
  uptime: 4h 36m 23.00s (Approximately 0.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 71747764224
  graphRootUsed: 25911939072
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1693251058
  BuiltTime: Mon Aug 28 16:30:58 2023
  GitCommit: ""
  GoVersion: go1.21.0
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.2

[root@desktop containers]# podman inspect zapzap
[
     {
          "Id": "52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db",
          "Created": "2023-09-02T14:40:12.282020642-03:00",
          "Path": "/usr/bin/systemd",
          "Args": [
               "/usr/bin/systemd"
          ],
          "State": {
               "OciVersion": "1.1.0-rc.3",
               "Status": "exited",
               "Running": false,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 0,
               "ExitCode": 137,
               "Error": "crun: chmod `run/shm`: Operation not supported: OCI runtime error",
               "StartedAt": "2023-09-10T16:25:01.059420166-03:00",
               "FinishedAt": "2023-09-10T16:26:03.745402871-03:00",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "253bda36c666e7be954f2e52ada8d4259412f863087b92e3965f52923bf7bf8c",
          "ImageDigest": "sha256:610813d59134492d072a030857df0c701ec722bec75aa68cbae471fc5d0948e6",
          "ImageName": "253bda36c666",
          "Rootfs": "",
          "Pod": "",
          "ResolvConfPath": "/run/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata/resolv.conf",
          "HostnamePath": "/run/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata/hostname",
          "HostsPath": "/run/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata/hosts",
          "StaticDir": "/var/lib/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata",
          "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata/conmon.pid",
          "PidFile": "/run/containers/storage/overlay-containers/52bcf643b1aeb0f90fd47003eded56560fe27a13b11ed397246225e0f34270db/userdata/pidfile",
          "Name": "zapzap",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "BoundingCaps": [
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_KILL",
               "CAP_NET_BIND_SERVICE",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYS_CHROOT"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/lib/containers/storage/overlay/784c2dc09391dc096f40dbecd347b974c11ffb7aa16bdcc842d098004ed0c5d0/diff",
                    "UpperDir": "/var/lib/containers/storage/overlay/b2167610491f268bcb037a2442d7619c57ca66bd60117ff65db03357ae575d0e/diff",
                    "WorkDir": "/var/lib/containers/storage/overlay/b2167610491f268bcb037a2442d7619c57ca66bd60117ff65db03357ae575d0e/work"
               }
          },
          "Mounts": [],
          "Dependencies": [],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {},
               "SandboxKey": "",
               "Networks": {
                    "podman": {
                         "EndpointID": "",
                         "Gateway": "",
                         "IPAddress": "",
                         "IPPrefixLen": 0,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "",
                         "NetworkID": "podman",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "52bcf643b1ae"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 1,
          "Config": {
               "Hostname": "52bcf643b1ae",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": false,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "TERM=xterm",
                    "container=podman",
                    "HOME=/root",
                    "container_uuid=52bcf643b1aeb0f90fd47003eded5656",
                    "HOSTNAME=52bcf643b1ae"
               ],
               "Cmd": [
                    "/usr/bin/systemd"
               ],
               "Image": "253bda36c666",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": "",
               "OnBuild": null,
               "Labels": null,
               "Annotations": {
                    "io.container.manager": "libpod",
                    "org.opencontainers.image.stopSignal": "37"
               },
               "StopSignal": 37,
               "HealthcheckOnFailureAction": "none",
               "CreateCommand": [
                    "podman",
                    "create",
                    "--name",
                    "zapzap",
                    "--ip=10.88.0.5",
                    "253bda36c666",
                    "/usr/bin/systemd"
               ],
               "SystemdMode": true,
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true,
               "sdNotifyMode": "container"
          },
          "HostConfig": {
               "Binds": [],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "bridge",
               "PortBindings": {},
               "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "shareable",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": false,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [],
               "Tmpfs": {},
               "UTSMode": "private",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 4194304,
                         "Hard": 4194304
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]



### Podman in a container

No

### Privileged Or Rootless

Privileged

### Upstream Latest Release

Yes

### Additional environment details

Additional environment details

### Additional information

_No response_

The text was updated successfully, but these errors were encountered:

Luap99 · 2023-09-12T08:40:59Z

What packages did you update? podman, crun something else? It would help if you can narrow it down and share commands that you use so I can juts copy and paste in order to see if we can reproduce.

giuseppe · 2023-09-12T10:23:17Z

I've tried running a container with systemd, both as root and rootless on 6.6.0-0.rc0.20230908gita48fa7efaf11.10.fc40.x86_64 with Podman 4.6.2 and crun 1.9.

How was the image created? At least on Fedora, systemd is not installed at /usr/bin/systemd

rogeriojlle · 2023-09-14T17:19:50Z

One way to reproduce the error is like this:
create a Dockerfile with the following content:
The same occurs using Debian as a base

FROM ubuntu:20.04
RUN apt-get update && apt-get install -y systemd

after:

podman run --rm -ti <image> /sbin/init

or else:

podman run --rm -ti <image> /usr/bin/systemd

the result is the same:

[rogerio@desktop debianbug]$ podman run -ti --rm 0272436f6e49 /sbin/init
Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported
[rogerio@desktop debianbug]$ podman run -ti --rm 0272436f6e49 /usr/bin/systemd
Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported

At the moment I don't have an older version of my operating system available, currently this is it:

PLATFORM_ID="platform:f40"
PRETTY_NAME="Fedora Linux 40 (Workstation Edition Prerelease)"
Linux desktop 6.6.0-0.rc1.13.fc40.x86_64

rogeriojlle · 2023-09-14T17:26:58Z

[rogerio@desktop debianbug]$ crun --version
crun version 1.9
commit: a538ac4ea1ff319bcfe2bf81cb5c6f687e2dc9d3
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

[rogerio@desktop debianbug]$ podman --version
podman version 4.6.2

rogeriojlle · 2023-09-14T18:42:31Z

in Fedora 37 works.

[rogerio@localhost-live ~]$ podman --version
podman version 4.6.2
[rogerio@localhost-live ~]$ crun --version
crun version 1.6
commit: 18cf2efbb8feb2b2f20e316520e0fd0b6c41ef4d
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +YAJL
[rogerio@localhost-live ~]$ uname -a
Linux localhost-live 6.0.7-301.fc37.x86_64 containers/podman#1 SMP PREEMPT_DYNAMIC Fri Nov 4 18:35:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[rogerio@localhost-live ~]$

Luap99 · 2023-09-21T08:32:14Z

@giuseppe Looks like a kernel regression maybe?

I see the same symptom on the CI image update PR in different tests: https://api.cirrus-ci.com/v1/artifact/task/6621245794418688/html/int-podman-rawhide-root-host-sqlite.log.html#t--Podman-run-with-volumes-podman-run-with-mount-flag--1

Error: OCI runtime error: crun: chmod `misc/tsget`: Operation not supported

Given the same works on f38 with the same crun the only logically thing is a kernel change that broke this:
the rawhide VM uses 6.6.0-0.rc2.20230919git2cf0f7156238.21.fc40.x86_64
f38 is on 6.5.4-200.fc38.x86_64

Could it be that crun is trying to change the permission of a symlink directly? IIRC the kernel changed the behaviour to block that.

commit 5d1f903f75a80daa4dfb3d84e114ec8ecbf29956 in the kernel, present in a release since Linux 6.6 doesn't allow anymore to change the mode of a symlink, so just ignore the failure. Closes: containers#1308 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe · 2023-09-22T09:37:28Z

yes that is related to a change in the kernel, opened a PR: #1309

commit 5d1f903f75a80daa4dfb3d84e114ec8ecbf29956 in the kernel, present in a release since Linux 6.6 doesn't allow anymore to change the mode of a symlink, so just ignore the failure. Closes: containers#1308 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

bviktor · 2023-11-15T19:45:56Z

Alright, so after all I figured out how to resolve this with Ubuntu containers. Here's my guide for the interested:

https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/

This is to workaround issues in the package shipped by the OS[1]. [1] containers/crun#1308 Closes-Bug: #2056210 Change-Id: I16f83d7e9cc127ce6997a85097d1517ce54fbefc

containers/crun#1308

Containers get stuck in the creating state. This is a known issue[1,2] and several workarounds are suggested in the ansible-collection-kolla bug report[3]. [1] containers/crun#1308 [2] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2052961 [3] https://bugs.launchpad.net/ansible-collection-kolla/+bug/2056210 Change-Id: I16f83d7e9cc127ce6997a85097d1517ce54fbefc

Luap99 mentioned this issue Sep 21, 2023

Update CI VM Images containers/podman#19751

Closed

giuseppe transferred this issue from containers/podman Sep 22, 2023

giuseppe mentioned this issue Sep 22, 2023

utils: ignore ENOTSUP when chmod a symlink #1309

Merged

giuseppe closed this as completed in #1309 Sep 25, 2023

aki-k mentioned this issue Oct 3, 2023

systemd in a docker container with cgroups v2 LewisGaul/systemd-containers#1

Closed

n-riesco mentioned this issue Nov 8, 2023

NixOS 23.05: crun: kernel 6.1.59 breaks compatibility with crun < 1.9.2 NixOS/nixpkgs#266302

Open

jwillikers mentioned this issue Nov 12, 2023

Container fails to start after attempting to chmod a symlink when run via Kubernetes YAML containers/podman#20666

Closed

crobert-1 mentioned this issue Feb 27, 2024

[CI/CD] build-package (deb) is broken open-telemetry/opentelemetry-collector-contrib#31443

Closed

mafalb mentioned this issue Mar 2, 2024

Incompatible kernel breaks crun on github runner mafalb/workflows#55

Closed

gselig1a mentioned this issue Mar 4, 2024

Logs not available when Kind Cluster creation fails containers/podman-desktop#6260

Open

This was referenced Mar 8, 2024

Prefer systemd container rather than sudoer kachick/dotfiles#451

Merged

2024-03-09 - systemd を走らせている間に操作して commit を加えた container image を作る kachick/times_kachick#277

Closed

znerol added a commit to znerol/molecule-docker that referenced this issue Mar 30, 2024

chore(ci): Patch crun

8cdaa6f

containers/crun#1308

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

container with systemd don't start #1308

container with systemd don't start #1308

rogeriojlle commented Sep 12, 2023

Luap99 commented Sep 12, 2023

giuseppe commented Sep 12, 2023

rogeriojlle commented Sep 14, 2023

rogeriojlle commented Sep 14, 2023

rogeriojlle commented Sep 14, 2023

Luap99 commented Sep 21, 2023

giuseppe commented Sep 22, 2023

bviktor commented Nov 15, 2023

container with systemd don't start #1308

container with systemd don't start #1308

Comments

rogeriojlle commented Sep 12, 2023

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Luap99 commented Sep 12, 2023

giuseppe commented Sep 12, 2023

rogeriojlle commented Sep 14, 2023

rogeriojlle commented Sep 14, 2023

rogeriojlle commented Sep 14, 2023

Luap99 commented Sep 21, 2023

giuseppe commented Sep 22, 2023

bviktor commented Nov 15, 2023