Cannot start rootless containers via docker-compose

[Werkov]

Issue Description

Since commit systemd/systemd@ce7de0b systemd user instance runs with reduced oom_score_adj. This is a PITA for podman and rootless containers because they may fail to start because they cannot set oom_score_adj = 0 (lower than the default 100) and they treat it fatally.

Steps to reproduce the issue

Steps to reproduce the issue

1. Have a 'docker-compose.yml`
2. docker-compose up -d
3. watch

Creating standalone_postgres_1 ... error

ERROR: for standalone_postgres_1  Cannot start service postgres: runc: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

time="2023-09-01T13:52:07+02:00" level=fatal msg="nsexec[29507]: failed to update /proc/self/oom_score_adj: Permission denied"
time="2023-09-01T13:52:07+02:00" level=error msg="runc create failed: unable to start container process: can't get final child's PID from pipe: EOF": OCI permission denied

ERROR: for postgres  Cannot start service postgres: runc: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

time="2023-09-01T13:52:07+02:00" level=fatal msg="nsexec[29507]: failed to update /proc/self/oom_score_adj: Permission denied"
time="2023-09-01T13:52:07+02:00" level=error msg="runc create failed: unable to start container process: can't get final child's PID from pipe: EOF": OCI permission denied
ERROR: Encountered errors while bringing up the project.


ERROR: for standalone_postgres_1  Cannot start service postgres: runc: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: OCI runtime error

ERROR: for postgres  Cannot start service postgres: runc: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: OCI runtime error
ERROR: Encountered errors while bringing up the project.

Describe the results you received

First service of docker-compose definition fails to start.

Describe the results you expected

All service container start.

podman info output

host:
  arch: amd64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: unknown'
  cpuUtilization:
    idlePercent: 96.81
    systemPercent: 1.91
    userPercent: 1.28
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20230828"
  eventLogger: journald
  freeLocks: 1960
  hostname: blackpad
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.4.11-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 322260992
  memTotal: 15442173952
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: |-
      cni-1.1.2-2.5.x86_64
      cni-plugins-1.1.1-2.5.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: runc-1.1.8-1.2.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.8
      commit: v1.1.8-0-g82f18fe0e44a
      spec: 1.0.2-dev
      go: go1.21.0
      libseccomp: 2.5.4
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.1.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.4
  swapFree: 4303159296
  swapTotal: 4304732160
  uptime: 3h 57m 4.00s (Approximately 0.12 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/mkoutny/.config/containers/storage.conf
  containerStore:
    number: 79
    paused: 0
    running: 0
    stopped: 79
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/mkoutny/.local/share/containers/storage
  graphRootAllocated: 505639653376
  graphRootUsed: 206388285440
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 36
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/mkoutny/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1692144000
  BuiltTime: Wed Aug 16 02:00:00 2023
  GitCommit: ""
  GoVersion: go1.21.0
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

> env|grep DOCK
DOCKER_BUILDKIT=0
DOCKER_HOST=unix:///run/user/1000/podman/podman.sock

Additional information

I suspect the fix is not effective because docker-compose explicitly synthesizes explicit "OomScoreAdj": 0.

I find it better to treat failure to set oom_score_adj as a soft error (report a warning, don't fail the container start, it's only a hint, it won't ensure QoS afterall).

[mheon]

I take it from the reproducer that Compose sets oom-sore-adj by default on all containers?

[mheon]

Ah, you note that it does in additional information.

I'm reluctant to make failures on oom_score_adj nonfatal by default, but given this breaks compose, we may not have many options. @vrothberg Do you have an opinion here?

[rhatdan]

This would only effect rootless mode in compatibility correct. We could assume that if oom-score-adj=0 and compatibility mode and rootless mode, then warn.

[vrothberg]

It will impact any rootless container, not only the ones created via the compat API.

I somehow agree with Lennart's take in systemd/systemd#29032 (comment):

why does runc even insist on resetting the oom adjust value to zero? that's pretty broken: allowing user container payloads to mark themselves as more relevant as the rest of the user code? weird. conceptually backwards if you ask me.

I don't think it's worth arguing much. Podman will be more portable if we make it smarter in dealing with oom_score_adj when running rootless.

@giuseppe WDYT?

[giuseppe]

would we break anything if we just do not set oom-score-adj unless the user doesn't override it? It is such a low level setting that I don't think it can break "docker compatibility"

[giuseppe]

since we have no control on Docker Compose and the value is hardcoded, we probably need a check to clamp the oom-score-adj to the current value when running rootless

[vrothberg]

since we have no control on Docker Compose and the value is hardcoded, we probably need a check to clamp the oom-score-adj to the current value when running rootless

That is a great idea and aligns with Lennart's comment. Shall we still allow setting it to a higher value than the current one?

[giuseppe]

yes, setting to a higher value is fine

[giuseppe]

I can play with it and see how the idea works

[giuseppe]

opened a PR: #19843

[blaisep]

FWIW, I get this error even when rootless: false on OS X 12.6.9

[steinsag]

Is there a work-around like an environment or config variable I can set? My distro is currently stuck at podman 4.6.1.

Found it:

sudo dnf downgrade crun

crun version 1.8.7 works fine