Rootless podman missing some permissions within the container

[lud-hu]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When trying to start up weblate as a rootless container the container crashes with

mkdir: cannot create directory ‘/run’: Permission denied

If I check user permissions, groups, directory permissions everything looks the same in a working docker vs non working podman environment. Is there a way to get it working on a RHEL7 machine with podman 1.6.4? I've seen some similar issues here but no of the mentioned solution (e.g. enabling user lingering) was working.
I thought the running as rootless just has differences for the host machine environment and not within the container.

Steps to reproduce the issue:

1. Startup weblate container in pod with
   podman run -v weblate-data:/app/data --env-file ./environment --env-file ./environment.override -d --pod weblatepod weblate/weblate

Describe the results you received:
Container does not start up due to "Permission denied" error.

Describe the results you expected:
Container starts up properly as it does in docker environment.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:
Due to usage of RHEL7 I cannot update podman to a higher version.

podman version 1.6.4

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.11.5
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.15-1.el7_8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 372b4a12f1c2df4f70c280d41173b60acd3f1260'
  Distribution:
    distribution: '"rhel"'
    version: "7.9"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 6100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 6100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 566820864
  MemTotal: 33557946368
  OCIRuntime:
    name: runc
    package: runc-1.0.0-67.rc10.el7_8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 4288925696
  SwapTotal: 4294963200
  arch: amd64
  cpus: 2
  eventlogger: file
  hostname: si0vm4034
  kernel: 3.10.0-1160.15.2.el7.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /bin/slirp4netns
    Package: slirp4netns-0.4.3-4.el7_8.x86_64
    Version: |-
      slirp4netns version 0.4.3
      commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
  uptime: 418h 26m 7.63s (Approximately 17.42 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  ConfigFile: /home/user/.config/containers/storage.conf
  ContainerStore:
    number: 137
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
      Version: |-
        fuse-overlayfs: version 0.7.2
        FUSE library version 3.6.1
        using FUSE kernel interface version 7.29
  GraphRoot: /.../home/user/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 231
  RunRoot: /run/user/6100/containers
  VolumePath: /.../home/user/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.4-27.el7_9.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No and Yes

[vrothberg]

Thanks for reaching out! Do other containers work? Are you running podman in podman or on the host?

[lud-hu]

Thanks for the quick reply! Other containers are running quite fine and I'm running podman on the host.

[rhatdan]

Permission denied, I always first check SELinux?

If you run the container --privileged does it work?

[lud-hu]

Great, if I start the container with the privileged flag it's starting up as expected. Are there any downsides on starting it like this?

[lud-hu]

And now there's another permission denied error where it can not execute a bash script file:

fatal: cannot exec '/app/data/ssh/ssh-weblate-wrapper-db5c5ca9e000131e': Permission denied

It looks like a SELinux problem again. Is there a way to get this working in podman?
I already tried this approach but it's not working: https://www.redhat.com/sysadmin/user-namespaces-selinux-rootless-containers

[rhatdan]

You need to use a newer version of Podman is the problem. Since this is fixed in current podman, this should be opened as a Bugzilla. and most likely if there is no Podman that supports this on RHEL7, then you are going to be told to upgrade to RHEL8. RHEL7 will not be getting any more updates.