podman run -v /run:/run

[karezza]

/kind bug

Description

This is not a plex question but it's what I'm using to describe the issue. The following script works if I run it as root, but not if I run it as regular user. Issue seems to be around the "-v /run:/run" on RHEL 8.

Steps to reproduce the issue:

podman run
-d
--privileged
--net host
--ipc host
--pid host
--name plex
-e TZ="America/Denver"
-e PLEX_UID=717201107
-e PLEX_GID=717200513
-v /home/travis/opt/plex/db:/config
-v /home/travis/opt/plex/transcode:/transcode
-v /home/travis/opt/plex/media:/data
-v /run:/run
-e CHANGE_CONFIG_DIR_OWNERSHIP="true"
plexinc/pms-docker

Describe the results you received:
container create failed: container_linux.go:336: starting container process caused "process_linux.go:399: container init caused "rootfs_linux.go:58: mounting \"tmpfs\" to rootfs \"/home/travis/.local/share/containers/storage/overlay/59b0879bc9f255137c05850c307d8c9f34543d1fa08658a44c40f43bd950a17a/merged\" at \"/tmp/runctop091524734/runctmpdir731776453\" caused \"tmpcopyup: failed to copy /home/travis/.local/share/containers/storage/overlay/59b0879bc9f255137c05850c307d8c9f34543d1fa08658a44c40f43bd950a17a/merged/run/lock to /tmp/runctop091524734/runctmpdir731776453: open /home/travis/.local/share/containers/storage/overlay/59b0879bc9f255137c05850c307d8c9f34543d1fa08658a44c40f43bd950a17a/merged/run/lock/lvm: permission denied\"""
: internal libpod error

Describe the results you expected:
Expected the container to start up normally.

Additional information you deem important (e.g. issue happens only occasionally):
I was directed to use '- /run' on this plex page, if you are interested in why: https://hub.docker.com/r/plexinc/pms-docker/, search for '-v /run'.

Output of podman version:

Version:       1.0.2-dev
Go Version:    go1.11.5
OS/Arch:       linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.11.5
  podman version: 1.0.2-dev
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-2.git921f98f.module+el8+2785+ff8a053f.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: be8255a19cda8a598d76dfa49e16e337769d4528-dirty'
  Distribution:
    distribution: '"rhel"'
    version: "8.0"
  MemFree: 681615360
  MemTotal: 2869587968
  OCIRuntime:
    package: runc-1.0.0-54.rc5.dev.git2abd837.module+el8+2769+577ad176.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 4242010112
  SwapTotal: 4261408768
  arch: amd64
  cpus: 1
  hostname: boldocker-dev
  kernel: 4.18.0-80.el8.x86_64
  os: linux
  rootless: true
  uptime: 5h 57m 41.59s (Approximately 0.21 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.redhat.io
  - quay.io
  - docker.io
store:
  ConfigFile: /home/travis/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/travis/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 4
  RunRoot: /run/user/717201107

Additional environment details (AWS, VirtualBox, physical, etc.):
Hyper-V virtual system

[rhatdan]

This usually means a UID is attempted to be used in the container that is not defined in the user namespace. Do you have some directory that is not searchable inside of the container.

For example if your rootless container is attempting to crete content in /run/lock, it will not be allowed. Rootless Podman is still governed by the rules of linux. If your user is not allowed to create content in /run/lock then just running a container and volume mounting in /run/lock does not allow you to break the rules.

[rhatdan]

This needs to be written up in a blog.

[mheon]

Is this tmpcopyup?

…

On Tue, May 14, 2019, 07:45 Daniel J Walsh ***@***.***> wrote: This needs to be written up in a blog. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3120>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB3AOCG6QNSZPR7W6FHAAJ3PVKQ6NANCNFSM4HMV2BPQ> .

[rhatdan]

No that only happens on tmpfs mounts, this is a usernamepaced container trying to write to /run/lock on the host and being blocked. The same way as if you were logged in and tried to do a touch /run/lock/foobar.

[mheon]

The error string is "failed to mount tmpfs" - I think this is tmpcopyup

[mheon]

The specific error string: https://github.com/opencontainers/runc/blob/master/libcontainer/rootfs_linux.go#L266

This is 100% tmpcopyup

[karezza]

Are your discoveries pointing at a problem with the pms-docker image? Not being the owner of that image, is there anything I can do other than run as root?

[karezza]

The image provides a way to specify a UID/GID of the user being used in the container. I'm specifying the values I am able to see when I run 'id'. Whether podman is run as root or a regular user the UID/GID being used inside the container should be the same.

[mheon]

The Dockerfile: https://github.com/plexinc/pms-docker/blob/master/Dockerfile

@rhatdan I think I know what this is.
The entrypoint is /init - this probably isn't systemd, but it's triggering our systemd detection.

This causes several tmpfs filesystems to be mounted into the container, including (in Podman 1.2.x and earlier) one at /run/lock. We set the tmpcopyup option on all tmpfs filesystems by default, causing them to copy anything already in the directory into the new tmpfs that we'll mount over it.

Problem: you've mounted in /run/ as a whole from your system, so we try to copy up content from root. We don't have any permissions there.

Temporary solution: set --systemd=false at the Podman commandline until 1.3.x is available, which should not actually make the tmpfs in question.

[mheon]

@rhatdan I think we need to figure out how --systemd/--read-only-tmpfs interacts with cases like this in rootless mode, where we might break tmpcopyup if the user mounted in something from the system proper.

[karezza]

removing and creating new container with additional '--systemd=false' yields in the logs:

podman logs plex
s6-printenv: fatal: unable to open /var/run/s6/container_environment/PATH: Permission denied

[mheon]

That seems like a separate issue - probably caused by whatever is running within the container

[karezza]

k, I'll run as root and check user level again when 1.3.x is available. -- thank you.

[mheon]

Does your user have permission to access /var/run/s6/container_environment/PATH on the system? /var/run is a symlink to /run, so this is pointing to the host - those files might be owned by root or some other user

[rhatdan]

-v /run:/run
Is the problem, they are not writable by a non root user.
I have no idea why a tmpfs is getting involved, but the non root user will not be allowed to write to any directory that is not world writable or owned by one of the UIDS that he controls.

[rhatdan]

@karezza Could you show again the podman command you are using. I am basing my thoughts on the podman command you show in the problem report, which does not specify a user.

If you expect to have a user
-e PLEX_UID=717201107
-e PLEX_GID=717200513
With these UIDs to work, it will NOT in rootless containers, because you don't have this user or enough UIDS mappend into the container.
You probably only have UID 0-65536
mapped into your container, you tell podman to run as uid 717201107 it should fail, since this UID isnot in the user namespace.

[baude]

@karezza any updates?

[karezza]

I tried without specifying PLEX_UID & PLEX_GID, not sure what user that tries. According to rhatdan, the "-v /run:/run" is the problem. The pms-docker image itself said to use "-v /run:/run" as a work around for a bug in a patched version of Docker. I don't know if the problem has to do with the pms-docker image or on the podman side. In the meantime I gave up and ran the image as root. As directed I'll try again when 1.3.x is released. If you had something else for me to try in the meantime let me know. I'll go ahead and close this issue.