v1.5.1 podman exec inconsistency with fedora/rhel v/s ubuntu containers

[lsm5]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
In my regular workflow I usually start containers in detached mode and exec to them as non-root user lsm5 and /home/lsm5 as my default dir. This worked fine with v1.4.4 on f30 but with the recent upgrade to v1.5.1, I'm getting OCI runtime permission denied errors.

Steps to reproduce the issue:

1. update rpm from 1.4.4 to 1.5.1
   dnf update podman

2. exec into a running container

3. see error

4. downgrade rpm to 1.4.4
   dnf downgrade podman

5. exec as expected

Describe the results you received:

Error: time="2019-08-17T14:25:35-04:00" level=error msg="exec failed: container_linux.go:346: starting container process caused \"chdir to cwd (\\\"/home/lsm5\\\") set in config.json failed: permission denied\"\n"
exec failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/home/lsm5\") set in config.json failed: permission denied": OCI runtime permission denied error

Additional information you deem important (e.g. issue happens only occasionally):
I get permission denied errors as above with rhel8 and fedora containers. But with ubuntu containers, I'm able to exec but only as root user, while with v1.4.4 I'm able to exec as non-root.

Output of podman version:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of rpm -q podman:

podman-1.5.1-3.fc30.x86_64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 9010958336
  MemTotal: 16449994752
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 8296329216
  SwapTotal: 8296329216
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: nagato.nanadai.me
  kernel: 5.2.8-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 26m 40.59s
registries:
  blocked: null
  insecure:
  - brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
  search:
  - rikudo.nanadai.me
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/lsm5/.config/containers/storage.conf
  ContainerStore:
    number: 14
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/lsm5/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 9
  RunRoot: /run/user/1000
  VolumePath: /home/lsm5/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
Local f30 host.

[mheon]

What exec command? @haircommander PTAL

…

On Sat, Aug 17, 2019, 14:42 Lokesh Mandvekar ***@***.***> wrote: *Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)* /kind bug *Description* In my regular workflow I usually start containers in detached mode and exec to them as non-root user lsm5 and /home/lsm5 as my default dir. This worked fine with v1.4.4 on f30 but with the recent upgrade to v1.5.1, I'm getting OCI runtime permission denied errors. *Steps to reproduce the issue:* 1. update rpm from 1.4.4 to 1.5.1 dnf update podman 2. exec into a running container 3. see error 4. downgrade rpm to 1.4.4 dnf downgrade podman 5. exec as expected *Describe the results you received:* Error: time="2019-08-17T14:25:35-04:00" level=error msg="exec failed: container_linux.go:346: starting container process caused \"chdir to cwd (\\\"/home/lsm5\\\") set in config.json failed: permission denied\"\n" exec failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/home/lsm5\") set in config.json failed: permission denied": OCI runtime permission denied error *Additional information you deem important (e.g. issue happens only occasionally):* I get permission denied errors as above with rhel8 and fedora containers. But with ubuntu containers, I'm able to exec but only as root user, while with v1.4.4 I'm able to exec as non-root. *Output of podman version:* Version: 1.5.1 RemoteAPI Version: 1 Go Version: go1.12.7 OS/Arch: linux/amd64 *Output of rpm -q podman:* podman-1.5.1-3.fc30.x86_64 *Output of podman info --debug:* debug: compiler: gc git commit: "" go version: go1.12.7 podman version: 1.5.1 host: BuildahVersion: 1.10.1 Conmon: package: podman-1.5.1-3.fc30.x86_64 path: /usr/libexec/podman/conmon version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty' Distribution: distribution: fedora version: "30" MemFree: 9010958336 MemTotal: 16449994752 OCIRuntime: package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64 path: /usr/bin/runc version: |- runc version 1.0.0-rc8+dev commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3 spec: 1.0.1-dev SwapFree: 8296329216 SwapTotal: 8296329216 arch: amd64 cpus: 8 eventlogger: journald hostname: nagato.nanadai.me kernel: 5.2.8-200.fc30.x86_64 os: linux rootless: true uptime: 26m 40.59s registries: blocked: null insecure: - brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888 search: - rikudo.nanadai.me - docker.io - registry.fedoraproject.org - quay.io - registry.access.redhat.com - registry.centos.org store: ConfigFile: /home/lsm5/.config/containers/storage.conf ContainerStore: number: 14 GraphDriverName: overlay GraphOptions: - overlay.mount_program=/usr/bin/fuse-overlayfs GraphRoot: /home/lsm5/.local/share/containers/storage GraphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" ImageStore: number: 9 RunRoot: /run/user/1000 VolumePath: /home/lsm5/.local/share/containers/storage/volumes *Additional environment details (AWS, VirtualBox, physical, etc.):* Local f30 host. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3838>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB3AOCEQZE4L6GRNXAFA3KTQFBBABANCNFSM4IMQEPIQ> .

[lsm5]

Run rawhide:

/usr/bin/podman run -dt --user lsm5 -h rawhide -w /home/lsm5 --net=host nanadai.me/rawhide

Exec rawhide:

/usr/bin/podman exec -it $(/usr/bin/podman ps | grep nanadai.me/rawhide | cut -c -12) /bin/bash

That's my own registry btw.

[lsm5]

Let me check if this happens with vanilla fedora images and report back..

[lsm5]

@mheon @haircommander hmm, so it seems I can't reproduce this with default docker.io/fedora:rawhide image but it happens consistently with my own image (which was built initially from the docker.io/fedora image but has been rebuilt and squashed over time).

lsm5 @ nagato : ~(master) $ podman ps
CONTAINER ID  IMAGE                             COMMAND    CREATED        STATUS            PORTS  NAMES
dd478160cac9  docker.io/library/fedora:rawhide  /bin/bash  4 minutes ago  Up 4 minutes ago         infallible_thompson
db715fa2b927  nanadai.me/rawhide:latest         /bin/bash  5 minutes ago  Up 5 minutes ago         infallible_benz

lsm5 @ nagato : ~(master) $ podman exec -it db715fa2b927 /bin/bash
Error: time="2019-08-17T16:48:29-04:00" level=error msg="exec failed: container_linux.go:346: starting container process caused \"chdir to cwd (\\\"/home/lsm5\\\") set in config.json failed: permission denied\"\n"
exec failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/home/lsm5\") set in config.json failed: permission denied": OCI runtime permission denied error

lsm5 @ nagato : ~(master) $ podman exec -it dd478160cac9 /bin/bash
[root@dd478160cac9 /]# exit
exit

lsm5 @ nagato : ~(master) $ rpm -q podman
podman-1.5.1-3.fc30.x86_64

[rhatdan]

SELinux?

[lsm5]

SELinux?

happens regardless.

[lsm5]

I see this with the latest build in rawhide as well podman-1.5.2-0.6.dev.git76f327f.fc32.x86_64

[edsantiago]

Reproducer:

$ cat Dockerfile
FROM fedora:rawhide
RUN adduser foo
$ podman build -t myrawhide --format=docker .
STEP 1: FROM fedora:rawhide
STEP 2: RUN adduser foo
STEP 3: COMMIT myrawhide
36120d9bc357d4afe1fe7e14252203307e792c613b160d545c29ae8b149469c9

$ podman run -dt --user foo -h rawhide -w /home/foo --net=host myrawhide
63df27ae3838789779f599acc2142e50a8a2fa63c1354fdb7e08dc97c7b3bc2c
$ podman exec -it 63df bash
Error: exec failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/home/foo\") set in config.json failed: permission denied": OCI runtime permission denied error

[edsantiago]

Not just fedora; s/fedora:rawhide/centos:latest/ in Dockerfile yields same results. The key seems to be that /home/foo is mode 700, and podman exec is defaulting to running as root.

-h rawhide and --net=host make no difference.

Adding --user foo to the exec command succeeds.

[haircommander]

@mheon Do you think exec should inherit the user of run? and --user would override that?

[rhatdan]

We should follow what Docker does.

[edsantiago]

This is nonroot

[haircommander]

/assign
I'll take this

[edsantiago]

@lsm5 can you retry and confirm with older podman? I'm seeing the same behavior with podman-1.4.4-4.fc30.x86_64

[rhatdan]

Why doesn't root in the user namespace have DAC_OVERRIDE and DAC_READSEARCH?
It looks like root does not have all capabilities that it normally does when execing into a container that was created for a user.

[rhatdan]

I still think we should follow the Docker standard, if the exec comes in as a the user or as root of the user namespace.

[haircommander]

docker retains the user of the run command in exec unless overriden. putting together a pr now