avc: denied starting a container

[alcir]

/kind bug

Description

selinux is enabled; please note: using setenforce 0, the container works.

Steps to reproduce the issue:
podman build --tag ubuntu:16.04base -f ./Mydockerfile

podman create --hostname ubuntu16workbench --interactive --name ubuntu16workbench --network host --privileged --security-opt label=disable --tty --volume $HOME:$HOME --volume $XDG_RUNTIME_DIR:$XDG_RUNTIME_DIR --volume /dev/dri:/dev/dri --volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=:0 ubuntu:16.04base

All goes well.

Running podman start -ia ubuntu16workbench I get this result:

Describe the results you received:

Error: unable to start container 6129aecf9d94cb665621e06db7b0437290502f2d74227623ec1b34f63d923a57: error reading container (probably exited) json message: EOF

Journalctl:

Mar 25 16:20:34 host audit[1369]: AVC avc:  denied  { create } for  pid=1369 comm="runc:[2:INIT]" scontext=unconfined_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=key permissive=0

Output of podman version:

Version:            1.1.2
RemoteAPI Version:  1
Go Version:         go1.11.5
Git Commit:         a95a49d3038462d033f84ac314ec8a3064a99cff
Built:              Tue Mar  5 19:10:31 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: a95a49d3038462d033f84ac314ec8a3064a99cff
  go version: go1.11.5
  podman version: 1.1.2
host:
  BuildahVersion: 1.7.1
  Conmon:
    package: podman-1.1.2-1.git0ad9b6b.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: a95a49d3038462d033f84ac314ec8a3064a99cff'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 1476567040
  MemTotal: 8072105984
  OCIRuntime:
    package: runc-1.0.0-85.dev.gitdd22a84.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: 1d3f73d4086533a858613bc4b6af2b5e882f4730
      spec: 1.0.1-dev
  SwapFree: 8195141632
  SwapTotal: 8205103104
  arch: amd64
  cpus: 4
  hostname: alessio.info.ms.fgm
  kernel: 5.0.3-200.fc29.x86_64
  os: linux
  rootless: true
  uptime: 43m 9.29s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/alessio/.config/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/alessio/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 9
  RunRoot: /run/user/1000
  VolumePath: /home/alessio/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Fully updated Fedora 29 (update-testing enabled) on baremetal

[debarshiray]

What does this say:

$ podman --log-level debug start -ia ubuntu16workbench

I had a user on Fedora 30 complain about something similar and I am trying to figure out if this is the same issue.

[mheon]

@rhatdan PTAL

[alcir]

I had a user on Fedora 30 complain about something similar and I am trying to figure out if this is the same issue.

Maybe I'm the same user 😄

What does this say:

$ podman --log-level debug start -ia ubuntu16workbench

This is the result (with selinux enforcing):

INFO[0000] running as rootless                          
WARN[0000] The configuration is using `runtime_path`, which is deprecated and will be removed in future.  Please use `runtimes` and `runtime` 
WARN[0000] If you are using both `runtime_path` and `runtime`, the configuration from `runtime_path` is used 
DEBU[0000] Initializing boltdb state at /home/alessio/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/alessio/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /home/alessio/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/alessio/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "vfs"   
DEBU[0000] Initialized SHM lock manager at path /libpod_rootless_lock_1000 
DEBU[0000] Handling terminal attach                     
DEBU[0000] mounted container "23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4" at "/home/alessio/.local/share/containers/storage/vfs/dir/ec1f59c46c8bc9f1eb20aa836e1277d048a4d5d0901d23bffb23eb599fdf089f" 
DEBU[0000] Created root filesystem for container 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4 at /home/alessio/.local/share/containers/storage/vfs/dir/ec1f59c46c8bc9f1eb20aa836e1277d048a4d5d0901d23bffb23eb599fdf089f 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
WARN[0000] failed to parse language "en_US.UTF-8": language: tag is not well-formed 
DEBU[0000] Created OCI spec for container 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4 at /home/alessio/.local/share/containers/storage/vfs-containers/23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args=[-c 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4 -u 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4 -r /usr/bin/runc -b /home/alessio/.local/share/containers/storage/vfs-containers/23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4/userdata -p /run/user/1000/vfs-containers/23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4/userdata/pidfile -l /home/alessio/.local/share/containers/storage/vfs-containers/23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --conmon-pidfile /run/user/1000/vfs-containers/23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/alessio/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg vfs --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4 --socket-dir-path /run/user/1000/libpod/tmp/socket -t --log-level debug --syslog]
WARN[0000] Failed to add conmon to cgroupfs sandbox cgroup: mkdir /sys/fs/cgroup/systemd/libpod_parent: permission denied 
DEBU[0000] Cleaning up container 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4" 
ERRO[0000] unable to start container 23c8bed901fbbf46c37dd2e22054edd3de6c1c5b50ce9cc3f7ff9c5975c517b4: error reading container (probably exited) json message: EOF

[TomSweeneyRedHat]

This won't cure the issue, but may give more diagnostics. Can you do ausearch -m avc -ts recent and grab the output? If you haven't done the create for a while, you may need to redo it then do this just after it fails.

[alcir]

sudo ausearch -m avc -ts recent

returns

time->Mon Mar 25 22:20:11 2019
type=AVC msg=audit(1553548811.464:304): avc:  denied  { create } for  pid=9209 comm="runc:[2:INIT]" scontext=unconfined_u:system_r:container_runtime_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=key permissive=0

[alcir]

I had a user on Fedora 30 complain about something similar and I am trying to figure out if this is the same issue.

BTW I have the same issue on a fresh Fedora 30 installation.

podman version
Version:            1.1.2
RemoteAPI Version:  1
Go Version:         go1.12
OS/Arch:            linux/amd64

[rhatdan]

There was a change to runc that is implementing keyring labeling. That might be causing this issue. Are these happening on SELinux disabled systems?

container-selinux-2.91 is being built to allow this.

[alcir]

There was a change to runc that is implementing keyring labeling. That might be causing this issue. Are these happening on SELinux disabled systems?

Disabling SELinux, on the same systems, podman start works.

[alatiera]

I had a user on Fedora 30 complain about something similar and I am trying to figure out if this is the same issue.

Maybe I'm the same user 😄

nah, that was me! Hi everybody :)

I just wanted to confirm that I also have SELinux set to enforcing, haven't touched any policies since I installed Fedora 30. The behavior and errors I hit seem to be identical to the ones pasted above. If there any more information I can provide, I would be happy to do so.

➜  ~ rpm -q podman
podman-1.1.2-2.dev.git0ad9b6b.fc30.x86_64
➜  ~ rpm -q container-selinux
container-selinux-2.90-1.git619db17.fc30.noarch

[alcir]

Follow up.

On Fedora 29 I updated to container-selinux-2:2.91-1.gitacc6941.fc29.noarch and now podman works.

On Fedora 30 there is still container-selinux-2.90-1.git619db17.fc30.noarch and it doesn't work.

Edit:
I also updated F30 to container-selinux-2:2.91-1.gitacc6941.fc30.noarch, and it works as well.

Thank you

[rhatdan]

Lease update the karma on f29.

https://bodhi.fedoraproject.org/updates/FEDORA-2019-32784d7fc6

[alatiera]

Same here on F30. The update to container-selinux-2.91-1.gitacc6941.fc30.noarch made podman work.