Rhel 7.6, rootless with portmapping hangs on Created Container

[michielhemme]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

About 2 months ago I did a research on Podman (1.2-DEV was the most recent) about podman and running rootless containers with portbindings in Red Hat 7.6. After strugling for a while I've got it working with manually installing new RPM's (Saved those RPMS) and finding one FORK for shadow-utils. It was all working perfectly. And still is in that test environment.

Once rhel8 was released we've tried to implement the same solution (- the shadow-utils fork) but always got to the same point (Stuck at Started Container). I started doubting and installed a fresh RHEL7.6 and applied the same packages and now that one hangs at Stuck at Started Container too. When looking at the conmon logs it will stop after the commando:
ctl fifo path: /home.........
with a: terminal_ctrl_fd: 14

Steps to reproduce the issue:

1. Install Rhel7.6 with podman 1.3-dev, slirp4netns-3.0, runc version 1.0.0-rc7+dev, and shadow-utils46-newxidmap

2. Enable namespaces and add rootless user to /etc/subuid /etc/subgid

3. Start a rootless container with port bindings (Podman log-level=debug run -d -p 80 nginx)

4. See it hang on Started Container

Describe the results you received:
in the logs of conmon it stops witha terminal_ctrl_fd: 14
May 16 12:41:51 rhel76 kernel: SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
May 16 12:41:51 rhel76 conmon: conmon 73a48d2051aa548bbd70 : container PID: 32266
May 16 12:41:51 rhel76 conmon: conmon 73a48d2051aa548bbd70 : attach sock path: /run/user/1000/libpod/tmp/socket/73a48d2051aa548bbd702ff7f048da5df04e13653bf5d2275063a9d44ebd64ac/attach
May 16 12:41:51 rhel76 conmon: conmon 73a48d2051aa548bbd70 : addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/73a48d2051aa548bbd702ff7f048da5df04e13653bf5d2275063a9d44ebd64ac/attach}
May 16 12:41:51 rhel76 conmon: conmon 73a48d2051aa548bbd70 : ctl fifo path: /home/test/.local/share/containers/storage/vfs-containers/73a48d2051aa548bbd702ff7f048da5df04e13653bf5d2275063a9d44ebd64ac/userdata/ctl
May 16 12:41:51 rhel76 conmon: conmon 73a48d2051aa548bbd70 : terminal_ctrl_fd: 14

Describe the results you expected:
The same as my previous installation (Which was also Red Hat 7.6) a container being created with port bindings on rootless users.

Additional information you deem important (e.g. issue happens only occasionally):
I get the same error in RHEL8 and CENTOS7.6.

Output of podman version:

podman version 1.3.0-dev

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.2
  podman version: 1.3.0-dev
host:
  BuildahVersion: 1.8-dev
  Conmon:
    package: podman-1.3.0-21.dev.gitb01fdcb.fc31.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 3c163e4635ea7ed15cde0814e3bbf87fb759ee25'
  Distribution:
    distribution: '"rhel"'
    version: "7.6"
  MemFree: 147369984
  MemTotal: 3973677056
  OCIRuntime:
    package: runc-1.0.0-92.dev.gitc1b8c57.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc7+dev
      commit: 7f820969cc1cc8ea3357b39f2e2e1514f71c6fec
      spec: 1.0.1-dev
  SwapFree: 3435130880
  SwapTotal: 3435130880
  arch: amd64
  cpus: 1
  hostname: rhel76.michiel
  kernel: 3.10.0-957.el7.x86_64
  os: linux
  rootless: true
  uptime: 51h 51m 46.8s (Approximately 2.12 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ConfigFile: /home/test/.config/containers/storage.conf
  ContainerStore:
    number: 14
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/test/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 1
  RunRoot: /run/user/1000
  VolumePath: /home/test/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
Running in a proxmox cluster.
1CPU, 4GB RAM.

[giuseppe]

thanks for the detailed report.

I am closing this as a duplicate, as we are already tracking the problem here: #2942.

[giuseppe]

PR here: #3162