Port not listening on host when PODMAN_USERNS=keep-id is used

[patrickdung]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
After testing out the setting in #12848
(remove all pod/containers) and update setuid/setgid.

Now I found if I start a container with PODMAN_USERNS=keep-id, the port is not listening on the host. I can't find it in netstat. Previously it is listening.
Edit 2: It is on FC35, rootless mode. The host also have K3S root mode with Cilium CNI.
I found the pod can listen a port on the host if keep-id is not used.

Steps to reproduce the issue:

1. docker-compose.yml

---
version: "3.9"
services:
  service-nginx:
    image: public.ecr.aws/nginx/nginx:1.21-alpine
    container_name: "container-nginx"
    hostname: "container-nginx"
    environment:
    ports:
      - 127.0.0.1:5018:80

2. start the container with
   PODMAN_USERNS=keep-ip

The port 5018 is not listening on the host.

Describe the results you received:
Attaching a log file for starting the container
debug-log.txt

podman ps --ns -a | grep nginx
b96b95581b68 container-nginx 0

Describe the results you expected:
It should listen the port on the host.

Additional information you deem important (e.g. issue happens only occasionally):
I have run podman system migrate and the problem persist.

Output of podman version:

Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8
Built:        Thu Dec  9 05:45:07 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    version: "35"
  eventLogger: journald
  hostname: server1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.10-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 11714322432
  memTotal: 67081912320
  ociRuntime:
    name: crun
    package: crun-1.4-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4
      commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 20401082368
  swapTotal: 20401082368
  uptime: 1h 42m 49.83s (Approximately 0.04 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
store:
  configFile: /data/ssd1/home/user1/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 7
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /data/ssd1/home/user1/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 123
  runRoot: /run/user/1000
  volumePath: /data/ssd1/home/user1/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1638999907
  BuiltTime: Thu Dec  9 05:45:07 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.4-1.fc35.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical machine, desktop.

[Luap99]

I think I fixed this in #12227