podman doesn't run pod due ghost id which can't be removed

[ikke-t]

/kind bug

Description

I had pods running. Then I updated podman. Now I'm in state that pod can't be started due id for the pod name exists (ghost). The ghost pod can't be found for podman rm either using pod name or id. Still it prevents new one starting

Steps to reproduce the issue:

[root@ohuska ~]#  /usr/bin/podman run --name weechat   --rm -ti -p 61067:9001/tcp -v "/var/lib/containers/exported_volumes/weechat:/weechat:Z" --hostname=weechat.ikenet --memory=128M -e UID=1017 -e GID=1017   jkaberg/weechat
Error: error creating container storage: the container name "weechat" is already in use by "42a995f4335a0e358105c48be50b8e9bdf10d6f6cf1b0cee02f9f9540a2a268e". You have to remove that container to be able to reuse that name.: that name is already in use
[root@ohuska ~]# podman ps -a --sync
CONTAINER ID  IMAGE                                         COMMAND               CREATED         STATUS             PORTS                                           NAMES
48b10962e194  docker.io/library/nextcloud:latest            /entrypoint.sh ap...  20 minutes ago  Up 20 minutes ago  0.0.0.0:8090->80/tcp                            nextcloud
2768455aad5a  docker.io/itzg/minecraft-server:latest        /start                2 hours ago     Up 2 hours ago     0.0.0.0:25565->25565/tcp                        topin-mc
9f055846f317  docker.io/openhab/openhab:2.3.0-amd64-alpine  /entrypoint.sh su...  2 hours ago     Up 2 hours ago     0.0.0.0:8083->8080/tcp, 0.0.0.0:5007->5007/tcp  openhab
20c0f5403d71  docker.io/grafana/grafana:latest              /run.sh               2 hours ago     Up 2 hours ago     0.0.0.0:3000->3000/tcp                          grafana
fc8d44c21ef4  docker.io/jacobalberty/unifi:latest           /usr/local/bin/do...  2 hours ago     Up 2 hours ago     0.0.0.0:3478->3478/udp, 0.0.0.0:8080->8080/tcp  unifi
32a882de137d  docker.io/gogs/gogs:latest                    /app/gogs/docker/...  2 hours ago     Up 2 hours ago     0.0.0.0:10022->22/tcp, 0.0.0.0:10080->3000/tcp  gogs
1c1220987e41  docker.io/sebp/lighttpd:latest                start.sh              2 hours ago     Up 2 hours ago     0.0.0.0:8091->80/tcp                            pilivi
e7a08b5bf6d7  docker.io/nodered/node-red-docker:latest      npm start -- --us...  2 hours ago     Up 2 hours ago     0.0.0.0:1880->1880/tcp                          nodered
27bab314b9a5  localhost/ikke/phpbb:72.1                     docker-php-entryp...  2 hours ago     Up 2 hours ago     0.0.0.0:8082->80/tcp                            phpbb
[root@ohuska ~]# podman rm 42a995f4335a0e358105c48be50b8e9bdf10d6f6cf1b0cee02f9f9540a2a268e
Error: no container with name or ID 42a995f4335a0e358105c48be50b8e9bdf10d6f6cf1b0cee02f9f9540a2a268e found: no such container
[root@ohuska ~]# podman images
REPOSITORY                          TAG                  IMAGE ID       CREATED         SIZE
docker.io/itzg/minecraft-server     latest               190f19c0c637   16 hours ago    342 MB
docker.io/library/nextcloud         latest               d563376bf562   18 hours ago    684 MB
docker.io/jacobalberty/unifi        latest               953e374abca0   2 weeks ago     718 MB
<none>                              <none>               3b5326b6cbed   3 weeks ago     684 MB
docker.io/library/postgres          9.6                  564cc4c97fef   2 months ago    237 MB
docker.io/gogs/gogs                 latest               08fa8ee955da   2 months ago    104 MB
localhost/ikke/phpbb                72.1                 ea44ce204c76   2 months ago    409 MB
docker.io/jkaberg/weechat           latest               a69d41367816   3 months ago    107 MB
docker.io/openhab/openhab           2.3.0-amd64-alpine   ec6203721951   3 months ago    220 MB
docker.io/library/memcached         alpine               f0c2443012f4   3 months ago    10.6 MB
docker.io/grafana/grafana           latest               ffd9c905f698   3 months ago    246 MB
<none>                              <none>               d5b9815c59e1   3 months ago    101 MB
docker.io/nodered/node-red-docker   latest               c2a44c1dcd08   3 months ago    979 MB
docker.io/library/fedora            latest               d7372e6c93c6   3 months ago    283 MB
docker.io/ansible/awx_task          3.0.1                885ea0fe022d   4 months ago    1.07 GB
docker.io/ansible/awx_web           3.0.1                70ac81469770   4 months ago    1.04 GB
docker.io/sebp/lighttpd             latest               373b9578e885   6 months ago    12.5 MB
docker.io/ansible/awx_rabbitmq      3.7.4                e08fe791079e   14 months ago   90.9 MB
k8s.gcr.io/pause                    3.1                  da86e6ba6ca1   17 months ago   747 kB

[root@ohuska ~]# buildah rm weechat
eerror removing container "weechat": error reading build container: container not known
[root@ohuska ~]# buildah rm 42a995f4335a0e358105c48be50b8e9bdf10d6f6cf1b0cee02f9f9540a2a268e
error removing container "42a995f4335a0e358105c48be50b8e9bdf10d6f6cf1b0cee02f9f9540a2a268e": error reading build container: container not known

Additional information you deem important (e.g. issue happens only occasionally):

It could be caused due me doing dnf upgrade while pods were running.

Output of podman version:

Version:            1.4.0
RemoteAPI Version:  1
Go Version:         go1.11.10
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.11.10
  podman version: 1.4.0
host:
  BuildahVersion: 1.8.3
  Conmon:
    package: podman-1.4.0-2.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 0.2.0, commit: 80f19b1591b03dbbc53bf28b154230c7ea372281'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 234553344
  MemTotal: 8052482048
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 82f4855a8421018c9f4d74fbcf2da7f8ad1e11fa
      spec: 1.0.1-dev
  SwapFree: 8177053696
  SwapTotal: 8199860224
  arch: amd64
  cpus: 4
  hostname: ohuska.localdomain
  kernel: 5.1.8-200.fc29.x86_64
  os: linux
  rootless: false
  uptime: 3h 19m 3.2s (Approximately 0.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 10
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 19
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora 29 baremetal

[ikke-t]

thanks, works for me now!