Inconsistent state recovery

[bnordgren]

/kind feature

Description

At some point in the past, removing a pod with podman failed and now I have an inconsistent state which I don't know how to fix. Apparently the storage for the infra container was deleted on disk, but podman still believes the container exists. I have one of these for my root instance as well as rootless. Podman should have some command to force cleanup of inconsistent state.

Current system behavior:

[bnordgren@brycelap ~]$ podman pod ls
POD ID         NAME   STATUS    CREATED       # OF CONTAINERS   INFRA ID
e61ac0328400   test   Created   4 weeks ago   1                 31770c3e3fda
[bnordgren@brycelap ~]$ podman ps -a
CONTAINER ID  IMAGE                 COMMAND  CREATED      STATUS   PORTS  NAMES
31770c3e3fda  k8s.gcr.io/pause:3.1           4 weeks ago  Created         e61ac0328400-infra
[bnordgren@brycelap ~]$ podman --log-level debug pod rm -f e61
INFO[0000] running as rootless                          
DEBU[0000] Initializing boltdb state at /var/home/bnordgren/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/bnordgren/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/bnordgren/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/bnordgren/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
INFO[0000] running as rootless                          
WARN[0000] The configuration is using `runtime_path`, which is deprecated and will be removed in future.  Please use `runtimes` and `runtime` 
WARN[0000] If you are using both `runtime_path` and `runtime`, the configuration from `runtime_path` is used 
DEBU[0000] Initializing boltdb state at /var/home/bnordgren/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/bnordgren/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/bnordgren/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/bnordgren/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Cleaning up container 31770c3e3fdabb91b2a8ccd2856a947782b5a6dcb02670efb4a39bb2a9773c74 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Storage is already unmounted, skipping...    
DEBU[0000] Storage is already unmounted, skipping...    
WARN[0000] Storage for container 31770c3e3fdabb91b2a8ccd2856a947782b5a6dcb02670efb4a39bb2a9773c74 already removed 
ERRO[0000] no such file or directory

Output of podman version:

On Fedora 30 Silverblue:

Version:            1.2.0
RemoteAPI Version:  1
Go Version:         go1.12
OS/Arch:            linux/amd64

This also happened prior to the upgrade, so whatever version Fedora 29 Silverblue was running.

Output of podman info --debug:

[bnordgren@brycelap ~]$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12
  podman version: 1.2.0
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2.0-2.git3bd528e.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: a13eadcaf0f07ba590c96d845a03ac3a0da7c56d'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 21758373888
  MemTotal: 33671335936
  OCIRuntime:
    package: runc-1.0.0-92.dev.gitc1b8c57.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc7+dev
      commit: d38f8a2d00ff444e52d16ab1ea5bbe5c1cb471e8
      spec: 1.0.1-dev
  SwapFree: 8380215296
  SwapTotal: 8380215296
  arch: amd64
  cpus: 4
  hostname: brycelap.usfs-i2.umt.edu
  kernel: 5.0.11-300.fc30.x86_64
  os: linux
  rootless: true
  uptime: 1h 47m 12.31s (Approximately 0.04 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/bnordgren/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/bnordgren/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 26
  RunRoot: /run/user/1000
  VolumePath: /var/home/bnordgren/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Laptop, Fedora 29-30 Silverblue

[mheon]

This sounds like it would be fixed by the recently-merged #3082

[mheon]

With #3082 merged and release inbound, I'm going to close.