Container left semi-alive after removal failed with error unmounting container: layer is not mounted

[mildred]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Container destruction (podman rm) failed, and since then the container name is blocked (cannot create a new container with the same name) but the old container does not show, and cannot be removed.

Steps to reproduce the issue:

Seems like an odd error, difficult to reproduce.

1. Container was deleted with podman rm project_cable_1 and the following error happened: ERRO[0003] cleanup storage: failed to cleanup container 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936 storage: error unmounting container 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936 root filesystem: layer is not mounted Error: error cleaning up container 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936: error unmounting container 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936 storage: error unmounting container 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936 root filesystem: layer is not mounted

2. Container is not showing up in podman ps

3. Creating a new container with the same name fails with: Error: error creating container storage: the container name "project_cable_1" is already in use by "20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936". You have to remove that container to be able to reuse that name.: that name is already in use

4. Deleting the container by name fails with: Error: no container with name or ID project_cable_1 found: no such container

5. Deleting container by ID fails with: Error: no container with name or ID 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936 found: no such container

Describe the results you received:

Cannot delete container, cannot recreate a container with the same name

Describe the results you expected:

Either deletion works or recreation works...

Initial container removal should have worked. When trying to umount a layer, the fact that it was not mounted should not have been fatal to container removal. In both cases, the layer ends up unmounted.

If the removal does not work, something should be left over to allow manual suppression afterwards.

Additional information you deem important (e.g. issue happens only occasionally):

Seems an off error that happened while using podman-compose which is issuing many podman commands quickly one after another.

The following sequence of commands were issues for that container:

• podman create --name=project_cable_1 --pod=project which succeeded
• podman start -a project_cable_1 which might have been interrupted and exited with status -2 (podman-compose was stopped at that moment)
• podman stop -t=1 project_cable_1 failed with Error: can only stop created, running, or stopped containers. 20956322f967c942eade94cb45f63b419c2871fafcc8c78fdb78bfe87fefe936 is in state configured: container state improper
• podman rm failed as described above

Output of podman version:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 4204826624
  MemTotal: 16501366784
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 17179865088
  SwapTotal: 17179865088
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: faile
  kernel: 5.2.13-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 42m 34.16s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/mildred/.config/containers/storage.conf
  ContainerStore:
    number: 15
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/bin/fuse-overlayfs
  GraphRoot: /var/home/mildred/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 63
  RunRoot: /run/user/1000
  VolumePath: /var/home/mildred/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-3.fc30.x86_64
fuse-overlayfs-0.6.2-2.git67a4afe.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Laptop running Fedora Silverblue

[giuseppe]

there is an open PR to take care of such issue: #3549

[mheon]

This doesn't need evict - Podman rm --storage should take care of it

…

On Mon, Sep 16, 2019, 08:39 Giuseppe Scrivano ***@***.***> wrote: there is an open PR to take care of such issue: #3549 <#3549> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4033>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB3AOCFPUAWWDBINWEFBL6LQJ5473ANCNFSM4IW7JIJA> .

[mheon]

The error here is properly strange, though - we should be ignoring that already during removal.

[mheon]

Ah, nevermind - we handle no-such-container errors, but not not-mounted errors.

Very strange that the container unmounted and the DB still showed it as mounted, still. That could have led to very strange undefined results if you tried to restart the container.

Fix for the condition that caused this is in #4035 but it's mainly just treating the symptoms. If we can get a reproducer for whatever allowed the container to unmount while the DB said it was mounted, that needs to be fixed.