[Bug]: error removing container when no free space is left on filesystem

[chilikk]

Issue Description

When /var/lib/containers is located on an XFS filesystem, it is impossible to remove container when no free space is left on that filesystem. Moreover, podman ends up in a bad state where the container is no longer visible in the summary but the container's storage is left behind.

This situation could, for example, be caused by a container that has exhausted its storage - it becomes impossible to remove such container.

In my reproduction the /var/lib/containers resides on XFS filesystem. I have not been able to reproduce this issue with ext4.

Steps to reproduce the issue

Steps to reproduce the issue

1. I am reproducing this in a VM, so initialize the environment first:

mkdir reproduction
cd reproduction
vagrant init generic/centos9s
vagrant up
vagrant ssh 

2. Inside the VM, install required packages and mount /var/lib/containers on an XFS filesystem, set the necessary SELinux attributes

sudo yum install -y xfsprogs podman
sudo fallocate -l 300M /xfs.bin
sudo mkfs.xfs /xfs.bin
sudo mount -t xfs -o loop /xfs.bin /var/lib/containers
sudo chcon -u system_u -t container_var_lib_t /var/lib/containers

3. Start a container that fills its own storage and exits

sudo podman pull docker.io/library/alpine:3.17
sudo podman run --name test docker.io/library/alpine:3.17 sh -c 'dd if=/dev/zero of=/bigfile || exit 1'

4. Try to remove the container, see the error message

$ sudo podman rm test
Error: removing container a3dcb9bde158e64c40429476da0362a6b305d36b0b05b60a732305d2fc2ec08a root filesystem: 2 errors occurred:
        * open /var/lib/containers/storage/overlay-layers/.tmp-layers.json16945529: no space left on device
        * open /var/lib/containers/storage/overlay-containers/.tmp-containers.json2124646358: no space left on device

5. Verify that despite the error above the container is gone from the podman container ls -a list, however the disk for /var/lib/containers is still full which means that the container's storage was left behind:

$ sudo podman container ls -a
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
$ df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      295M  295M   32K 100% /var/lib/containers

Describe the results you received

Error message when deleting a container, container gone from the list of containers while container storage is left behind.

Describe the results you expected

Container successfully removed.

podman info output

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: 48adb81a22c26f0660f0f37d984baebe7b9ade98'
  cpuUtilization:
    idlePercent: 94.9
    systemPercent: 1.74
    userPercent: 3.37
  cpus: 2
  distribution:
    distribution: '"centos"'
    version: "9"
  eventLogger: journald
  hostname: centos9s.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-205.el9.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 221835264
  memTotal: 1864462336
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-2.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.2.0-2.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 0h 7m 18.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 308969472
  graphRootUsed: 308936704
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1669638068
  BuiltTime: Mon Nov 28 12:21:08 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

I have been able to reproduce the issue with XFS filesystem, but not with ext4 filesystem.

[Luap99]

Duplicate of #13967