overlay2 cannot be cleaned despite 0 containers / volumes / images

[antoinetran]

Description

After a day or two, our continuous deployment of new docker images to VM fills the space of /var/lib/docker. Removing all containers/images/volumes, and then doing docker system prune --all --volumes --force does not reclaim space.
Workaround: remove docker RPM, then remove the directory manually.

Steps to reproduce the issue:

1. Push docker images each night (around 40GB, but with lots of common layers)
2. Wait for /var/lib/docker/overlay2 size > 70GB

docker rm $(docker ps -aq) -f
docker system prune --all --volumes --force
docker volume rm $(docker volume ls -q)
docker rmi $(docker images -aq) -f

Describe the results you received:

[root@ci1l2pdck-s00 ~]# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B
[root@ci1l2pdck-s00 ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@ci1l2pdck-s00 ~]# docker images -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
[root@ci1l2pdck-s00 ~]# docker volume ls
DRIVER    VOLUME NAME

[root@ci1l2pdck-s00 ~]# du -sh /var/lib/docker
76G     /var/lib/docker
[root@ci1l2pdck-s00 ~]# du -sh /var/lib/docker/*
76K     /var/lib/docker/buildkit
0       /var/lib/docker/containers
97M     /var/lib/docker/image
184K    /var/lib/docker/network
76G     /var/lib/docker/overlay2
0       /var/lib/docker/plugins
0       /var/lib/docker/runtimes
0       /var/lib/docker/swarm
0       /var/lib/docker/tmp
0       /var/lib/docker/trust
180K    /var/lib/docker/volumes
[root@ci1l2pdck-s00 ~]# sudo du -h --max-depth 1 --one-file-system /var/lib/docker
0       /var/lib/docker/plugins
76G     /var/lib/docker/overlay2
97M     /var/lib/docker/image
180K    /var/lib/docker/volumes
0       /var/lib/docker/trust
184K    /var/lib/docker/network
0       /var/lib/docker/swarm
76K     /var/lib/docker/buildkit
0       /var/lib/docker/containers
0       /var/lib/docker/tmp
0       /var/lib/docker/runtimes
76G     /var/lib/docker

Describe the results you expected:
du -sh /var/lib/docker/overlay2 should be less than 10MB with 0 containers 0 images 0 volumes etc.

Additional information you deem important (e.g. issue happens only occasionally):
Systematic but take a day or two to reproduce. Happens in cloud and onpremise machines.

Output of docker version:

[root@ci1l2pdck-s00 ~]# docker version
Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        dea9396
 Built:             Thu Nov 18 00:38:53 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:37:17 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

[root@ci1l2pdck-s00 ~]# docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.49.1.el7.x86_64
 Operating System: Red Hat Enterprise Linux Server 7.9 (Maipo)
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 58.82GiB
 Name: ci1l2pdck-s00.localdomain
 ID: BQUW:D4LN:GWHW:IEIM:FHRN:7HIV:DTDV:3N2G:AZQI:CHQR:ILWR:MP6K
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Cloud VM CentOS 7.9:

[root@ci1l2pdck-s00 ~]# uname -ar
Linux ci1l2pdck-s00.localdomain 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 9 16:09:48 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Physical machines Redhat 7.9
RPM: docker-ce-20.10.11-3.el7.x86_64 in both
Kubernetes:

[rancher@ci1l2psto-s00 ~]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:35:38Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

[antoinetran]

It seems we did not have this issue in an older version. We migrated a few month ago from 19.03.12-3.el7.x86_64 (mirror of 2020-09-11) to 20.10.11-3.el7.x86_64 (mirror of 2021-12-07).

[antoinetran]

For info I did the removal of docker-ce and persistence many time, so this is not just an issue of old layers orphaned from the migration of docker upgrade.

[antoinetran]

#32420