ZFS storage driver with Docker-in-Docker leaves orphaned datasets

[AntiPaste]

When using the ZFS storage driver, Docker creates legacy ZFS datasets. These are automatically cleaned up when the container, image or volume in question is removed in Docker.

However, when using Docker-in-Docker, if the containers / images / volumes within DinD are not explicitly removed, removing the DinD container will orphan these datasets.

Steps to reproduce the issue:

1. Start with a clean ZFS dataset for Docker: zfs create -o mountpoint=/var/lib/docker -p zroot/data/docker

$ zfs list
NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot               96.7G   826G       24K  /zroot
zroot/data          93.8G   826G       24K  none
zroot/data/docker    462K   826G      462K  /var/lib/docker

2. Start up Docker-in-Docker daemon: docker run --privileged -d docker:dind
3. Start a container in DinD:

$ docker exec -i -t (...) sh
/ # docker run -i -t alpine:latest id

4. Take a note of the container and image dataset names:

/ # docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
alpine              latest              961769676411        6 weeks ago         5.58MB
/ # docker inspect -f '{{json .GraphDriver.Data}}' 961769676411 
{"Dataset":"zroot/data/docker/436757856ae60b2b78123322a5ad3a98a48d04af7ed8da8a3dcd1b1ebd951ed1","Mountpoint":"/var/lib/docker/zfs/graph/436757856ae60b2b78123322a5ad3a98a48d04af7ed8da8a3dcd1b1ebd951ed1"}
/ # docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
b34cd1d44b4c        alpine:latest       "id"                3 minutes ago       Exited (0) 3 minutes ago                       admiring_clarke
/ # docker inspect -f '{{json .GraphDriver.Data}}' b34cd1d44b4c
{"Dataset":"zroot/data/docker/86cbc95b7a50b7c09018455ce09bdfd2e88e0d7844aedd84b083da0321c2568d","Mountpoint":"/var/lib/docker/zfs/graph/86cbc95b7a50b7c09018455ce09bdfd2e88e0d7844aedd84b083da0321c2568d"}

5. Remove the DinD container on host and run docker system prune --all --volumes
6. Check that Docker is clean:

$ docker container ls -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$ docker image ls -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
$ docker volume ls
DRIVER              VOLUME NAME

7. Note that we still have 3 datasets:

NAME                                                                                      USED  AVAIL     REFER  MOUNTPOINT
zroot                                                                                    96.7G   826G       24K  /zroot
zroot/data                                                                               93.8G   826G       24K  none
zroot/data/docker                                                                        6.26M   826G      466K  /var/lib/docker
zroot/data/docker/436757856ae60b2b78123322a5ad3a98a48d04af7ed8da8a3dcd1b1ebd951ed1       5.77M   826G     5.77M  legacy
zroot/data/docker/86cbc95b7a50b7c09018455ce09bdfd2e88e0d7844aedd84b083da0321c2568d         15K   826G     5.76M  legacy
zroot/data/docker/86cbc95b7a50b7c09018455ce09bdfd2e88e0d7844aedd84b083da0321c2568d-init    28K   826G     5.76M  legacy

8. Note that the remaining datasets match the dataset names from step 4.

Describe the results you received:
There are 3 ZFS datasets created by Docker.

Describe the results you expected:
There should be no ZFS datasets created by Docker.

Additional information you deem important (e.g. issue happens only occasionally):
There are no Docker daemon logs that indicate any dataset removal was attempted.

Output of docker version:

Client:
 Version:           19.03.1-ce
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        74b1e89e8a
 Built:             Fri Aug 16 14:00:42 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.1-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       74b1e89e8a
  Built:            Fri Aug 16 13:58:02 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.7.m
  GitCommit:        85f6aa58b8a3170aec9824568f7a31832878b603.m
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.1-ce
 Storage Driver: zfs
  Zpool: zroot
  Zpool Health: ONLINE
  Parent Dataset: zroot/data/docker
  Space Used By Parent: 473600
  Space Available: 886434176512
  Parent Quota: no
  Compression: off
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 85f6aa58b8a3170aec9824568f7a31832878b603.m
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.2.9-arch1-1-ARCH
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.7GiB
 Name: pangea
 ID: FHUY:4LRE:QPTK:DEBE:KTNG:AP7U:GPMN:KATF:3PF4:C2DW:XWRE:LP65
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[cpuguy83]

Sounds like possibly the same issue we had with btrfs and we need to recursively delete datasets?

[D4nny-dev]

We could had the same issue , docker don't delete the orphan snapshots when clones are deleted. Bellow you can see that the REFER value not is equald to the USED.

(Before to apply manual clean up)zfs list:

NAME                                                                                   USED  AVAIL     REFER  MOUNTPOINT
docdata                                                                               73,3G  23,1G       24K  none
docdata/data                                                                            24K  20,0G       24K  /data
docdata/docker                                                                        73,3G  23,1G     39,1G  /srv/docker
docdata/docker/0002b19ac261305ce88fc7a9541a1e9d3c7f9ca28da220a53fe8272a0d4dca1c        128K  23,1G     1,30G  legacy

For soveld this we use this command
zfs list -t snapshot -r docdata/docker | awk '/docker\// { print $1 }' | xargs -l zfs destroy -v -R
after to apply the docker comands to prune all possibly data that docker can clean up!

zfs list:

NAME                                                                                   USED  AVAIL     REFER  MOUNTPOINT
docdata                                                                               39,7G  56,6G       24K  none
docdata/data                                                                            24K  20,0G       24K  /data
docdata/docker                                                                        39,7G  56,6G     39,1G  /srv/docker
docdata/docker/0644d88cec4b1a4194a7bc83f2102f1cd8bb0e67fcb2d8d4c1122039f62806ac       5,76M  56,6G     5,76M  legacy

Now the REFER value is the same as USED!