containerd integration: builder prune -a removes rootfs of built image

[vvoland]

Description

Building an image with buildkit and then pruning all build cache results in rootfs layers being deleted.
The image itself (containerd image) is not deleted, but the image is no longer complete, so it doesn't show in docker images.

$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
$ echo 'FROM ubuntu' | docker buildx build -t asdf -
[+] Building 4.1s (5/5) FINISHED
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                               0.0s
 => => transferring dockerfile: 49B                                                                                                                                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:latest                                                                                                                                                                                                                                                   1.7s
 => [1/1] FROM docker.io/library/ubuntu@sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21                                                                                                                                                                                                    1.6s
 => => resolve docker.io/library/ubuntu@sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21                                                                                                                                                                                                    0.0s
 => => sha256:cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38 27.35MB / 27.35MB                                                                                                                                                                                                                   1.5s
 => exporting to image                                                                                                                                                                                                                                                                                             2.3s
 => => exporting layers                                                                                                                                                                                                                                                                                            0.0s
 => => exporting manifest sha256:5c469ba407ba7ee47fdb025546c8490dcca6aa3d2806f51d2c4f183ff72fa2b2                                                                                                                                                                                                                  0.0s
 => => exporting config sha256:65ed03ad4005007a4116dfb415a623442ee3f61033ef77eb3b44203c437f67b5                                                                                                                                                                                                                    0.0s
 => => exporting attestation manifest sha256:f2cae65b78a3bf90c1c1f056e0fc1a913d2e3d655cc31c47f5166f12e11ac52e                                                                                                                                                                                                      0.0s
 => => exporting manifest list sha256:34958107a139de8ae6d3fe2514e56bc00d0936c1f65223efb3a3bcd66db65ebf                                                                                                                                                                                                             0.0s
 => => naming to docker.io/library/asdf:latest                                                                                                                                                                                                                                                                     0.0s
 => => unpacking to docker.io/library/asdf:latest                                                                                                                                                                                                                                                                  2.2s
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
asdf         latest    34958107a139   5 seconds ago   27.4MB
$ ctr images check
REF                           TYPE                                    DIGEST                                                                  STATUS         SIZE              UNPACKED
docker.io/library/asdf:latest application/vnd.oci.image.index.v1+json sha256:34958107a139de8ae6d3fe2514e56bc00d0936c1f65223efb3a3bcd66db65ebf complete (2/2) 26.1 MiB/26.1 MiB true


$ docker builder prune
WARNING! This will remove all dangling build cache. Are you sure you want to continue? [y/N] y
ID                                              RECLAIMABLE     SIZE            LAST ACCESSED
d8ra4weadt44eeaeg19d09y6u*                      true            4.096kB         10 seconds ago
70d2v2tc3zi2f8rppnzz5xtjs*                      true    8.192kB         10 seconds ago
Total:  12.29kB

$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
asdf         latest    34958107a139   16 seconds ago   27.4MB

$ ctr images check
REF                           TYPE                                    DIGEST                                                                  STATUS         SIZE              UNPACKED
docker.io/library/asdf:latest application/vnd.oci.image.index.v1+json sha256:34958107a139de8ae6d3fe2514e56bc00d0936c1f65223efb3a3bcd66db65ebf complete (2/2) 26.1 MiB/26.1 MiB true


$ ctr content ls
DIGEST                                                                  SIZE    AGE             LABELS
sha256:34958107a139de8ae6d3fe2514e56bc00d0936c1f65223efb3a3bcd66db65ebf 855B    20 seconds      containerd.io/gc.ref.content.0=sha256:5c469ba407ba7ee47fdb025546c8490dcca6aa3d2806f51d2c4f183ff72fa2b2,containerd.io/gc.ref.content.1=sha256:f2cae65b78a3bf90c1c1f056e0fc1a913d2e3d655cc31c47f5166f12e11ac52e
sha256:537da24818633b45fcb65e5285a68c3ec1f3db25f5ae5476a7757bc8dfae92a3 424B    20 seconds      containerd.io/distribution.source.docker.io=library/ubuntu
sha256:5c469ba407ba7ee47fdb025546c8490dcca6aa3d2806f51d2c4f183ff72fa2b2 482B    20 seconds      containerd.io/gc.ref.content.0=sha256:65ed03ad4005007a4116dfb415a623442ee3f61033ef77eb3b44203c437f67b5
sha256:65ed03ad4005007a4116dfb415a623442ee3f61033ef77eb3b44203c437f67b5 1.519kB 20 seconds      containerd.io/gc.ref.snapshot.overlayfs=sha256:874b048c963ab55b06939c39d59303fb975d323822a4ea48a02ac8dc635ea371
sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc295a0428a6d21 1.133kB 21 seconds      containerd.io/distribution.source.docker.io=library/ubuntu
sha256:7fd7c87af8b212c7b56a4e536dea00981f86d3debbf587b53949d0f36a0e75ee 4.062kB 18 seconds      -
sha256:b5bcd034b594609d8145f633863b7208942f441bf4cc195d875841eeba66cc6b 167B    20 seconds      -
sha256:bab8ce5c00ca3ef91e0d3eb4c6e6d6ec7cffa9574c447fd8d54a8d96e7c1c80e 2.316kB 20 seconds      containerd.io/distribution.source.docker.io=library/ubuntu
sha256:cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38 27.35MB 18 seconds      buildkit.io/blob/annotation.containerd.io/uncompressed=sha256:874b048c963ab55b06939c39d59303fb975d323822a4ea48a02ac8dc635ea371,buildkit.io/blob/mediatype=application/vnd.oci.image.layer.v1.tar+gzip,containerd.io/gc.ref.content.blob-sha256:cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38=sha256:cd741b12a7eaa64357041c2d3f4590c898313a7f8f65cd1577594e6ee03a8c38
sha256:d1c292de84abfaf5f601f64626f44658e1c2ef021ffc301da725fcf91ccca519 4.19kB  18 seconds      -
sha256:e41003bf072899b87a572f8dfdff918649d1d26b22800b7a5f6a10950677d078 1.031kB 20 seconds      -
sha256:f2cae65b78a3bf90c1c1f056e0fc1a913d2e3d655cc31c47f5166f12e11ac52e 566B    20 seconds      containerd.io/gc.ref.content.0=sha256:b5bcd034b594609d8145f633863b7208942f441bf4cc195d875841eeba66cc6b,containerd.io/gc.ref.content.1=sha256:e41003bf072899b87a572f8dfdff918649d1d26b22800b7a5f6a10950677d078



$ docker builder prune -a
WARNING! This will remove all build cache. Are you sure you want to continue? [y/N] y
ID                                              RECLAIMABLE     SIZE            LAST ACCESSED
l6sqxi8kkhlh2q4qkrs0up7kx                       true            27.35MB         23 seconds ago
Total:  27.35MB
$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE
$ ctr images check
REF                           TYPE                                    DIGEST                                                                  STATUS           SIZE             UNPACKED
docker.io/library/asdf:latest application/vnd.oci.image.index.v1+json sha256:34958107a139de8ae6d3fe2514e56bc00d0936c1f65223efb3a3bcd66db65ebf incomplete (1/2) 1.5 KiB/26.1 MiB true
$ ctr content ls
DIGEST                                                                  SIZE    AGE             LABELS
sha256:34958107a139de8ae6d3fe2514e56bc00d0936c1f65223efb3a3bcd66db65ebf 855B    44 seconds      containerd.io/gc.ref.content.0=sha256:5c469ba407ba7ee47fdb025546c8490dcca6aa3d2806f51d2c4f183ff72fa2b2,containerd.io/gc.ref.content.1=sha256:f2cae65b78a3bf90c1c1f056e0fc1a913d2e3d655cc31c47f5166f12e11ac52e
sha256:5c469ba407ba7ee47fdb025546c8490dcca6aa3d2806f51d2c4f183ff72fa2b2 482B    44 seconds      containerd.io/gc.ref.content.0=sha256:65ed03ad4005007a4116dfb415a623442ee3f61033ef77eb3b44203c437f67b5
sha256:65ed03ad4005007a4116dfb415a623442ee3f61033ef77eb3b44203c437f67b5 1.519kB 44 seconds      containerd.io/gc.ref.snapshot.overlayfs=sha256:874b048c963ab55b06939c39d59303fb975d323822a4ea48a02ac8dc635ea371
sha256:7fd7c87af8b212c7b56a4e536dea00981f86d3debbf587b53949d0f36a0e75ee 4.062kB 41 seconds      -
sha256:b5bcd034b594609d8145f633863b7208942f441bf4cc195d875841eeba66cc6b 167B    44 seconds      -
sha256:d1c292de84abfaf5f601f64626f44658e1c2ef021ffc301da725fcf91ccca519 4.19kB  41 seconds      -
sha256:e41003bf072899b87a572f8dfdff918649d1d26b22800b7a5f6a10950677d078 1.031kB 44 seconds      -
sha256:f2cae65b78a3bf90c1c1f056e0fc1a913d2e3d655cc31c47f5166f12e11ac52e 566B    44 seconds      containerd.io/gc.ref.content.0=sha256:b5bcd034b594609d8145f633863b7208942f441bf4cc195d875841eeba66cc6b,containerd.io/gc.ref.content.1=sha256:e41003bf072899b87a572f8dfdff918649d1d26b22800b7a5f6a10950677d078

Reproduce

$ echo 'FROM ubuntu' | docker buildx build -t asdf -
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
asdf         latest    34958107a139   16 seconds ago   27.4MB
$ docker buildx prune -a
WARNING! This will remove all build cache. Are you sure you want to continue? [y/N] y
ID                                              RECLAIMABLE     SIZE            LAST ACCESSED
l6sqxi8kkhlh2q4qkrs0up7kx                       true            27.35MB         23 seconds ago
Total:  27.35MB
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED          SIZE

Expected behavior

No response

docker version

master - 2fa8f54d0869511f88efd318f67fe1d06915e78a

docker info

-

Additional Info

No response

[vvoland]

Looks like image manifest is missing a gc.ref label for the layer (it only references config):

sha256:5c469ba407ba7ee47fdb025546c8490dcca6aa3d2806f51d2c4f183ff72fa2b2 482B    44 seconds      containerd.io/gc.ref.content.0=sha256:65ed03ad4005007a4116dfb415a623442ee3f61033ef77eb3b44203c437f67b5

[thaJeztah]

• I see Copy History API objects to a different namespace buildkit#3797 is part of BuildKit v0.12

Does that mean #45966 will fix this?

/cc @neersighted

[neersighted]

Oh! Good question! I think so, cc @tonistiigi to be sure, as my mental model of what is stored in which namespace isn't 100% clear.

[tonistiigi]

Unless it is covered by https://github.com/moby/buildkit/pull/3972/files#diff-93b81ebb2d74a7ff89a643dfd137603e309b87754076de895b0c15dd4626a169R382 it shouldn't be related to #45966 . The image itself needs to track all the objects it needs.