Use containerd image readiness check function to properly recover image during restart.

[Random-Liu]

Currently, during restart we read all image references from containerd image metadata store, and recover the image list.

However, a image reference in containerd image metadata store doesn't mean the image is ready for use, the image could be corrupted because of aborted image pulling.

We should use the newly added Check function containerd/containerd#1567 to check content completeness and check top level snapshot existence.

[yanxuean]

I can help on this. @Random-Liu
What is the deadline of alpha.1?

[Random-Liu]

Not decided yet. I think at least not next week. :)

[yanxuean]

I will enjoy my holiday:)

[Random-Liu]

containerd/containerd#1567 is merged.

[Random-Liu]

@yanxuean Please only check image readiness during restart recovery for now. We'll consider whether we should always check readiness in list and inspect later.

And when check readiness, you need to use the new containerd function to check whether all contents are present, and also whether the top level snapshot exists (with ChainID).

If yes, good to add the image into our image store.
If not, remove the corresponding image reference from containerd, and not add it to our image store.

[yanxuean]

@Random-Liu I hit some problem. I need your help.
1.How to check whether the top snapshot exist?
c.client.SnapshotService(c.config.ContainerdConfig.Snapshotter).Stat(ctx, image.ChainID)

Seem like The Stat() only check if it is exist in io.containerd.metadata.v1.bolt/meta.db.

2. The Check() is not effective when I delete the layer content in blob on test.
   The Check() can not check it.
   The Check() only check if layer metadata is exist in io.containerd.metadata.v1.bolt/meta.db, regardless of if the layer content is exist in io.containerd.content.v1.content/blobs/sha256.

I1012 10:41:29.498036   18653 restart.go:336] check image readiness for docker.io/library/nginx:latest: {MediaType:application/vnd.docker.distribution.manifest.list.v2+json Digest:sha256:004ac1d5e791e705f12a17c80d7bb1e8f7f01aa7dca7deee6e65a03465392072 Size:2029 URLs:[] Annotations:map[] Platform:<nil>}

root@ubuntu:/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256# cat 004ac1d5e791e705f12a17c80d7bb1e8f7f01aa7dca7deee6e65a03465392072 | python -m json.tool
{
    "manifests": [
        {
            "digest": "sha256:2e6775f4300fc79b9d7fe6bb60c83b5fefe584258d9318ed408746789af48885",
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            },
            "size": 948
        },


root@ubuntu:/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256# cat 2e6775f4300fc79b9d7fe6bb60c83b5fefe584258d9318ed408746789af48885 |python -m json.tool
{
    "config": {
        "digest": "sha256:1e5ab59102ce46c277eda5ed77affaa4e3b06a59fe209fe0b05200606db3aa7a",
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 5835
    },
    "layers": [
        {
            "digest": "sha256:bc95e04b23c06ba1b9bf092d07d1493177b218e0340bd2ed49dac351c1e34313",
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 22492350
        },
        {
            "digest": "sha256:110767c6efff0c07d3a2a7e033e3961203af477b7797fa6c1d1b34d2ef822788",
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 21869756
        },
        {
            "digest": "sha256:f081e0c4df75654a42ca06d46db3c6935582d2ec6aedc9c64fa342c24fedeb41",
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 203
        }
    ],
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "schemaVersion": 2
}

I renamed the files related to layer.
root@ubuntu:/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256# ll f081e0c4df75* 110767c6ef* bc95e04b23c*
ls: cannot access f081e0c4df75*: No such file or directory
ls: cannot access 110767c6ef*: No such file or directory
ls: cannot access bc95e04b23c*: No such file or directory
root@ubuntu:/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256# ll back*
-r--r--r-- 1 root root 21869756 Oct 11 10:37 back110767c6efff0c07d3a2a7e033e3961203af477b7797fa6c1d1b34d2ef822788
-r--r--r-- 1 root root 22492350 Oct 11 10:37 back-bc95e04b23c06ba1b9bf092d07d1493177b218e0340bd2ed49dac351c1e34313
-r--r--r-- 1 root root      203 Oct 11 10:37 backf081e0c4df75654a42ca06d46db3c6935582d2ec6aedc9c64fa342c24fedeb41
root@ubuntu:/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256#

[yanxuean]

@Random-Liu need help :)