containerd integration: verify "Size" vs "VirtualSize" in images

[thaJeztah]

Description

When looking at the API response for GET /images/json with containerd integration enabled, I noticed that Size and VirtualSize were not the same;

curl --unix-socket /var/run/docker.sock 'http://anyhost/images/json' | jq .

[
  {
    "Containers": -1,
    "Created": 1678271418,
    "Id": "sha256:d8dc78532e9eb3759344bf89e6e7236a34132ab79150607eb08cc746989aa047",
    "Labels": null,
    "ParentId": "",
    "RepoDigests": [
      "mysql:latest@sha256:d8dc78532e9eb3759344bf89e6e7236a34132ab79150607eb08cc746989aa047"
    ],
    "RepoTags": [
      "mysql:latest"
    ],
    "SharedSize": -1,
    "Size": 155450613,
    "VirtualSize": 584658944
  },
  {
    "Containers": -1,
    "Created": 1681236201,
    "Id": "sha256:acaddd9ed544f7baf3373064064a51250b14cfe3ec604d65765a53da5958e5f5",
    "Labels": null,
    "ParentId": "",
    "RepoDigests": [
      "busybox@sha256:acaddd9ed544f7baf3373064064a51250b14cfe3ec604d65765a53da5958e5f5@sha256:acaddd9ed544f7baf3373064064a51250b14cfe3ec604d65765a53da5958e5f5"
    ],
    "RepoTags": [
      "busybox@sha256:acaddd9ed544f7baf3373064064a51250b14cfe3ec604d65765a53da5958e5f5"
    ],
    "SharedSize": -1,
    "Size": 2594229,
    "VirtualSize": 4993024
  }
]

However, they are expected to be the same or at least, that's what the API documents it as;

Total size of the image including all layers it is composed of.

In versions of Docker before v1.10, this field was calculated from
the image itself and all of its parent images. Docker v1.10 and up
stores images self-contained, and no longer use a parent-chain, making
this field an equivalent of the Size field.

This field is kept for backward compatibility, but may be removed in
a future version of the API.

The new containerd integration shows different sizes for Size and VirtualSize;

moby/daemon/containerd/image_list.go

Lines 64 to 67 in 98d8343

	VirtualSize: virtualSize,
	ID: img.Target().Digest.String(),
	Created: img.Metadata().CreatedAt.Unix(),
	Size: size,

moby/daemon/containerd/service.go

Lines 185 to 198 in 5bf405b

	snapshotSizeFn := func(d digest.Digest) (int64, error) {
	if s, ok := sizeCache[d]; ok {
	return s, nil
	}
	u, err := snapshotter.Usage(ctx, d.String())
	if err != nil {
	return 0, err
	}
	sizeCache[d] = u.Size
	return u.Size, nil
	}
	chainIDs := identity.ChainIDs(img.RootFS.DiffIDs)
	virtualSize, err := computeVirtualSize(chainIDs, snapshotSizeFn)

moby/daemon/containerd/image_list.go

Lines 479 to 489 in 5bf405b

	func computeVirtualSize(chainIDs []digest.Digest, sizeFn func(d digest.Digest) (int64, error)) (int64, error) {
	var virtualSize int64
	for _, chainID := range chainIDs {
	size, err := sizeFn(chainID)
	if err != nil {
	return virtualSize, err
	}
	virtualSize += size
	}
	return virtualSize, nil
	}

Whereas the old implementation considered them equal;

moby/daemon/images/image_list.go

Lines 251 to 252 in 98d8343

	Size: size,
	VirtualSize: size,

VirtualSize was a concept from "before" Docker 1.10, where images were not self-contained, and had "parent" images; in this situation Size was the size of the image, and VirtualSize the size of the image including the layer-size of all its parents.

However, starting with Docker 1.10, images are self-contained, which means that an image contains a list of all of its layers, so Size became an equivalent for VirtualSize.

investigate using containerd's WithSnapshotUsage()

We currently get image Size using image.Size(), which includes the size of the Manifest (and compressed layers???);
https://github.com/containerd/containerd/blob/8c27ce41930c52d2a536ff1c31d80ffdc01845e6/image.go#L162-L164

We should look into containerd's WithSnapshotUsage , which looks to include the size of the snapshots as well (which looks to be a close(r) match to what we want to present?
https://github.com/containerd/containerd/blob/8c27ce41930c52d2a536ff1c31d80ffdc01845e6/image.go#L94-L101