Provide the official images in the stargz format

[gabrieldemarmiesse]

Hi!

With buildx and buildkit, it's possible to leverage Docker image in the stargz format. It's now even easier since Docker Desktop added support for containerd as the image store in beta, with the stargz snapshotter preinstalled.

This allow for much, much faster start of containers and could be a game changer in CI/CD where a lot of time is spent pulling images. Overall it's a net win for the user experience in general as docker pulls are much faster.

Docker image in the stargz format are backward compatible as the format is OCI-complient. See https://github.com/containerd/stargz-snapshotter

It's possible to build and push stargz Docker image by using buildkit and buildx. By using the option: compression=estargz,oci-mediatypes=true and optionally force-compression=true if the base image is not in the stargz format.

See https://github.com/moby/buildkit/blob/master/docs/stargz-estargz.md#creating-stargzestargz-images for more information about stargz and buildkit.

It's backward compatible so shouldn't be an issue to build all the new Docker images with this format, but we can be careful and for a time provide a new suffix -esgz for users who want to try lazy pulling with stargz. This doubles the number of images build but ensure maybe a smoother path to stargz while bugs are flushed out of the buildkit exporter (if any)

[tianon]

I did a few straight conversions in https://hub.docker.com/u/glitter a short while ago and honestly was a little disappointed that in most cases, the on-registry compressed images were larger, even with more aggressive compression levels than what Docker's using by default. 😬

I've been interested in https://github.com/awslabs/soci-snapshotter specifically because it's essentially stargz but without requiring a costly conversion or format change, but I haven't had a chance to actually dig in and play with it at all yet.

[ktock]

the on-registry compressed images were larger, even with more aggressive compression levels than what Docker's using by default.

@tianon Thank you very much for trying it. estargz can become larger than the original image (python:3.9 esgz 922MB and original 885MB on docker) mainly because it adds a metadata file to the layer blob. But pulling it can be much faster by lazy pulling 😄
If you observed that estargz is problematically large, please share it with us and we'll work on fixing it.

[ktock]

@tianon
FYI: Stargz snapshotter v0.13.0 supports creating images without large increasing of the image size by using --estargz-min-chunk-size and --estargz-external-toc.

• Docs: https://github.com/containerd/stargz-snapshotter/blob/v0.13.0/docs/smaller-estargz.md
• Size comparison (python and deneon/plasma images can be converted to eStargz without increasing size): Support creating smaller eStargz images (--estargz-external-toc and --estargz-min-chunk-size) containerd/stargz-snapshotter#956

[AkihiroSuda]

@tianon @yosifkit Any idea on how to move this forward?

• Option 1: keep the existing images as-is, and add eStargz images with :<version>-esgz suffix

• Option 2: convert the images to eStargz w/o external TOC, and add eStargz w/ external TOC images with :<version>-esgz suffix. The total footprint is smaller than Option 1.

[tianon]

Unfortunately, we will not be adding any additional tags to support this. We will also not be using an extra tool to "munge" our images after build. If we are to support something here, it either has to be a 100% optional add-on that does not change the original image (ala https://github.com/awslabs/soci-snapshotter if generating that content map post-build were more straightforward) or it needs to become not only part of buildkit/buildx, but with high enough confidence from that team/those maintainers that it becomes enabled-by-default behavior.

[AkihiroSuda]

Unfortunately, we will not be adding any additional tags to support this. We will also not be using an extra tool to "munge" our images after build. If we are to support something here, it either has to be a 100% optional add-on that does not change the original image (ala https://github.com/awslabs/soci-snapshotter if generating that content map post-build were more straightforward) or it needs to become not only part of buildkit/buildx, but with high enough confidence from that team/those maintainers that it becomes enabled-by-default behavior.

@tianon @ktock

Would it be acceptable to just use stargz for layers, but split the stargz.index.json blob into a separate manifest just like provenances and SBOM JSON blobs?

There will be just a single tag (== single OCI index), and no need to have an extra tool to "munge", if we can add this functionality to buildx.

This will also retain 100% compatibility with legacy engines.