Is zstd:chunked layer compression supported?

[polarathene]

I know that images with zstd compressed layers are supported, but I'm not having much luck verifying zstd:chunked with it's optimizations for pulls is available like it is with Podman?

One of the big advantages of zstd:chunked for example is:

FROM large-base-image as layered
RUN echo 'example' > /opt/example

FROM scratch
COPY --from=layered / /

When large-base-image is available as zstd:chunked compressed layers, if you have already pulled this content then the cost of pulling that squashed image above would only be to pull the /opt/example file.

• This is quite useful with multi-layer images that are built inefficiently (extra copies of a file in previous layers from modification/removal), since that content can be avoided from pulling (and across images a file that only differs by metadata also avoids needing to be pulled).
• Without additional features from a snapshotter like eStargz, the image pull would still need to fetch any missing content before the image is started (no on-demand pull of file content, aka lazy image pull), what zstd:chunked is doing is referred to as partial pulling. eStargz or similar would be complimentary.

I am aware that:

• Both zstd + zstd:chunked use the same OCI mediatype, where zstd:chunked layers are backwards-compatible to being treated as zstd layers.
• The eStargz snapshotter has support for zstd:chunked in addition to gzip but the documentation on zstd:chunked is missing (from what I understand they would not be standard zstd:chunked images but include additional changes to integrate with eStargz?).

[ChengyuZhu6]

@polarathene You can use nerdctl to verify zstd:chunked images: https://github.com/containerd/nerdctl/blob/main/docs/stargz.md#tips-3-using-zstd-instead-of-gzip-aka-zstdchunked

[polarathene]

Short-reply:

• I know of eStargz snapshotter for pull support and tooling like nerdctl for image conversion. That is extending a standard zstd-chunked image with eStargz AFAIK?
• I wanted to confirm if containerd supports pulling zstd:chunked (with improvement over zstd format, such as partial layer pulls), and if not would there be plans to support this in future or only delegate to third-party support?

---

? You misunderstood I think? (or my understanding of containerd role is lacking) I have updated my original question with more information.

The link you provided is:

• Convert an image to use zstd:chunked + eStargz compression.
  • With --estargz-min-chunk-size + --estargz-external-toc unsupported for zstd:chunked (presumably an upstream limitation with the eStargz snapshotter, but their docs don't mention this as they only focus on gzip)
• Pull support requires eStargz? There is no native zstd:chunked support for containerd?

The latter is what my question here was about.

• zstd:chunked is distinct from eStargz, that snapshotter just adopted support for that as an alternative to gzip.
• Podman supports both eStargz and zstd:chunked images for pulling. I am not sure of semantics when using eStargz + zstd:chunked together there, but their storage implementation distinguishes the two as separate image formats, whilst using reflinks for deduplication for on-disk storage (XFS or BTRFS), otherwise opt-in config to use hardlinks (but this has some compatibility issues IIRC).

I know that containerd supports zstd, so when pulling zstd:chunked formatted image (no eStargz involved) it will pull the whole layer without any file-level deduplication/sharing.

With Podman the file content is distinct from file attributes such as mtime which is tracked separately (from the tar-split IIRC), but that storage like Docker is uncompressed, whereas I think containerd stores the compressed layers on-disk too instead of discarding them? (if so that may make sense for why support is delegated to a snapshotter plugin like eStargz, I have seen an alternative like SOCI snapshotter support a config setting to not persist compressed layers (with it's parallel mode))

I recall that Docker had it's own pkg/archive but has since adopted the equivalent containerd library for that functionality. This includes the support for decompressing zstd tar layers, and is where I thought support for zstd:chunked would belong too if it were supported.

[AkihiroSuda]

cc @ktock

[saolof]

Zstd:chunked images are valid zstd images. So they will be supported by any runtime that supports zstd, the real question is if they actually do something clever to do partial pulls like what podman and cri-o do