[wip] Support additional layer store (patch for podman)

[ktock]

#4739

This enables podman to create containers using layers stored in a specified directory instead of pulling them from the registry. Leveraging this feature with remotely-mountable layers provided by stargz/zstd:chunked or CVMFS, podman can achieve lazy pulling.

That directory is named "additional layer store" (call ALS in this doc) and has the following structure.

<ALS root>
`-- <ID of layer1>
`-- <ID of layer2>
...
`-- .pool

Each layer is extracted and stored in ALS with a unique directory name (<ID of layer1>, <ID of layer2> in the above). There is .pool directory which is used for layer discovery described later.

Changes in contianers/storage: containers/storage#795

When the storage driver (e.g. overlay) is configured to use ALS, it uses layers in ALS with regarding each directory name as layer ID. In overlay driver, each layer directory in ALS is treated as the layer's diff directory, with assuming lower = "" and no link in the l directory.

When the storage.Store is configured to use ALS, this loads layers from ALS as read-only with assuming they don't have compressed digest nor uncompressed digest. This assumption should be the right thing just because exploded layers aren't archived nor compressed.

storage.Store also has layer discovery functionality which is indispensable for enabling lazy pulling. When a layer is queried with an ID via Layer() API with annotations (key1=value1, key2=value2...) and if that ID doesn't exist in ALS, storage.Store searches that layer from .pool directory with the following path.

<ALS root>/.pool/base64("key1=value1")/base64("key2=value2")/.../rootfs.<layer ID>

Each path element is base64 encoded key-value pair in the annotation.
The underlying filesystem (e.g. stargz/zstd:chunked-based filesystem or CVMFS) should show the exploded view of the target layer at that location. Example filesystem implementation (currently stargz-based) is https://github.com/ktock/stargz-snapshotter/tree/als-pool-example (this must be mounted under <ALS root>/.pool)

If storage.Store successfully accessed to this location, this creates the following symlink.

<ALS root>/<layer ID> -> <ALS root>/base64(key1=value1)/base64(key2=value2)/.../rootfs.<layer ID>

Then this layer (named <layer ID>) is accessible by the driver (with <layer ID>). So podman don't need to pull this layer from the registry.

Changes in containers/image: containers/image#1109

Now copier leverages this store. Every time this pulls images, it first tries to reuse blobs using Layer() API. This passes all annotations appended to the OCI descriptor of each layer and it also adds the following annotations by default for supporting registry-backed filesystems (e.g. stargz/zstd:chunked).

• containers/image/target.layerdigest: The digest of the target layer.
• containers/image/target.reference: The image reference of an image that contains the target layer.

When this copier successfully acquired that layer, it reused it without pulling.

Changes in containers/podman: #8837

This patch also fixes libpod to eliminate unnecessary check of digest and size information of each layer
at runtime.

cc: @siscia @giuseppe

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ktock
To complete the pull request process, please assign umohnani8 after the PR has been reviewed.
You can assign the PR to them by writing /assign @umohnani8 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[AkihiroSuda]

How will this work with Podman CLI?

[ktock]

Assuming /tmp/storage as the additional layer store and https://github.com/ktock/stargz-snapshotter/tree/als-pool-example is mounted at /tmp/storage/.pool (see also https://github.com/ktock/stargz-snapshotter/blob/als-pool-example/script/demo/run2.sh).

Image can be pulled with skipping downloading layers using --storage-opt:

podman --storage-opt "layerstore=/tmp/storage" pull ghcr.io/stargz-containers/rethinkdb:2.3.6-esgz

This image can run using the same storage option:

podman --storage-opt "layerstore=/tmp/storage" run --rm -it ghcr.io/stargz-containers/rethinkdb:2.3.6-esgz /bin/echo 'Hello, World!'

Image pulled in this way cannot be exported (e.g. podman saved).
For doing this, that image needs to be re-pulled without --storage-opt.

podman pull ghcr.io/stargz-containers/rethinkdb:2.3.6-esgz
podman save ghcr.io/stargz-containers/rethinkdb:2.3.6-esgz > /tmp/img.tar

[mtrmac]

I don’t understand this; NewImage is pretty much NewImageSource+FromSource already. How does this change behavior?

[ktock]

This is for skipping *storageImageSource.getSize() which includes checks of size >= 0 and uncompressedDigest != "" (exploded layers provided through additional layer store aren't archived nor compressed).
Additionally, the size information seems not to be used in *storageService.CreateContainerStorage()?

[mtrmac]

This is for skipping *storageImageSource.getSize() which includes checks of size >= 0 and uncompressedDigest != "" (exploded layers provided through additional layer store aren't archived nor compressed).

Ah, thanks. But that just looks like a design issue to be fixed, rather than worked around here. At the very least CRI-O actually calls types.Image.Size, so it should continue to work — and in this case the image object has already been created locally, so there should have been an opportunity to create all the relevant metadata.

Also, storage.Layer.UncompressedDigest and the like is a part of the API, not something that can be just ignored and not filled in without reviewing the full ecosystem — and in this case, an essential value without which things like podman push would AFAICS stop working.

[mtrmac]

FromSource “takes ownership” of src, so now this closes src twice.

[ktock]

Redesigning not to depend on this patch. (please see containers/image#1109 and containers/storage#795)

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[ktock]

After several redesigning, this PR is now not needed.
(containers/image#1109 and containers/storage#795 are still needed)