Nydus: refactor build workflow with packer #15

imeoer · 2022-01-05T03:39:14Z

Introduce a new package called packer, which implements the build workflow of nydus image, as well as the export and import of build cache, the build workflow previously implemented by the nydus driver has been wrapped into
this package.

The reason for introducing packer is that nydus intends to integrate packer as a generic component into other build toolkits such as buildkit to more easily support building and exporting of nydus image, we have an available implementation on buildkit in this PR. cc @AkihiroSuda @ktock

Unlike other compression format (gzip, estargz etc.) in OCI image, nydus is divided into two types of layer, blob and bootstrap, where blob serves as the data part of each layer of the image, and bootstrap serves as the metadata of the whole image, the bootstrap is equivalent to the view of the whole image filesystem after all layers overlay. For example, for an OCI image with 3 layers, the corresponding nydus image is 4 layers (3 layers of blob + 1 layer of bootstrap), and since nydus supports chunk-level data de-duplication, it can do cross-layer or cross-image data de-duplication, so it is possible that an OCI layer does not have a corresponding nydus blob layer, because it was de-duplicated during the build workflow.

The packer also implements support for build cache, which is used to speed up the conversion or build of nydus image, i.e. when a built image layer appears in a new image, it does not need to be built again. nydus' build cache principle is to write the blob and bootstrap data of each layer to the content store at each build, and store the cache record. In the next build, will find the layers that have already been built from the cache record, skip the build of these cached layer, and reuse the cached bootstrap as parent to build a new nydus image with new layers.

The build cache interface (SetCache/GetCache) in packer does not implement in acceld yet, but only buildkit. Next, acceld will implement a cache manager to support build cache for each driver (nydus, estargz etc.). cc @ktock

Nydus compression type support on buildkit: moby/buildkit#2581

Signed-off-by: Yan Song imeoer@linux.alibaba.com

ktock · 2022-01-05T12:36:53Z

pkg/driver/nydus/packer/packer.go

+		_parentBootstrapPath := path.Join(bootstrapDir, "parent-bootstrap")
+		parentBootstrapPath = &_parentBootstrapPath
+		layer := layers[*diffSkip]
+		cachedBootstrap, _ := layer.GetCache(ctx, CompressionTypeBootstrap)


error shouldn't be ignored or should have a comment at least.

Added a comment.

ktock · 2022-01-05T12:37:07Z

pkg/driver/nydus/packer/packer.go

+		layer := layers[idx]
+
+		// Find the layer cache.
+		cachedBootstrapDesc, _ := layer.GetCache(ctx, CompressionTypeBootstrap)


error shouldn't be ignored or should have a comment at least.

Added a comment.

ktock · 2022-01-05T12:37:36Z

pkg/driver/nydus/packer/mount.go

@@ -0,0 +1,109 @@
+// Copyright Project BuildKit Authors


Should have a reference to the original file

Thanks, added a comment for this.

Introduce a new package called packer, which implements the build workflow of nydus image, as well as the export and import of build cache, the build workflow previously implemented by the nydus driver has been wrapped into this package. The reason for introducing packer is that nydus intends to integrate packer as a generic component into other build toolkits such as buildkit to more easily support building and exporting of nydus image. Unlike other compression format (gzip, estargz etc.) in OCI image, nydus is divided into two types of layer, blob and bootstrap, where blob serves as the data part of each layer of the image, and bootstrap serves as the metadata of the whole image, the bootstrap is equivalent to the view of the whole image filesystem after all layers overlay. For example, for an OCI image with 3 layers, the corresponding nydus image is 4 layers (3 layers of blob + 1 layer of bootstrap), and since nydus supports chunk-level data de-duplication, it can do cross-layer or cross-image data de-duplication, so it is possible that an OCI layer does not have a corresponding nydus blob layer, because it was de-duplicated during the build workflow. The packer also implements support for build cache, which is used to speed up the conversion or build of nydus image, i.e. when a built image layer appears in a new image, it does not need to be built again. nydus' build cache principle is to write the blob and bootstrap data of each layer to the content store at each build, and store the cache record. In the next build, will find the layers that have already been built from the cache record, skip the build of these cached layer, and reuse the cached bootstrap as parent to build a new nydus image with new layers. The build cache interface (SetCache/GetCache) in packer does not implement in acceld yet, but only buildkit. Next, acceld will implement a cache manager to support build cache for each driver (nydus, estargz etc.). Signed-off-by: Yan Song <imeoer@linux.alibaba.com>

imeoer · 2022-01-06T09:45:25Z

Hi @AkihiroSuda @ktock

Nydus is supporting (PR) the export of image in buildkit as a compression type based on the packer package in this PR, compared to other compression types, nydus has the following points to be noted:

Nydus image layers don't correspond one-to-one with OCI image layers, some OCI image layers may not have a corresponding nydus blob layer, and there is an additional bootstrap layer in nydus image.
Nydus image layers don't support being placed in an image manifest together with other compression types, e.g. can't use a layer combination like [tar.gz + tar.gz + nydus.blob].
Nydus build and runtime (Proposal for taking a non-core project "nydus-snapshotter" in containerd/project#83) only supports linux image for now, not windows yet.

Help to see if there are any comments or suggestions? We'll submit a formal PR to buildkit next, thanks.

Nydus driver majorly works for registry backend, which means blob is stored in registry as per OCI distribution specification. But nydus can also make other storage services as backend storage. This commit add the support of aliyun OSS. Signed-off-by: Yan Song <imeoer@linux.alibaba.com>

ktock · 2022-01-11T11:37:33Z

Nydus image layers don't correspond one-to-one with OCI image layers, some OCI image layers may not have a corresponding nydus blob layer, and there is an additional bootstrap layer in nydus image.

Can BuildKit use the built nydus image as base image or build caches? How does it work with remote cache exporter?

imeoer · 2022-01-12T04:05:36Z

@ktock Thanks for the feedback.

Can BuildKit use the built nydus image as base image or build caches?

Haven't yet, currently this pre commit only support to export nydus image, I will take a look at the implementation of remote snapshot, to support it based on nydus snapshotter next.

How does it work with remote cache exporter?

Does the remote cache mean skipping the layer export of compressed blob already in remote registry, but not in local content store?

ktock · 2022-01-13T13:16:42Z

Does the remote cache mean skipping the layer export of compressed blob already in remote registry, but not in local content store?

I meant https://github.com/moby/buildkit#export-cache

imeoer · 2022-01-13T16:43:58Z

Does the remote cache mean skipping the layer export of compressed blob already in remote registry, but not in local content store?

I meant https://github.com/moby/buildkit#export-cache

Please correct me if I have misunderstood: :)

As I understand it, this cache is only applied in the build stage but not in the export stage, is that right? The current implementation of nydus image export does not affect the build stage.

I found that the inline cache is a point that nydus manifest needs to be compatible, will feedback later after some testing.

imeoer · 2022-01-19T03:57:32Z

I found that the inline cache is a point that nydus manifest needs to be compatible, will feedback later after some testing.

Is still working for inline cache export.

ktock · 2022-01-20T04:53:02Z

imeoer/buildkit@1f37cb5

	layer.sr.SetString(key, string(valBytes), "")

Nydus blob doesn't seems to be recorded as a blob in BuildKit's cache metadata? I'm a bit worry about does it work with other parts of the cache. Could it guarantee that this works with other cache APIs like DiskUsage, Prune, etc?

imeoer · 2022-01-21T10:59:37Z

imeoer/buildkit@1f37cb5
	layer.sr.SetString(key, string(valBytes), "")
Nydus blob doesn't seems to be recorded as a blob in BuildKit's cache metadata? I'm a bit worry about does it work with other parts of the cache. Could it guarantee that this works with other cache APIs like DiskUsage, Prune, etc?

@ktock Thanks for the reply. There is indeed this problem. Since nydus has cross-layer de-duplication, some cache records may not have nydus blob, and nydus blob cannot be mixed with other compression types in the same image.

This makes it unsuitable to record nydus blob directly to cache metadata using SetBlob, so adding a new KV item for recording. I'll give some more thought to how to be compatible with DiskUsage.

imeoer · 2022-01-24T09:32:19Z

imeoer/buildkit@1f37cb5
	layer.sr.SetString(key, string(valBytes), "")
Nydus blob doesn't seems to be recorded as a blob in BuildKit's cache metadata? I'm a bit worry about does it work with other parts of the cache. Could it guarantee that this works with other cache APIs like DiskUsage, Prune, etc?

@ktock Fixed the issue by recording nydus bootstrap to cache metadata using setBlob:

// Try to cache nydus bootstrap if no other compression type is set.
if err := layer.sr.setBlob(ctx, compression.Parse(packer.CompressionTypeBootstrap), desc.Bootstrap); err != nil {
	return nil, errors.Wrap(err, "set nydus bootstrap")
}

If there are no more problems with this commit for now, I will submit a formal PR to buildkit next, thanks.

imeoer requested review from bergwolf, liubogithub, wy65701436, steven-zou and ktock January 5, 2022 03:39

imeoer changed the title ~~Nydus: refactor build workflow with packer~~ [WIP] Nydus: refactor build workflow with packer Jan 5, 2022

imeoer force-pushed the main branch from 3615fb8 to d1518ff Compare January 5, 2022 06:36

imeoer changed the title ~~[WIP] Nydus: refactor build workflow with packer~~ Nydus: refactor build workflow with packer Jan 5, 2022

ktock reviewed Jan 5, 2022

View reviewed changes

imeoer force-pushed the main branch from d1518ff to 52219af Compare January 6, 2022 02:45

ktock approved these changes Jan 11, 2022

View reviewed changes

imeoer force-pushed the main branch 10 times, most recently from aceef38 to 050f0a9 Compare January 17, 2022 10:06

imeoer force-pushed the main branch 24 times, most recently from b575e97 to 8d7732f Compare January 18, 2022 08:38

imeoer merged commit a835c46 into goharbor:main Jan 18, 2022

imeoer mentioned this pull request Jan 26, 2022

Support for exporting nydus compression type moby/buildkit#2581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nydus: refactor build workflow with packer #15

Nydus: refactor build workflow with packer #15

imeoer commented Jan 5, 2022 •

edited

Loading

ktock Jan 5, 2022

imeoer Jan 6, 2022

ktock Jan 5, 2022

imeoer Jan 6, 2022

ktock Jan 5, 2022

imeoer Jan 6, 2022

imeoer commented Jan 6, 2022 •

edited

Loading

ktock commented Jan 11, 2022

imeoer commented Jan 12, 2022

ktock commented Jan 13, 2022

imeoer commented Jan 13, 2022 •

edited

Loading

imeoer commented Jan 19, 2022

ktock commented Jan 20, 2022

imeoer commented Jan 21, 2022 •

edited

Loading

imeoer commented Jan 24, 2022 •

edited

Loading

Nydus: refactor build workflow with packer #15

Nydus: refactor build workflow with packer #15

Conversation

imeoer commented Jan 5, 2022 • edited Loading

ktock Jan 5, 2022

Choose a reason for hiding this comment

imeoer Jan 6, 2022

Choose a reason for hiding this comment

ktock Jan 5, 2022

Choose a reason for hiding this comment

imeoer Jan 6, 2022

Choose a reason for hiding this comment

ktock Jan 5, 2022

Choose a reason for hiding this comment

imeoer Jan 6, 2022

Choose a reason for hiding this comment

imeoer commented Jan 6, 2022 • edited Loading

ktock commented Jan 11, 2022

imeoer commented Jan 12, 2022

ktock commented Jan 13, 2022

imeoer commented Jan 13, 2022 • edited Loading

imeoer commented Jan 19, 2022

ktock commented Jan 20, 2022

imeoer commented Jan 21, 2022 • edited Loading

imeoer commented Jan 24, 2022 • edited Loading

imeoer commented Jan 5, 2022 •

edited

Loading

imeoer commented Jan 6, 2022 •

edited

Loading

imeoer commented Jan 13, 2022 •

edited

Loading

imeoer commented Jan 21, 2022 •

edited

Loading

imeoer commented Jan 24, 2022 •

edited

Loading