WIP: CC Adding image pull on host with snapshotter

[stevenhorsman]

This feature builds upon the initial basic image pull work in #8103 as one of the possible next steps to extend it to support more of the current CCv0 feature set.

The scope of this is to add the ability to pull the image on the host via the nydus snapshotter using tarfs to share data from the host to the guest. There are two modes for this - image_block and image_block_with_verity which also adds integrity protection to the images that are shared to ensure that anyone with host access cannot modify them

Things to note/include with this work

• We want to port over the code from:
  • CC | support dm-verity tarfs disk image in CoCo #7688 (although some of it might have been merged over as dependencies on the pull on guest code in CC: Basic pull image on guest #8103)
  • CC | tools: Install dependencies with dm-verity in rootfs #7601
  • image-builder: fix udev error when using docker to build image #7880
• New tests
  • I think we are primarily focussed on regression tests to ensure that the things that work for pull on guest, also work for pull on host, but we might want to/be able to add some dm-verity integrity specific tests later?

Questions

• Should we split this up into two issues and address pull on host with tarfs and snapshotter first and then the integrity protection? @ChengyuZhu6

[ChengyuZhu6]

Should we split this up into two issues and address pull on host with tarfs and snapshotter first and then the integrity protection? @ChengyuZhu6

I like this idea @stevenhorsman. But the main issue is that the guest kernel used in the CoCo is blocking the tests of pulling on host, which I had created an issue to track(#8083). So I think we need to fix it first.

[ChengyuZhu6]

Now the following tasks need to be completed::

• prepare the workflow to deploy nydus snapshotter enabling host-share-with-block feature.
• Support for sharing images to guest with integrity check using virtio-blk.

[ChengyuZhu6]

I will outline various scenarios that this feature currently supports, as well as some challenges that need addressing. Thanks @wainersm for the nice summary table.

Case #	pull-type	fs_driver	export_mode	CoCo supported
1	guest-pull	proxy	N/A	Y
2	host-share-image-block	blockdev	image_block_with_verity	Y
3	host-share-image-block	blockdev	image_block	Y
4	host-share-image-block	blockdev	layer_block_with_verity	Y
5	host-share-image-block	blockdev	layer_block	Y

[ChengyuZhu6]

For guest-pull, we can support to pull images in the guest by image-rs with two image types: oci and nydus.

Image type	support encrypted images	support on-demand pull
OCI	Y	N
Nydus	Y	Y

[ChengyuZhu6]

For image host sharing, we offer two sharing types: per image (image_block/image_block_with_verity), per layer (layer_block/layer_block_with_verity). For each type, users have the option to either enable data integrity verification or disable it. Disabling integrity verification implies that the data shared with the guest by virtio-blk/virtio-scsi will lack dm-verity information.

All cases for image sharing could be enabled both on TEE and non-TEE platforms.

[ChengyuZhu6]

To deploy nydus snapshotter for image sharing on CI, I have submitted a patch to deploy it with image_block_with_verity by default : aac003c

[ChengyuZhu6]

Unfortunately, currently the snapshotter is limited to handling a single export_mode at any given time. Consequently, setting export_mode to image_block_with_verity precludes support for case 2, case 3 and case 4. 😢

[ChengyuZhu6]

It’s important to note that the processes of image_block_with_verity and image_block (layer_block_with_verity and layer_block) are fundamentally similar 572ea7f. The key difference is that image_block_with_verity/layer_block_with_verity calls dm-verity handler to mount the dm-device, whereas image_block/layer_block doesn't require this step. Based on this, it’s reasonable to assume that if case 2 (or case 4) is working, case 3 (or case 5) should also work as expected.

[ChengyuZhu6]

One concern I have is whether we should create a new matrix of jobs to run image sharing tests for four sharing types, or simply introduce new columns to the current k8s jobs matrix (such as .github/workflows/run-kata-coco-tests.yaml). If we opt for the latter, is it reasonable to test all sharing types, or should we limit the tests to some of them to reduce resources? I'm concerned that the CI might become unstable due to issue #8407 if we frequently switch the snapshotter between different modes, although I'm not certain this will occur.

[ChengyuZhu6]

cc @stevenhorsman @fidencio @fitzthum @wainersm

[stevenhorsman]

This is a tricky one. On one hand my general principle is that we only say that things are support which we have CI tests for, so that would suggest that we should be testing all four export_modes (image and layer without and without verity checking). I think for non-TEE where we are running on AKS clusters that we can spin up and delete after testing this should be fine (though run-kata-coco-tests.yaml doesn't currently test on non-TEE, but I think @fidencio mentioned that @wainersm had some suggestions on this?).

The more concerning bit, as you pointed out, is for the TEE platforms, which run on bare-metal, so we can't just throw away the clusters and configuration after each run and this means that we need to be more careful about the snapshotter switching bugs in containerd. Maybe to reduce this we could test all four modes on non-TEE, but only the verity modes on the TEEs? I'm not sure that massively helps us though.

As a wildcard, I've seen that containerd 2.0 is in rc now, so has anyone checked to see if the snapshotter issues and/or image transfer service have made it in as stable yet as I'd hate us to do a lot of work to get around issues that we could simplify just by bumping to a newer containerd in the near future?

[ChengyuZhu6]

As a wildcard, I've seen that containerd 2.0 is in rc now, so has anyone checked to see if the snapshotter issues and/or image transfer service have made it in as stable yet as I'd hate us to do a lot of work to get around issues that we could simplify just by bumping to a newer containerd in the near future?

I’ll check the latest updates and get back to you on that.

[ChengyuZhu6]

As a wildcard, I've seen that containerd 2.0 is in rc now, so has anyone checked to see if the snapshotter issues and/or image transfer service have made it in as stable yet as I'd hate us to do a lot of work to get around issues that we could simplify just by bumping to a newer containerd in the near future?

I’ll check the latest updates and get back to you on that.

Good news, the related patches to support per runtime class image management are merged both on containerd and kubernetes.
kubernetes:kubernetes/kubernetes#121456 -> kubernetes 1.29.0
containerd: containerd/containerd#9815 -> containerd 2.0

Bad news, the related patches are hard to backport to containerd 1.6.x or containerd 1.7.x, and kubernetes 1.24.0 which is the kubernetes version used on CI.

[stevenhorsman]

Good news, the related patches to support per runtime class image management are merged both on containerd and kubernetes. kubernetes:kubernetes/kubernetes#121456 -> kubernetes 1.29.0 containerd: containerd/containerd#9815 -> containerd 2.0

Bad news, the related patches are hard to backport to containerd 1.6.x or containerd 1.7.x, and kubernetes 1.24.0 which is the kubernetes version used on CI.

Great - thanks for looking into this. So my opinion is that Kubernetes 1.24 is 2 years and 5 release old, so I have no issues in bumping our CI to use that as a when needed (though we'd need to be clear of the dependency for this feature to users). containerd is tricker, but I think given that several features we want to use are just experimental in 1.7, it is inevitable that we'll have to depend on containerd 2.0 in the near future, once we've validated that everything works as we expect.