oss: support CoCo direct volume feature #882

mozillazg · 2023-10-09T07:57:18Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

oss: support CoCo direct volume feature

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

https://github.com/kata-containers/kata-containers/blob/main/docs/design/direct-blk-device-assignment.md
kata-containers/kata-containers#7943
kata-containers/kata-containers#7965
confidential-containers/guest-components#345

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2023-10-09T07:57:26Z

Welcome @mozillazg!

It looks like this is your first PR to kubernetes-sigs/alibaba-cloud-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/alibaba-cloud-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2023-10-09T07:57:26Z

Hi @mozillazg. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mowangdk · 2023-10-09T08:12:28Z

pkg/utils/kata/directvolume/direct_volume.go

+
+// Remove deletes the direct volume path including all the files inside it.
+func Remove(volumePath string) error {
+	return os.RemoveAll(filepath.Join(kataDirectVolumeRootPath, b64.URLEncoding.EncodeToString([]byte(volumePath))))


Remove each file using os.Remove? It's safer.

huww98 · 2023-10-09T08:45:12Z

pkg/utils/kata/directvolume/direct_volume.go

+}
+
+// Add writes the mount info of a direct volume into a filesystem path known to Kata Container.
+func Add(volumePath string, mountInfo string) error {


pass mountInfo as []byte?

Add is only called from AddMountInfo. Combine them to one func?

This file is copied from https://github.com/kata-containers/kata-containers/blob/c800d0739fe42414a15ac8967f3fa5ccb98c28d7/src/runtime/pkg/direct-volume/utils.go.
We should keep it sync with upstream without any changes.

OK. then how about just importing it? That is a one file package.

We can't just import it. That will cause we depend on the whole kata runtime project.

I think that is not a big deal. We are using vendor mode, which will only introduce one go file into the vendor directory. But I've tried that, this requires several upgrades to other dependencies. So, it is a pity. Maybe we can import this after we sort out our go.mod, and upgrade our dependencies.

huww98 · 2023-10-09T08:45:44Z

pkg/utils/kata/directvolume/direct_volume.go

+	}
+
+	var deserialized MountInfo
+	if err := json.Unmarshal([]byte(mountInfo), &deserialized); err != nil {


Why unmarshal it?

huww98 · 2023-10-09T08:58:48Z

pkg/oss/nodeserver.go

+	if isDirectVolumePath(mountPoint) {
+		return ns.unPublishDirectVolume(ctx, req)
+	}



This is not idempotent. I suppose isDirectVolumePath would return false on a second unpublish? But it might be OK, because the remaining logic would still determine the volume as not mounted, then return success.

huww98 · 2023-10-09T09:06:25Z

pkg/utils/kata/directvolume/direct_volume.go

+	stat, err := os.Stat(volumeDir)
+	if err != nil {
+		if !errors.Is(err, os.ErrNotExist) {
+			return err
+		}
+		if err := os.MkdirAll(volumeDir, 0700); err != nil {
+			return err
+		}
+	}
+	if stat != nil && !stat.IsDir() {
+		return fmt.Errorf("%s should be a directory", volumeDir)
+	}
+


How about just call os.MkdirAll? It will do the Stat and the checks for you.

Suggested change

stat, err := os.Stat(volumeDir)

if err != nil {

if !errors.Is(err, os.ErrNotExist) {

return err

}

if err := os.MkdirAll(volumeDir, 0700); err != nil {

return err

}

}

if stat != nil && !stat.IsDir() {

return fmt.Errorf("%s should be a directory", volumeDir)

}

if err := os.MkdirAll(volumeDir, 0700); err != nil {

return err

}

huww98 · 2023-10-09T09:12:40Z

pkg/oss/coco.go

+		case optKmsKeyId:
+			kmsKeyId = strings.TrimSpace(value)
+		case optAnnotations:
+			json.Unmarshal([]byte(value), &annotationsObj)


Is it intentional to ignore the errors here?

Yes. This option is optional, and the invalid value can be ignored.

huww98 · 2023-10-09T09:19:21Z

pkg/oss/coco.go

+		Device:     device,
+		FsType:     fsType,
+		Metadata:   metadata,
+		Options:    strings.Fields(opt.OtherOpts),


by using strings.Fields(), we dropped the support for space in the options, e.g.: -o key="hello world" will not work.

I don't think the ossfs have such cases.

Maybe. But this breaks the current interface. We are parsing the options with shell now, e.g., -o 'key' is parsed as ["-o", "key"], not ["-o", "'key'"]

AlbeeSo · 2023-10-09T12:39:52Z

will the kata containers follow the ossfs version in CSI ? ACK ossfs 1.80.1 is the min version support KMS encryption.

mowangdk · 2023-10-09T13:05:59Z

/ok-to-test

mowangdk · 2023-10-13T07:03:58Z

/lgtm
/approve

k8s-ci-robot · 2023-10-13T07:04:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mowangdk, mozillazg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mowangdk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

oss: support CoCo direct volume feature

801d1a6

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 9, 2023

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 9, 2023

k8s-ci-robot requested a review from mowangdk October 9, 2023 07:57

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 9, 2023

mozillazg marked this pull request as ready for review October 9, 2023 08:22

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 9, 2023

mowangdk reviewed Oct 9, 2023

View reviewed changes

huww98 suggested changes Oct 9, 2023

View reviewed changes

huww98 reviewed Oct 9, 2023

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 9, 2023

k8s-ci-robot assigned mowangdk Oct 13, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 13, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 13, 2023

k8s-ci-robot merged commit 79369da into kubernetes-sigs:master Oct 13, 2023
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oss: support CoCo direct volume feature #882

oss: support CoCo direct volume feature #882

mozillazg commented Oct 9, 2023 •

edited

k8s-ci-robot commented Oct 9, 2023

k8s-ci-robot commented Oct 9, 2023

mowangdk Oct 9, 2023

huww98 Oct 9, 2023

mozillazg Oct 9, 2023

huww98 Oct 9, 2023

mozillazg Oct 9, 2023

huww98 Oct 10, 2023

huww98 Oct 9, 2023

huww98 Oct 9, 2023

huww98 Oct 9, 2023

mowangdk Oct 13, 2023

huww98 Oct 9, 2023

mozillazg Oct 9, 2023

huww98 Oct 9, 2023

mozillazg Oct 9, 2023

huww98 Oct 10, 2023

AlbeeSo commented Oct 9, 2023 •

edited

mowangdk commented Oct 9, 2023

mowangdk commented Oct 13, 2023

k8s-ci-robot commented Oct 13, 2023

oss: support CoCo direct volume feature #882

oss: support CoCo direct volume feature #882

Conversation

mozillazg commented Oct 9, 2023 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Oct 9, 2023

k8s-ci-robot commented Oct 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlbeeSo commented Oct 9, 2023 • edited

mowangdk commented Oct 9, 2023

mowangdk commented Oct 13, 2023

k8s-ci-robot commented Oct 13, 2023

mozillazg commented Oct 9, 2023 •

edited

AlbeeSo commented Oct 9, 2023 •

edited