ephemeral mode check #80568

pohly · 2019-07-25T08:34:10Z

What type of PR is this?
/kind api-change

What this PR does / why we need it:

This makes it safer to use ephemeral inline volumes (an alpha feature in 1.15) because it prevents accidental usage of a normal CSI driver for an ephemeral inline volume. The whitelisting of CSI drivers via a pod security policy would have the same effect, but might not be active in a cluster.

Special notes for your reviewer:

Which issue(s) this PR fixes:
Fixes: #79624
Related-to: #79983, which has been merged

Does this PR introduce a user-facing change?:

a CSI driver that supports ephemeral inline volumes must explicitly declare that by providing a CSIDriver object where the new "mode" field is set to "ephemeral" or "persistent+ephemeral"

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190122-csi-inline-volumes.md#driver-mode

pohly · 2019-07-25T08:35:52Z

/priority important-soon
/cc @vladimirvivien

cwdsuzhou · 2019-07-25T10:40:46Z

pkg/volume/csi/csi_plugin_test.go

+		}
+		driver := getTestCSIDriver(testDriver, nil, nil, &driverMode)
+		fakeClient := fakeclient.NewSimpleClientset(driver)
+		plug, tmpDir := newTestPlugin(t, fakeClient)
 		defer os.RemoveAll(tmpDir)



since we use defer in foreach，using t.Run is better

Right, good catch. Other tests already did that and I missed that this one doesn't. Will fix it.

Actually there was a t.Run, I just moved the code into the wrong spot. Fixed.

pohly · 2019-07-25T11:46:14Z

Before actually asking for an API review, I need to rebase and split the commit into smaller ones (just the API change, generated files, additional code and tests).

Right now I am just trying to find out what I might have missed (like updating some more generated files) and why the roundtrip test is failing.

vladimirvivien

Looking good, left comments for some minor issues.

vladimirvivien · 2019-07-25T14:23:16Z

pkg/volume/csi/csi_plugin.go

+// supportsVolumeMode looks up CSIDriver object associated with the
+// driver name to determine if the driver supports the given volume
+// mode. An error indicates that it isn't supported and explains why.
+func (p *csiPlugin) supportsVolumeMode(driver string, volumeMode csiVolumeMode) error {


This probably would be better as a isVolumeSuppoerted(driver, mode) (bool, error) Returns true if supported, false if not, and error if something went wrong while attempting to check.

I had that initially, but then there's no way to convey to the caller what the reason is when the mode isn't supported.

These two error message would have to be removed:
https://github.com/kubernetes/kubernetes/blob/1a14c40a538a6324f867420897d11b4378585941/pkg/volume/csi/csi_plugin.go#L812-L817

vladimirvivien · 2019-07-25T14:26:37Z

pkg/volume/csi/csi_plugin.go

+}
+
+// lookupDriverMode looks up CSIDriver object associated with the driver name and
+// returns what it finds there or the "persistent" default.


Please clarify in comment what the bool return param is for is for.

Will do. It tells the caller whether the returned string was explicitly set or is the default value. That's relevant when it comes to producing a good error message.

Would it be better to return the empty string for "not set" and drop the bool?

Would it be better to return the empty string for "not set" and drop the bool?

I changed it to that. @vladimirvivien: okay?

vladimirvivien · 2019-07-25T14:28:45Z

pkg/volume/csi/csi_plugin.go

+	if csiDriver.Spec.Mode == nil {
+		return storage.PersistentDriverMode, true, nil
+	}
+	return *csiDriver.Spec.Mode, false, nil


Should true be returned here since driver mode was set ?

vladimirvivien · 2019-07-25T14:29:31Z

pkg/volume/csi/csi_plugin.go

+		return "", false, err
+	}
+	if csiDriver.Spec.Mode == nil {
+		return storage.PersistentDriverMode, true, nil


Should false be returned since driver mode is not set. the boolean needs to be clarified I suppose.

msau42 · 2019-07-25T15:35:30Z

/assign @msau42 @jsafrane

msau42 · 2019-07-25T18:28:09Z

pkg/apis/storage/types.go

+	// - "persistent+ephemeral": both modes supported
+	// More modes may be added in the future.
+	// +optional
+	Mode *CSIDriverMode


Validation for this new field needs to be added in https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/storage/validation/validation.go

Agreed, will add this. I just wanted to fix the roundtrip error first before working on making the API nicer to use.

msau42 · 2019-07-25T18:32:23Z

pkg/apis/storage/v1beta1/defaults.go

@@ -47,4 +47,8 @@ func SetDefaults_CSIDriver(obj *storagev1beta1.CSIDriver) {
 		obj.Spec.PodInfoOnMount = new(bool)
 		*(obj.Spec.PodInfoOnMount) = false
 	}
+	if obj.Spec.Mode == nil {


I think this needs to be feature gated

We also need to drop the alpha field in PrepareForCreate, PrepareForUpdate calls in https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/storage/csidriver/strategy.go. Take a look at this example

Okay. But will this help with the roundtrip problem?

And which part of that example are the hand-written files? They all come from bb45b8e#diff-da9973f9d53ca756ac31a053eb90161c, which includes everything in a single commit.

the entire strategy.go is a handwritten file

To fix the roundtripping test, I think we need to modify the fuzzer

I added the feature gate check to defaults.go and strategy.go. With the feature disabled, the roundtrip test passes, but of course breaks again when enabling it by default.

To fix the roundtripping test, I think we need to modify the fuzzer

Yes, that did the trick.

I'm going to squash the following change into the commit:

diff --git a/pkg/apis/storage/fuzzer/fuzzer.go b/pkg/apis/storage/fuzzer/fuzzer.go index 2081ffe846..c205f2c4f7 100644 --- a/pkg/apis/storage/fuzzer/fuzzer.go +++ b/pkg/apis/storage/fuzzer/fuzzer.go @@ -46,6 +46,10 @@ var Funcs = func(codecs runtimeserializer.CodecFactory) []interface{} { obj.Spec.PodInfoOnMount = new(bool) *(obj.Spec.PodInfoOnMount) = false } + if obj.Spec.Mode == nil { + obj.Spec.Mode = new(storage.CSIDriverMode) + *obj.Spec.Mode = storage.PersistentDriverMode + } }, } } diff --git a/pkg/apis/storage/v1beta1/defaults.go b/pkg/apis/storage/v1beta1/defaults.go index 05bc6dbb3e..75acec10f3 100644 --- a/pkg/apis/storage/v1beta1/defaults.go +++ b/pkg/apis/storage/v1beta1/defaults.go @@ -20,6 +20,8 @@ import ( "k8s.io/api/core/v1" storagev1beta1 "k8s.io/api/storage/v1beta1" "k8s.io/apimachinery/pkg/runtime" + utilfeature "k8s.io/apiserver/pkg/util/feature" + "k8s.io/kubernetes/pkg/features" ) func addDefaultingFuncs(scheme *runtime.Scheme) error { @@ -47,7 +49,7 @@ func SetDefaults_CSIDriver(obj *storagev1beta1.CSIDriver) { obj.Spec.PodInfoOnMount = new(bool) *(obj.Spec.PodInfoOnMount) = false } - if obj.Spec.Mode == nil { + if obj.Spec.Mode == nil && utilfeature.DefaultFeatureGate.Enabled(features.CSIInlineVolume) { obj.Spec.Mode = new(storagev1beta1.CSIDriverMode) *(obj.Spec.Mode) = storagev1beta1.PersistentDriverMode } diff --git a/pkg/registry/storage/csidriver/strategy.go b/pkg/registry/storage/csidriver/strategy.go index 1f534b2e76..1b31a7b23a 100644 --- a/pkg/registry/storage/csidriver/strategy.go +++ b/pkg/registry/storage/csidriver/strategy.go @@ -22,9 +22,11 @@ import ( "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/apiserver/pkg/storage/names" + utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/apis/storage" "k8s.io/kubernetes/pkg/apis/storage/validation" + "k8s.io/kubernetes/pkg/features" ) // csiDriverStrategy implements behavior for CSIDriver objects @@ -41,8 +43,12 @@ func (csiDriverStrategy) NamespaceScoped() bool { return false } -// ResetBeforeCreate clears the Status field which is not allowed to be set by end users on creation. +// PrepareForCreate clears the Mode field if the corresponding feature is disabled. func (csiDriverStrategy) PrepareForCreate(ctx context.Context, obj runtime.Object) { + if !utilfeature.DefaultFeatureGate.Enabled(features.CSIInlineVolume) { + csiDriver := obj.(*storage.CSIDriver) + csiDriver.Spec.Mode = nil + } } func (csiDriverStrategy) Validate(ctx context.Context, obj runtime.Object) field.ErrorList { @@ -62,8 +68,12 @@ func (csiDriverStrategy) AllowCreateOnUpdate() bool { return false } -// PrepareForUpdate sets the Status fields which is not allowed to be set by an end user updating a CSIDriver +// PrepareForUpdate clears the Mode field if the corresponding feature is disabled. func (csiDriverStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) { + if !utilfeature.DefaultFeatureGate.Enabled(features.CSIInlineVolume) { + newCSIDriver := obj.(*storage.CSIDriver) + newCSIDriver.Spec.Mode = nil + } } func (csiDriverStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList {

msau42 · 2019-08-14T16:35:56Z

/lgtm

pohly · 2019-08-21T07:39:16Z

@smarterclayton: ping - is the API okay?

smarterclayton · 2019-08-21T11:16:14Z

Need to remove the validation gating as Jordan described.

kfox1111 · 2019-08-21T15:56:54Z

Is this the last thing needed to get ephemeral to beta?

msau42 · 2019-08-21T16:09:09Z

@kfox1111 feature gate needs to be flipped + docs

pohly · 2019-08-22T07:19:12Z

/retest

msau42 · 2019-08-22T16:02:43Z

/lgtm

pohly · 2019-08-22T21:24:22Z

@smarterclayton: sorry for the delay, I am on vacation and missed the new comments last week. The PR should now be as discussed.

smarterclayton · 2019-08-26T15:49:15Z

/approve

API change for alpha gated feature in 1.16 approved.

k8s-ci-robot · 2019-08-26T15:50:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msau42, pohly, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~api/OWNERS~~ [smarterclayton]
~~pkg/apis/OWNERS~~ [smarterclayton]
~~pkg/registry/OWNERS~~ [smarterclayton]
~~pkg/volume/csi/OWNERS~~ [msau42,smarterclayton]
~~staging/src/k8s.io/api/OWNERS~~ [smarterclayton]
~~test/e2e/storage/OWNERS~~ [msau42,smarterclayton]
~~test/e2e/testing-manifests/storage-csi/OWNERS~~ [msau42,smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This revised test changes one field at a time in various ways. This ensures that one failure reason doesn't mask the other. An incorrect comment also gets fixed. Suggested in kubernetes#80568 (review).

k8s-ci-robot requested review from davidz627 and deads2k July 25, 2019 08:35

k8s-ci-robot requested a review from vladimirvivien July 25, 2019 08:35

k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 25, 2019

cwdsuzhou reviewed Jul 25, 2019

View reviewed changes

pohly force-pushed the ephemeral-mode-check branch from df562ba to 4de69dc Compare July 25, 2019 14:21

pohly mentioned this pull request Jul 25, 2019

persistent and ephemeral csi volumes #79983

Merged

vladimirvivien reviewed Jul 25, 2019

View reviewed changes

pohly mentioned this pull request Jul 25, 2019

Remove external-attacher and add CSIDriver kubernetes-csi/csi-driver-host-path#78

Closed

pohly force-pushed the ephemeral-mode-check branch from 4de69dc to 1a14c40 Compare July 25, 2019 14:46

k8s-ci-robot assigned jsafrane and msau42 Jul 25, 2019

msau42 reviewed Jul 25, 2019

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 25, 2019

pohly force-pushed the ephemeral-mode-check branch from 1a14c40 to 16aa09b Compare July 26, 2019 10:10

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 14, 2019

bazel update

8270fe8

pohly force-pushed the ephemeral-mode-check branch from eef8a70 to 8270fe8 Compare August 22, 2019 06:59

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 22, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 22, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2019

k8s-ci-robot merged commit 087aafc into kubernetes:master Aug 26, 2019

k8s-ci-robot added this to the v1.16 milestone Aug 26, 2019

This was referenced Aug 26, 2019

storage: enhance test for ValidateCSIDriverUpdate #81961

Merged

ephemeral volume tests #81960

Merged

storage: fix CSIInlineVolume round-trip test #82012

Merged

liggitt moved this from Assigned to API review completed, 1.16 in API Reviews Aug 29, 2019

adisky mentioned this pull request Sep 3, 2019

Cinder CSI Inline volume support kubernetes/cloud-provider-openstack#719

Merged

pohly mentioned this pull request Feb 9, 2021

PSP ephemeral volume validation #98918

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ephemeral mode check #80568

ephemeral mode check #80568

pohly commented Jul 25, 2019 •

edited

pohly commented Jul 25, 2019

cwdsuzhou Jul 25, 2019

pohly Jul 25, 2019

pohly Jul 26, 2019

pohly commented Jul 25, 2019

vladimirvivien left a comment

vladimirvivien Jul 25, 2019

pohly Jul 25, 2019

vladimirvivien Jul 25, 2019

pohly Jul 25, 2019

pohly Jul 26, 2019

vladimirvivien Jul 25, 2019

vladimirvivien Jul 25, 2019

msau42 commented Jul 25, 2019

msau42 Jul 25, 2019

pohly Jul 25, 2019

pohly Jul 26, 2019

msau42 Jul 25, 2019

msau42 Jul 25, 2019

pohly Jul 25, 2019

pohly Jul 25, 2019

msau42 Jul 25, 2019

msau42 Jul 25, 2019

pohly Jul 26, 2019

msau42 commented Aug 14, 2019

pohly commented Aug 21, 2019

smarterclayton commented Aug 21, 2019

kfox1111 commented Aug 21, 2019

msau42 commented Aug 21, 2019

pohly commented Aug 22, 2019

msau42 commented Aug 22, 2019

pohly commented Aug 22, 2019

smarterclayton commented Aug 26, 2019

k8s-ci-robot commented Aug 26, 2019

ephemeral mode check #80568

ephemeral mode check #80568

Conversation

pohly commented Jul 25, 2019 • edited

pohly commented Jul 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pohly commented Jul 25, 2019

vladimirvivien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Jul 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Aug 14, 2019

pohly commented Aug 21, 2019

smarterclayton commented Aug 21, 2019

kfox1111 commented Aug 21, 2019

msau42 commented Aug 21, 2019

pohly commented Aug 22, 2019

msau42 commented Aug 22, 2019

pohly commented Aug 22, 2019

smarterclayton commented Aug 26, 2019

k8s-ci-robot commented Aug 26, 2019

pohly commented Jul 25, 2019 •

edited