Adding credentials support for k8s core CSI #60118

sbezverk · 2018-02-21T04:14:23Z

PR implements changes proposed in: kubernetes/community#1816

CSI now allows credentials to be specified on CreateVolume/DeleteVolume, ControllerPublishVolume/ControllerUnpublishVolume, and NodePublishVolume/NodeUnpublishVolume operations

sbezverk · 2018-02-21T04:14:52Z

/sig storage
/kind feature

sbezverk · 2018-02-21T22:50:50Z

@vladimirvivien @saad-ali Please review.
This implementation adds two new objects and modifies code only for NodePublish and NodeUnpublish, since these operations are executed by csi_mounter.go and csi_client.go
Changes for ControllerCreate/Delete/Publish/Unpublish are outside of k8s core and will be done after merging this change.

sbezverk · 2018-02-22T20:25:48Z

@saad-ali addressed changes in the proposal, PTAL.

saad-ali · 2018-02-22T22:58:04Z

pkg/apis/core/types.go

+	// ControllerPublishVolume and ControllerUnpublishVolume calls.
+	// This may be empty if no secret is required. If the secret object contains
+	// more than one secret, all secrets are passed.
+	// +optional


nit: Update comment

// ControllerPublishSecretRef is a reference to the secret object containing // sensitive information to pass to the CSI driver to complete the CSI // ControllerPublishVolume and ControllerUnpublishVolume calls. // This field is optional, and may be empty if no secret is required. If the // secret object contains more than one secret, all secrets are passed. // +optional

saad-ali · 2018-02-22T23:03:43Z

pkg/apis/core/types.go

+	// NodeStageSecretRef is a reference to the secret object containing sensitive
+	// information to pass to the CSI driver to complete the CSI NodeStageVolume
+	// and NodeStageVolume and NodeUnstageVolume calls.
+	// This secret will be fetched by the kubelet.


You can remove this line.

saad-ali · 2018-02-22T23:03:51Z

pkg/apis/core/types.go

+	// NodePublishSecretRef is a reference to the secret object containing
+	// sensitive information to pass to the CSI driver to complete the CSI
+	// NodePublishVolume and NodeUnpublishVolume calls.
+	// This secret will be fetched by the kubelet.


You can remove this line.

saad-ali · 2018-02-22T23:09:44Z

pkg/volume/csi/csi_util.go

+
+func getCredentialsFromSecret(k8s kubernetes.Interface, sn, ns string) map[string]string {
+	credentials := map[string]string{}
+	if len(sn) == 0 || len(ns) == 0 {


I think empty namespace maybe a valid case (default namespace?).

done, in reality it will not be ever empty as I take it from pod's spec. Removed ns check.

saad-ali · 2018-02-22T23:16:30Z

pkg/volume/csi/csi_mounter.go

@@ -195,7 +195,12 @@ func (c *csiMountMgr) SetUpAt(dir string, fsGroup *int64) error {
 	if len(fsType) == 0 {
 		fsType = defaultFSType
 	}
-
+	nodePublishCredentials := map[string]string{}


Not your code, but, can you add a commit to remove probe related code in this PR too:

... if err := csi.NodeProbe(ctx, csiVersion); err != nil { ...

and all related comments, e.g.:

// TODO (vladimirvivien) use this method to probe controller using CSI.NodeProbe() call

saad-ali · 2018-02-22T23:21:05Z

pkg/volume/csi/csi_mounter.go

-
+	nodePublishCredentials := map[string]string{}
+	if c.spec.PersistentVolume.Spec.CSI.NodePublishSecretRef != nil {
+		sn := c.spec.PersistentVolume.Spec.CSI.NodePublishSecretRef.Name


Just use csiSource.NodePublishSecretRef.Name everywhere instead.

saad-ali · 2018-02-22T23:22:32Z

pkg/volume/csi/csi_mounter.go

@@ -195,7 +195,12 @@ func (c *csiMountMgr) SetUpAt(dir string, fsGroup *int64) error {
 	if len(fsType) == 0 {
 		fsType = defaultFSType
 	}
-
+	nodePublishCredentials := map[string]string{}
+	if c.spec.PersistentVolume.Spec.CSI.NodePublishSecretRef != nil {


I'm not 100% sure, but I think an empty namespace is valid. So maybe just check if NodePublishSecretRef.Name != nil

Here I check for the presence of Secret Object, Name is string, it could be not nil but "". If you do not mind I would like to keep this check for object not nil and secret name validation in getCredentialsFromSecret

sbezverk · 2018-02-23T01:36:29Z

@saad-ali addressed your comments, PTAL.

saad-ali · 2018-02-23T02:03:41Z

I forgot one more set of changes: The Kubernetes API server's node authorizer must be updated to allow kubelet to access the secrets referenced by CSIPersistentVolumeSource.

saad-ali · 2018-02-23T02:10:02Z

I forgot one more set of changes: The Kubernetes API server's node authorizer must be updated to allow kubelet to access the secrets referenced by CSIPersistentVolumeSource.

Actually I think this might be handled by VisitPVSecretNames.
@liggitt to confirm and review.

/assign @liggitt

saad-ali · 2018-02-23T02:55:22Z

verify gofmt is failing, please run gofmt

liggitt · 2018-02-23T14:01:54Z

I forgot one more set of changes: The Kubernetes API server's node authorizer must be updated to allow kubelet to access the secrets referenced by CSIPersistentVolumeSource.

Actually I think this might be handled by VisitPVSecretNames.
@liggitt to confirm and review.

Close, though the node authorizer today allows the kubelet to read any secret referenced by a PV. As of this PR, only some secret references should be kubelet-visible (specifically, ControllerPublishSecretRef should not be). You can make that a parameter to the visitor (func(namespace, name string, kubeletVisible bool)) and omit appropriately in the node authorizer visitor

liggitt · 2018-02-23T14:33:21Z

pkg/api/persistentvolume/util.go

@@ -111,6 +111,32 @@ func VisitPVSecretNames(pv *api.PersistentVolume, visitor Visitor) bool {
 		if source.StorageOS.SecretRef != nil && !visitor(source.StorageOS.SecretRef.Namespace, source.StorageOS.SecretRef.Name) {
 			return false
 		}
+	case source.CSI != nil:
+		if source.CSI.ControllerPublishSecretRef != nil {
+			if len(source.CSI.ControllerPublishSecretRef.Name) == 0 || len(source.CSI.ControllerPublishSecretRef.Namespace) == 0 {


api validation should be added to prevent these from being empty... no need to recheck here (or in the other blocks)

@liggitt could you please point me where the validation you mentioned should be added.

kubernetes/pkg/apis/core/validation/validation.go

Lines 1432 to 1456 in 82eeda0

func validateCSIPersistentVolumeSource(csi *core.CSIPersistentVolumeSource, fldPath *field.Path) field.ErrorList {

allErrs := field.ErrorList{}

if !utilfeature.DefaultFeatureGate.Enabled(features.CSIPersistentVolume) {

allErrs = append(allErrs, field.Forbidden(fldPath, "CSIPersistentVolume disabled by feature-gate"))

}

if len(csi.Driver) == 0 {

allErrs = append(allErrs, field.Required(fldPath.Child("driver"), ""))

}

if len(csi.Driver) > 63 {

allErrs = append(allErrs, field.TooLong(fldPath.Child("driver"), csi.Driver, 63))

}

if !csiDriverNameRexp.MatchString(csi.Driver) {

allErrs = append(allErrs, field.Invalid(fldPath.Child("driver"), csi.Driver, validation.RegexError(csiDriverNameRexpErrMsg, csiDriverNameRexpFmt, "csi-hostpath")))

}

if len(csi.VolumeHandle) == 0 {

allErrs = append(allErrs, field.Required(fldPath.Child("volumeHandle"), ""))

}

return allErrs

}

sbezverk · 2018-02-23T17:01:07Z

/test pull-kubernetes-unit

liggitt · 2018-02-23T17:23:29Z

pkg/apis/core/validation/validation.go

@@ -1452,6 +1452,33 @@ func validateCSIPersistentVolumeSource(csi *core.CSIPersistentVolumeSource, fldP
 		allErrs = append(allErrs, field.Required(fldPath.Child("volumeHandle"), ""))
 	}

+	if csi.ControllerPublishSecretRef != nil {


for all of these, ensure name is a valid secret name and namespace is a valid namespace name

liggitt · 2018-02-23T17:25:38Z

pkg/volume/csi/csi_mounter.go

+	nodePublishCredentials := map[string]string{}
+	if csiSource.NodePublishSecretRef != nil {
+		sn := csiSource.NodePublishSecretRef.Name
+		ns := c.pod.ObjectMeta.Namespace


this should be csiSource.NodePublishSecretRef.Namespace

to protect against this type of mistake, consider passing the secret ref to getCredentialsFromSecret instead of loose name/namespace fields

liggitt · 2018-02-23T17:25:51Z

pkg/volume/csi/csi_mounter.go

+	nodeUnpublishCredentials := map[string]string{}
+	if csiSource.NodePublishSecretRef != nil {
+		sn := csiSource.NodePublishSecretRef.Name
+		ns := c.pod.ObjectMeta.Namespace


NodePublishSecretRef.Namespace

Will be passing SecreteReference to getCredentialsFromSecret.

sbezverk · 2018-02-23T19:46:36Z

@saad-ali @liggitt addressed latest comments, PTAL

saad-ali

LGTM mod nit

@liggitt for node authorizer review

saad-ali · 2018-02-23T20:19:09Z

pkg/api/persistentvolume/util.go

@@ -40,11 +40,11 @@ func VisitPVSecretNames(pv *api.PersistentVolume, visitor Visitor) bool {
 	switch {
 	case source.AzureFile != nil:
 		if source.AzureFile.SecretNamespace != nil && len(*source.AzureFile.SecretNamespace) > 0 {
-			if len(source.AzureFile.SecretName) > 0 && !visitor(*source.AzureFile.SecretNamespace, source.AzureFile.SecretName) {
+			if len(source.AzureFile.SecretName) > 0 && !visitor(*source.AzureFile.SecretNamespace, source.AzureFile.SecretName, true) {


nit: when it is unclear what the value means, it's nice to have a comment, for example:

!visitor( *source.AzureFile.SecretNamespace, source.AzureFile.SecretName, true /* kubeletVisible */) {

But it's a personal preference, up to you if you want to do that.

agree, it is more next developer friendly :)

liggitt · 2018-02-23T20:39:36Z

pkg/volume/csi/csi_util.go

+	credentials := map[string]string{}
+	secret, err := k8s.CoreV1().Secrets(secretRef.Namespace).Get(secretRef.Name, meta.GetOptions{})
+	if err != nil {
+		glog.Warningf("failed to find the secret %s in the namespace %s with error: %v\n", secretRef.Name, secretRef.Namespace, err)


I expected this to return the error and avoid making the CSI call

I am not sure, credentials are not mandatory, so if not specified I do not think we should fail CSI call. @saad-ali what do you think?

if they were specified by the PV, then it seems like an error resolving them should be considered an error in input gathering and short-circuit

liggitt · 2018-02-23T20:40:35Z

pkg/volume/csi/csi_mounter.go

-
+	nodePublishCredentials := map[string]string{}
+	if csiSource.NodePublishSecretRef != nil {
+		nodePublishCredentials = getCredentialsFromSecret(c.k8s, csiSource.NodePublishSecretRef)


here and below, I expected to handle an error return from getCredentialsFromSecret by short-circuiting and returning the error, rather than making the csi call with an empty credentials map

liggitt · 2018-02-23T20:41:33Z

API and node authorizer bits LGTM
one comment on error handling on secret lookup

liggitt · 2018-02-23T20:41:41Z

/approve

saad-ali

/lgtm
/approve

saad-ali · 2018-02-23T22:13:09Z

For vendored dep update:
/assign @thockin

brendandburns · 2018-02-23T23:16:57Z

/approve

k8s-ci-robot · 2018-02-23T23:17:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brendandburns, liggitt, saad-ali, sbezverk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~Godeps/OWNERS~~ [brendandburns]
~~api/OWNERS~~ [brendandburns,liggitt]
~~docs/api-reference/OWNERS~~ [brendandburns,liggitt]
~~pkg/api/OWNERS~~ [brendandburns,liggitt]
~~pkg/apis/core/OWNERS~~ [brendandburns,liggitt]
~~pkg/volume/csi/OWNERS~~ [brendandburns,saad-ali]
~~plugin/pkg/auth/authorizer/node/OWNERS~~ [brendandburns,liggitt]
~~staging/src/k8s.io/api/OWNERS~~ [brendandburns,liggitt]
~~vendor/OWNERS~~ [brendandburns]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-02-24T19:16:09Z

/test all [submit-queue is verifying that this PR is safe to merge]

sbezverk · 2018-02-24T20:06:37Z

/test pull-kubernetes-unit

k8s-github-robot · 2018-02-24T20:36:11Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 21, 2018

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. kind/feature Categorizes issue or PR as related to a new feature. labels Feb 21, 2018

k8s-ci-robot requested review from david-mcmahon, gnufied, lavalamp, nikhiljindal, thockin, wojtek-t and yifan-gu February 21, 2018 04:14

k8s-github-robot added the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Feb 21, 2018

david-mcmahon removed their request for review February 21, 2018 22:59

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 21, 2018

saad-ali self-assigned this Feb 22, 2018

saad-ali requested review from saad-ali and removed request for lavalamp, nikhiljindal and wojtek-t February 22, 2018 19:05

saad-ali reviewed Feb 22, 2018

View reviewed changes

k8s-ci-robot assigned liggitt Feb 23, 2018

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2018

liggitt reviewed Feb 23, 2018

View reviewed changes

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2018

liggitt reviewed Feb 23, 2018

View reviewed changes

saad-ali reviewed Feb 23, 2018

View reviewed changes

liggitt reviewed Feb 23, 2018

View reviewed changes

sbezverk added 3 commits February 23, 2018 16:50

api changes

8b09d4e

k8s csi code change

a6ca466

autogenerated api changes

3aa462e

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 23, 2018

saad-ali approved these changes Feb 23, 2018

View reviewed changes

k8s-ci-robot assigned thockin Feb 23, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 23, 2018

k8s-github-robot merged commit 8e8601a into kubernetes:master Feb 24, 2018

saad-ali mentioned this pull request Feb 25, 2018

CSI MountDevice/UnmountDevice Implementation #60115

Merged

sbezverk deleted the csi_core_credentials branch March 20, 2018 21:27

	func validateCSIPersistentVolumeSource(csi core.CSIPersistentVolumeSource, fldPath field.Path) field.ErrorList {
	allErrs := field.ErrorList{}

	if !utilfeature.DefaultFeatureGate.Enabled(features.CSIPersistentVolume) {
	allErrs = append(allErrs, field.Forbidden(fldPath, "CSIPersistentVolume disabled by feature-gate"))
	}

	if len(csi.Driver) == 0 {
	allErrs = append(allErrs, field.Required(fldPath.Child("driver"), ""))
	}

	if len(csi.Driver) > 63 {
	allErrs = append(allErrs, field.TooLong(fldPath.Child("driver"), csi.Driver, 63))
	}

	if !csiDriverNameRexp.MatchString(csi.Driver) {
	allErrs = append(allErrs, field.Invalid(fldPath.Child("driver"), csi.Driver, validation.RegexError(csiDriverNameRexpErrMsg, csiDriverNameRexpFmt, "csi-hostpath")))
	}

	if len(csi.VolumeHandle) == 0 {
	allErrs = append(allErrs, field.Required(fldPath.Child("volumeHandle"), ""))
	}

	return allErrs
	}

Adding credentials support for k8s core CSI #60118

Adding credentials support for k8s core CSI #60118

Conversation

sbezverk commented Feb 21, 2018 • edited Loading

sbezverk commented Feb 21, 2018

sbezverk commented Feb 21, 2018

sbezverk commented Feb 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbezverk commented Feb 23, 2018

saad-ali commented Feb 23, 2018

saad-ali commented Feb 23, 2018

saad-ali commented Feb 23, 2018

liggitt commented Feb 23, 2018

liggitt Feb 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbezverk commented Feb 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbezverk commented Feb 23, 2018

saad-ali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Feb 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Feb 23, 2018

liggitt commented Feb 23, 2018

saad-ali left a comment

Choose a reason for hiding this comment

saad-ali commented Feb 23, 2018

brendandburns commented Feb 23, 2018

k8s-ci-robot commented Feb 23, 2018

k8s-github-robot commented Feb 24, 2018

sbezverk commented Feb 24, 2018

k8s-github-robot commented Feb 24, 2018

sbezverk commented Feb 21, 2018 •

edited

Loading

liggitt Feb 23, 2018 •

edited

Loading

liggitt Feb 23, 2018 •

edited

Loading