StorageOS Volume Plugin #42156

croomes · 2017-02-27T12:59:08Z

What this PR does / why we need it:
This PR adds a new volume plugin for StorageOS volumes. StorageOS runs as a container on Kubelet nodes, aggregating local or attached storage and making its capacity available to all nodes within the cluster. More information at http://storageos.com.

The StorageOS plugin supports:

Dynamic Provisioning using Storage Classes
Persistent Volumes and Persistent Volume Claims.

Which issue this PR fixes
A feature request has been created:
kubernetes/enhancements#190

This isn't on the schedule for 1.6 as I wasn't sure when it would be ready. We intend to make the StorageOS container openly available within the 1.6 lifetime.

Special notes for your reviewer:

Separate commits for feature and godep changes.

Release note:

StorageOS Volume Driver
[StorageOS](http://www.storageos.com) can be used as a storage provider for Kubernetes.  With StorageOS, capacity from local or attached storage is pooled across the cluster, providing converged infrastructure for cloud-native applications.

k8s-ci-robot · 2017-02-27T12:59:10Z

Hi @croomes. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-reviewable · 2017-02-27T12:59:15Z

This change is

croomes · 2017-04-07T13:53:30Z

I'll rebase this as soon as I get back from DockerCon

rootfs · 2017-04-27T16:30:11Z

/assign

rootfs · 2017-04-27T16:30:27Z

@k8s-bot ok to test

rootfs · 2017-04-27T16:56:46Z

pkg/api/types.go

+	// namespace is specified, the namespace of the pod will be used.  Set to
+	// "default" if you are not using namespaces within StorageOS.
+	// +optional
+	Namespace string


why do you need NameSpace in volume source?

We use namespaces within StorageOS too, so by default it will use the Pod's namespace but you can force it not to use StorageOS namespace's at all by setting to "default". You can also specify a different StorageOS namespace here. Normally the StorageOS namespaces will mirror Kubernetes'.

rootfs · 2017-04-27T16:59:44Z

pkg/volume/storageos/doc.go

@@ -0,0 +1,19 @@
+/*
+Copyright 2015 The Kubernetes Authors.


rootfs · 2017-04-27T16:59:52Z

pkg/volume/storageos/storageos_test.go

@@ -0,0 +1,333 @@
+/*
+Copyright 2016 The Kubernetes Authors.


rootfs · 2017-04-27T17:07:12Z

pkg/volume/storageos/storageos.go

+			podUID:          pod.UID,
+			pvName:          spec.Name(),
+			volName:         source.VolumeName,
+			namespace:       source.Namespace,


you can get namespace from Pod pod.Namespace

As comment above, this is the volume namespace and could be different from the Pod namespace.

rootfs · 2017-04-27T17:09:16Z

pkg/volume/storageos/storageos.go

+		storageos: &storageos{
+			pvName:    spec.Name(),
+			volName:   spec.PersistentVolume.Spec.StorageOS.VolumeName,
+			namespace: spec.PersistentVolume.Spec.StorageOS.Namespace,


you can get namespace from PV's ClaimRef

This is the volume's namespace, not necessarily the Pod's. Starting to think I should rename it volNamespace to make it more clear - what do you think?

rootfs · 2017-04-27T17:14:08Z

pkg/volume/storageos/storageos.go

+
+	// Set parameters from StorageClass
+	if len(v.options.Parameters) > 0 {
+		if description, ok := v.options.Parameters["description"]; ok {


iterate parameters and use switch/case

much better - done

rootfs · 2017-04-27T17:16:35Z

pkg/volume/storageos/storageos.go

+	v.volName = v.options.PVName
+	v.namespace = v.options.PVC.Namespace
+	v.labels = make(map[string]string)
+	for k, val := range v.options.PVC.Labels {


why setting labels here?

We copy the labels from the PVC to the requested volume. StorageOS uses the labels from k8s to set policy. Fox example, if the PVC has environment=production set, a policy rule within StorageOS might set the number of replicas to 2.

rootfs · 2017-04-27T17:20:02Z

pkg/volume/storageos/storageos.go

+			pv.Labels = make(map[string]string)
+		}
+		for k, v := range vol.Labels {
+			pv.Labels[k] = v


passing labels from PVC to PV?

This is setting labels created by StorageOS when the volume is provisioned and applying them to the PV. So in the example above, the label storageos.feature.replicas=2 would be set on the PV. These labels are only informational.

croomes · 2017-05-02T17:45:42Z

Hi @rootfs - thanks for the review. I'm updating and will reply to the remaining points soon.

liggitt · 2017-06-07T18:52:58Z

plugin/pkg/auth/authorizer/node/graph.go

@@ -251,7 +251,7 @@ func (g *Graph) AddPV(pv *api.PersistentVolume) {

 		// since we don't know the other end of the pvc -> pod -> node chain (or it may not even exist yet), we can't decorate these edges with kubernetes node info
 		g.graph.SetEdge(simple.Edge{F: pvVertex, T: g.getOrCreateVertex_locked(pvcVertexType, pv.Spec.ClaimRef.Namespace, pv.Spec.ClaimRef.Name)})
-		pvutil.VisitPVSecretNames(pv, func(secret string) bool {
+		pvutil.VisitPVSecretNames(pv, func(namespace, secret string) bool {
 			// This grants access to the named secret in the same namespace as the bound PVC
 			g.graph.SetEdge(simple.Edge{F: g.getOrCreateVertex_locked(secretVertexType, pv.Spec.ClaimRef.Namespace, secret), T: pvVertex})


use the provided namespace here instead of pv.Spec.ClaimRef.Namespace

liggitt · 2017-06-07T19:25:41Z

pkg/api/persistentvolume/util_test.go

 	for _, pv := range pvs {
-		VisitPVSecretNames(pv, func(name string) bool {
+		VisitPVSecretNames(pv, func(namespace, name string) bool {
+			fmt.Printf("%s,%s\n", namespace, name)


drop the printf

whoops - done

liggitt · 2017-06-07T19:57:59Z

pkg/volume/storageos/storageos.go

+
+func (plugin *storageosPlugin) NewMounter(spec *volume.Spec, pod *v1.Pod, _ volume.VolumeOptions) (volume.Mounter, error) {
+
+	source, _, err := getPersistentVolumeSource(spec)


doesn't this mean the storageos plugin would only support persistentvolumes, not a storageos volume source specified in a pod spec?

No - this is a mistake. I added the code to detect both in newMounterInternal but not NewMounter (my tests were failing on it). I will refactor a bit and add to both. I couldn't think of a better way to handle both cases.

yeah, this is part of the ripple of not being able to share a single volume source type :-/

liggitt · 2017-06-07T20:00:29Z

pkg/volume/storageos/storageos.go

+	}
+
+	var apiCfg *storageosAPIConfig
+	if source.SecretRef != nil {


I expected two paths here, one using secret name/namespace for a PV source, one using secret name and pod namespace for a pod source

actually, you can do this inside newMounterInternal, where you already have two branches

saad-ali · 2017-06-07T21:13:09Z

@dchen1107 held the v1.7 exceptions meeting today. She left it up to @liggitt and I to decide if this PR gets in for 1.7 or not.

@liggitt and I just had a meeting to discuss the current revision of this PR where the VolumeSource for this plugin is split between a PV (with ObjectReference) and non-PV (with LocalObjectReference) VolumeSource. We agree that while this is different from existing plugins, it is necessary to handle the unique access pattern for StorageOS--where they have a global credential that is required for provisioning/mounting/unmounting/deleting. For the dynamically provisioned volumes both secret namespace and name are specified, but since the PV object is non-namespaced, it can only be created with some one with admin privileges so that is ok. In this case the secret can live in a non-user namespace and therefore even if the user deletes their entire namespace, the volume will still be deleted correctly. For direct volume references, the secret must live in the users namespace (deletion is not a worry).

So overall we are fine with this approach. @liggitt and I will both take a final pass reviewing this PR. In the meantime, @croomes please test all the paths for this code (provisioning, mounting, unmounting, deleting, etc.). Once we have a thumbs up from you on the testing front and no further feedback, we will approve/LGTM this for merge (Friday at the latest).

croomes · 2017-06-07T21:29:39Z

Fantastic, thanks! I'm going to do another push in 30 mins that should fix the last comments, but I will spend the day tomorrow double-checking everything. I've re-run our basic acceptance tests and they pass for PVs and volumes in Pod specs and dynamic provisioning. I need to do more with the various secret namespace combinations.

To help review latest, I've only changed NewMounter and newMounterInternal, and changed validation.go to return proper path to secretRef name/namespace: fldPath.Child("secretRef", "namespace")

Really appreciate all the time you've both put into helping me get this in. 👍

liggitt · 2017-06-07T20:33:20Z

pkg/volume/storageos/storageos.go

+	}
+
+	var apiCfg *storageosAPIConfig
+	if source.SecretRef != nil {


actually, you can do this inside newMounterInternal, where you already have two branches

liggitt · 2017-06-08T06:37:25Z

pkg/volume/storageos/storageos.go

+	capacity := c.options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]
+	c.sizeGB = int(volume.RoundUpSize(capacity.Value(), 1024*1024*1024))
+
+	if adminSecretName == "" {


|| adminSecretNamespace == ""?

actually, does requiring this mean you can no longer use default storageos credentials?

liggitt · 2017-06-08T06:49:04Z

pkg/volume/storageos/storageos.go

+
+// parseSecret will lookup secret in secretNamespace if set, otherwise default
+// to pod namespace.
+func parseSecret(secretNamespace, secretName string, pod *v1.Pod, kubeClient clientset.Interface) (*storageosAPIConfig, error) {


I don't think parseSecret() should exist... we should either call parsePodSecret() or parsePVSecret() directly, based on whether the source of the data was a PV or a Pod

liggitt · 2017-06-08T06:56:32Z

pkg/volume/storageos/storageos.go

+
+func (plugin *storageosPlugin) NewMounter(spec *volume.Spec, pod *v1.Pod, _ volume.VolumeOptions) (volume.Mounter, error) {
+
+	secretNamespace, secretName, err := getSecretInfoFromSpec(spec)


rather than spreading this logic between three functions, and inferring the source was a PV because namespace == "" in parseSecret, it would be much more straightforward to combine getSecretInfoFromSpec and parseSecret

func getAPICfg(spec *volume.Spec, pod *v1.Pod, kubeClient clientset.Interface) (*storageosAPIConfig, error) { if spec.PersistentVolume != nil { source, _, err := getPersistentVolumeSource(spec) if err != nil { return nil, err } if source.SecretRef == nil { // not sure whether you require apiCfg in this case // would you expect to fall back to default values, or return an error? return nil, nil } return parsePVSecret(source.SecretRef.Namespace, source.SecretRef.Name, kubeClient) } if spec.Volume != nil { source, _, err := getVolumeSource(spec) if err != nil { return nil, err } if source.SecretRef == nil { // not sure whether you require apiCfg in this case // would you expect to fall back to default values, or return an error? return nil, nil } return parsePodSecret(pod, source.SecretRef.Name, kubeClient) } return nil, fmt.Errorf("spec not Volume or PersistentVolume") }

liggitt · 2017-06-08T13:52:16Z

pkg/api/persistentvolume/util.go

@@ -20,34 +20,46 @@ import (
 	"k8s.io/kubernetes/pkg/api"
 )

+func getNamespace(pv *api.PersistentVolume) string {


nit: rename this to getClaimRefNamespace

liggitt · 2017-06-08T13:53:32Z

pkg/api/validation/validation.go

+	if len(storageos.VolumeNamespace) > 0 {
+		allErrs = append(allErrs, ValidateDNS1123Label(storageos.VolumeNamespace, fldPath.Child("volumeNamespace"))...)
+	}
+	return allErrs


if secretRef != nil, ensure a name is set?

liggitt · 2017-06-08T13:54:07Z

pkg/api/validation/validation.go

+	if len(storageos.VolumeNamespace) > 0 {
+		allErrs = append(allErrs, ValidateDNS1123Label(storageos.VolumeNamespace, fldPath.Child("volumeNamespace"))...)
+	}
+	if storageos.SecretRef != nil && (len(storageos.SecretRef.Name) > 0 || len(storageos.SecretRef.Namespace) > 0) {


if secretRef != nil, I'd expect the name and namespace to be set... a non-nil secretRef with empty data seems wrong

liggitt · 2017-06-08T16:02:08Z

one comment about combining getSecretInfoFromSpec and parseSecret to make it very clear that parsePVSecret() is only called with data from a PV, and a couple validation questions.

the graph and pv secret util changes LGTM

croomes · 2017-06-08T21:49:32Z

I've made the changes suggested by @liggitt (good ideas), and we've done a lot of testing. Functionally we're happy, but I want to make one final pass on the examples and README to make sure we didn't miss anything. I'll have it rebased and ready to go for start of day (PT) Friday.

croomes · 2017-06-09T14:17:48Z

@saad-ali / @liggitt - we're good from our side. A few minor updates to the docs and we've re-tested everything and it all looks fine.

saad-ali

/lgtm
/approve

saad-ali · 2017-06-09T18:40:46Z

pkg/api/types.go

+	// StorageOS represents a StorageOS volume that is attached to the kubelet's host machine and mounted into the pod
+	// More info: https://releases.k8s.io/HEAD/examples/volumes/storageos/README.md
+	// +optional
+	StorageOS *StorageOSPersistentVolumeSource


nit: Maybe add a comment indicating why it is not StorageOSVolumeSource.

croomes · 2017-06-09T19:22:47Z

👍 It's like another election night - the team is in the pub waiting for the votes to come in. Big cheer from over here - thanks.

k8s-github-robot · 2017-06-10T01:25:03Z

Automatic merge from submit-queue

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 27, 2017

k8s-github-robot assigned jbeda Feb 27, 2017

croomes changed the title ~~Storageos Volume Plugin~~ StorageOS Volume Plugin Feb 27, 2017

croomes mentioned this pull request Feb 27, 2017

StorageOS Volume Plugin kubernetes/enhancements#190

Closed

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2017

grodrigues3 added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 5, 2017

croomes force-pushed the storageos branch from f8ca096 to a27b9c6 Compare April 27, 2017 14:27

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 27, 2017

croomes force-pushed the storageos branch from a27b9c6 to c9bf54a Compare April 27, 2017 15:16

k8s-ci-robot assigned rootfs Apr 27, 2017

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 27, 2017

rootfs reviewed Apr 27, 2017

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 28, 2017

croomes force-pushed the storageos branch from c9bf54a to 5e3952f Compare May 9, 2017 15:24

croomes force-pushed the storageos branch 2 times, most recently from f8e244a to dda964f Compare June 7, 2017 17:55

liggitt reviewed Jun 7, 2017

View reviewed changes

croomes force-pushed the storageos branch from dda964f to 8e545a0 Compare June 7, 2017 19:51

liggitt reviewed Jun 7, 2017

View reviewed changes

croomes force-pushed the storageos branch from 8e545a0 to e274e0f Compare June 7, 2017 22:01

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 8, 2017

liggitt reviewed Jun 8, 2017

View reviewed changes

croomes force-pushed the storageos branch from e274e0f to 5bf6fbb Compare June 8, 2017 21:44

Add StorageOS volume plugin

5e2503e

croomes force-pushed the storageos branch from 5bf6fbb to 5e2503e Compare June 9, 2017 12:33

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 9, 2017

saad-ali approved these changes Jun 9, 2017

View reviewed changes

saad-ali added this to the v1.7 milestone Jun 9, 2017

saad-ali added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2017

k8s-github-robot merged commit 9df40c3 into kubernetes:master Jun 10, 2017

croomes mentioned this pull request Jun 15, 2017

Storageos Volume Plugin kubernetes/website#4095

Merged

liggitt mentioned this pull request Jun 16, 2017

add secret namespace to azure file #47660

Merged


		func (plugin storageosPlugin) NewMounter(spec volume.Spec, pod *v1.Pod, _ volume.VolumeOptions) (volume.Mounter, error) {

		source, _, err := getPersistentVolumeSource(spec)


		func (plugin storageosPlugin) NewMounter(spec volume.Spec, pod *v1.Pod, _ volume.VolumeOptions) (volume.Mounter, error) {

		secretNamespace, secretName, err := getSecretInfoFromSpec(spec)

StorageOS Volume Plugin #42156

StorageOS Volume Plugin #42156

Conversation

croomes commented Feb 27, 2017 • edited Loading

k8s-ci-robot commented Feb 27, 2017

k8s-reviewable commented Feb 27, 2017

croomes commented Apr 7, 2017

rootfs commented Apr 27, 2017

rootfs commented Apr 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

croomes commented May 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saad-ali commented Jun 7, 2017

croomes commented Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Jun 8, 2017

croomes commented Jun 8, 2017

croomes commented Jun 9, 2017

saad-ali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

croomes commented Jun 9, 2017

k8s-github-robot commented Jun 10, 2017

croomes commented Feb 27, 2017 •

edited

Loading

liggitt Jun 7, 2017 •

edited

Loading

liggitt Jun 7, 2017 •

edited

Loading

croomes commented Jun 7, 2017 •

edited

Loading