Add support to resize Portworx volume #62308

harsh-px · 2018-04-10T01:05:19Z

Signed-off-by: Harsh Desai harsh@portworx.com

What this PR does / why we need it:

This PR adds support in the Portworx volume plugin to expand an existing PVC.

Which issue(s) this PR fixes:

Release note:

Add support to resize Portworx volumes.

lpabon

LGTM

dixudx · 2018-04-10T07:55:30Z

pkg/volume/portworx/portworx.go

+	spec *volume.Spec,
+	newSize resource.Quantity,
+	oldSize resource.Quantity) (resource.Quantity, error) {
+	glog.Infof("Expanding: %s from %v to %v", spec.Name(), oldSize, newSize)


using level 4 log?

Switched to level 4 logging.

dixudx · 2018-04-10T07:56:22Z

pkg/volume/portworx/portworx.go

+		return oldSize, err
+	}
+
+	glog.Infof("Successfully resized %s to %v", spec.Name(), newSize)


Fixed in latest incremental

dixudx · 2018-04-10T08:00:50Z

pkg/volume/portworx/portworx_util.go

@@ -38,6 +41,7 @@ const (
 	pvcNamespaceLabel = "namespace"
 	pxServiceName     = "portworx-service"
 	pxDriverName      = "pxd-sched"
+	oneGig            = 1024 * 1024 * 1024


use oneGi instead?

Removed this and now using existing functions in https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/util.go

dixudx · 2018-04-10T08:05:28Z

pkg/volume/portworx/portworx_util.go

+		return fmt.Errorf("failed to inspect Portworx volume: %s. Found: %d volumes", spec.Name(), len(vols))
+	}
+
+	vol := vols[0]


only the first volume will be resized?

Yes, because we inspected one volume above. So only one volume is expected is the response. Line 200 also explicitly makes that check. ResizeVolume is a per-volume call.

gnufied · 2018-04-10T17:30:49Z

pkg/volume/portworx/portworx_util.go

+
+func capacityToGB(capacity resource.Quantity) int {
+	return int(volutil.RoundUpSize(capacity.Value(), oneGig))
+}


there is already https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/util.go#L372

Thanks. Fixed.

gnufied · 2018-04-10T17:31:19Z

/ok-to-test

harsh-px · 2018-04-12T18:30:23Z

@dixudx / @gnufied I have fixed the review comments and the tests are passing.

dixudx · 2018-04-13T01:24:07Z

@harsh-px Would you please rebase your commits. Thanks.

I'll lgtm.

harsh-px · 2018-04-13T05:41:44Z

@dixudx done.

dixudx · 2018-04-16T03:22:19Z

/lgtm

ping @msau42 @jsafrane for approval. Thanks.
/cc @kubernetes/sig-storage-api-reviews

harsh-px · 2018-04-16T17:48:47Z

@deads2k and @gnufied can you approve?

gnufied · 2018-04-17T13:01:43Z

pkg/volume/portworx/portworx_util.go

@@ -56,7 +59,7 @@ func (util *PortworxVolumeUtil) CreateVolume(p *portworxVolumeProvisioner) (stri

 	capacity := p.options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]
 	// Portworx Volumes are specified in GB


Why does this comment says Portworx volumes are specified in GB and then code proceeds to use GiB ?

Fixed in latest incremental.

gnufied · 2018-04-17T13:04:22Z

pkg/volume/portworx/portworx.go

+	newSize resource.Quantity,
+	oldSize resource.Quantity) (resource.Quantity, error) {
+	glog.V(4).Infof("Expanding: %s from %v to %v", spec.Name(), oldSize, newSize)
+	err := plugin.util.ResizeVolume(spec, newSize, plugin.host)


This function should be idempotent. If portworx volume already has the user requested size, ExpandVolumeDevice should be a no-op followed by a successful response. Please see other volume plugins as an example.

@gnufied the Portworx driver implementation on the server side is already idempotent. So invoking it multiple times will have the same effect as it internally checks if the volume has the size. So I don't need any logic here on the client side. Best to keep it simple.

There is another reason for keeping this idempotent in Kubernetes rather than in Portworx server side. Lets say:

User created a PVC of 5GB size and is matched with a PV which is of 7GiB size.

Later on User edits the PVC to be 6GiB. Kuberenetes may still call ExpandVolumeDevice even though underlying PV already matches the requested size.

What happens in portworx server side now? Will it shrink existing 7GiB disk to 6GiB or will it return an error?

Obviously shrinking will be bad. We never shrink volumes in Kubernetes. Returning an error will be bad too. Even if theoretically Portworx in Server side does the "right thing" and returns successful response - there is no guarantee that the behaviour will remain the same always. That is why - it is important to not rely on Portworx behaviour in server side.

@gnufied good point ! Updated the review with check to see if existing volume is already equal or greater than requested size.

gnufied · 2018-04-17T13:06:38Z

pkg/volume/portworx/portworx_util.go

+
+	vol := vols[0]
+	vol.Spec.Size = uint64(volutil.RoundUpToGiB(newSize))
+	return driver.Set(spec.Name(), vol.Locator, vol.Spec)


Is there a way to get size after resizing? The reason most volume plugins returns "new size" from this function is because - after alignment to GB or GiB boundaries, actual volume size may be different than user requested size(but as long as it is more than user requested size we are fine)

Done. Added check in the latest incremental if the size actually got updated.

gnufied · 2018-04-17T13:11:16Z

Nearly there. Please make sure plugin uses one unit throughout the code (GB or GiB). Also - ideally ExpandVolumeDevice should be idempotent, if somehow we can't do that for portworx then the idempotency should be handled by portworx itself. Under no circumstance - if user requested 15GB space and volume is of >=15GB size, an error should be thrown from ExpandVolumeDevice function.

gnufied · 2018-04-19T17:17:21Z

pkg/volume/portworx/portworx_util.go

+	}
+
+	updatedVol := vols[0]
+	if updatedVol.Spec.Size != vol.Spec.Size {


I don't think we need to check exact equality, may be just updatedVol.Size >= vol.Size should be sufficient?

Done. Now I'm allowing the updated volume to be greater than the requested size.

harsh-px · 2018-04-19T19:07:29Z

/retest

gnufied · 2018-04-19T19:12:41Z

@harsh-px please squash the commits.

Closes kubernetes#62305 Signed-off-by: Harsh Desai <harsh@portworx.com> update comment and variable references to GiB Signed-off-by: Harsh Desai <harsh@portworx.com> explicitly check volume size after resize and fix size volume spec Signed-off-by: Harsh Desai <harsh@portworx.com> If Portworx volume is already greater than new size, skip resize Signed-off-by: Harsh Desai <harsh@portworx.com> Allow updated volume to be greater than requested size Signed-off-by: Harsh Desai <harsh@portworx.com>

harsh-px · 2018-04-19T19:59:55Z

@gnufied Squashed the changes. It seems the tests for failing due to an automation issue.

ERROR: (gcloud.auth.activate-service-account) There was a problem refreshing your current auth tokens: Unable to find the server at accounts.google.com

I'm going to retry these.

/retest

harsh-px · 2018-04-19T20:00:18Z

/retest

gnufied · 2018-04-19T20:13:03Z

Looks good to me. Assinging @deads2k for approval

/lgtm
/assign @deads2k

deads2k · 2018-04-19T20:21:45Z

/approve

approving the admission changes.

k8s-ci-robot · 2018-04-19T20:22:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, dixudx, gnufied, harsh-px

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/volume/OWNERS~~ [gnufied]
~~plugin/pkg/admission/OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-04-19T21:50:15Z

Automatic merge from submit-queue (batch tested with PRs 59592, 62308, 62523, 62635, 62243). If you want to cherry-pick this change to another branch, please follow the instructions here.

gnufied · 2018-07-10T15:54:28Z

@harsh-px I was just reviewing some of the stuff for a blog post and noticed that Portworx still returns false from RequiresFSReisze function. Is that an oversight? Does this needs fixing. I don't know much about portworx.. cc @lpabon

k8s-ci-robot requested review from davidz627 and derekwaynecarr April 10, 2018 01:05

lpabon approved these changes Apr 10, 2018

View reviewed changes

dixudx reviewed Apr 10, 2018

View reviewed changes

gnufied reviewed Apr 10, 2018

View reviewed changes

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 10, 2018

harsh-px force-pushed the px-resize-master branch from 09925b3 to 5487761 Compare April 11, 2018 23:25

harsh-px force-pushed the px-resize-master branch from cda50e9 to 5e57210 Compare April 13, 2018 01:30

k8s-ci-robot assigned dixudx Apr 16, 2018

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. lgtm "Looks good to me", indicates that a PR is ready to be merged. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Apr 16, 2018

gnufied reviewed Apr 17, 2018

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 17, 2018

harsh-px force-pushed the px-resize-master branch 2 times, most recently from fa522a9 to 7753d1a Compare April 19, 2018 16:49

gnufied reviewed Apr 19, 2018

View reviewed changes

harsh-px force-pushed the px-resize-master branch from b587a5c to adc7185 Compare April 19, 2018 19:15

k8s-ci-robot assigned gnufied and deads2k Apr 19, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 19, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 19, 2018

k8s-github-robot merged commit 607edb8 into kubernetes:master Apr 19, 2018

This was referenced Apr 19, 2018

Automated cherry pick of #62308: Add support to resize Portworx volume #62882

Closed

Automated cherry pick of #62308 #62884

Closed

		@@ -56,7 +59,7 @@ func (util PortworxVolumeUtil) CreateVolume(p portworxVolumeProvisioner) (stri

		capacity := p.options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]
		// Portworx Volumes are specified in GB

Add support to resize Portworx volume #62308

Add support to resize Portworx volume #62308

Conversation

harsh-px commented Apr 10, 2018

lpabon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Apr 10, 2018

harsh-px commented Apr 12, 2018

dixudx commented Apr 13, 2018

harsh-px commented Apr 13, 2018

dixudx commented Apr 16, 2018

harsh-px commented Apr 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Apr 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

harsh-px commented Apr 19, 2018

gnufied commented Apr 19, 2018

harsh-px commented Apr 19, 2018

harsh-px commented Apr 19, 2018

gnufied commented Apr 19, 2018

deads2k commented Apr 19, 2018

k8s-ci-robot commented Apr 19, 2018

k8s-github-robot commented Apr 19, 2018

gnufied commented Jul 10, 2018