CSI volume expansion #74863

gnufied · 2019-03-03T21:40:50Z

Add support for CSI volume expansion.

Added tests for both online and offline expansion of CSI volumes.

Note to reviewers

This PR uses 1.1RC branch of CSI spec
~~Uses https://quay.io/repository/gnufied/external-resizer repo for pulling external-resizer.~~

/sig storage
cc @msau42 @jsafrane @mlmhl

Add support for node side CSI volume expansion

pkg/volume/csi/expander.go

gnufied · 2019-03-04T21:25:57Z

/kind feature
/priority important-soon

pkg/volume/util/operationexecutor/operation_generator.go

msau42 · 2019-03-04T22:57:55Z

pkg/volume/csi/expander.go

+)
+
+func (c *csiPlugin) RequiresFSResize() bool {
+	// We could check plugin's node capability but we instead are going to rely on


This is different from how we implement skip attach. Can we be consistent in how we check a plugin's capability?

But CanAttach check depends on CSIDriver field too right? Whereas this check is more static. I do not like changing the behaviour of FindByXXX function fwiw depending on feature tags and other dynamic things.

maybe we should consider changing skipAttach then (separately) cc @jsafrane

pkg/volume/csi/expander.go

msau42 · 2019-03-04T23:14:13Z

pkg/volume/util/operationexecutor/operation_generator.go


-	fsResizeFunc := func() (error, error) {
-		_, resizeError := og.resizeFileSystem(volumeToMount, volumeToMount.DevicePath, deviceMountPath, volumePlugin.GetPluginName())
+		if attachableVolumePlugin != nil {


Are device mountable plugins supported here? (doesn't support attach but does support global device mount)

could use some unit tests here

The only volume type that is not attachable but implements device mountable interface is local storage. So for in-tree volumes this check should be good.

Also if CSI plugin is not attachable but supports node expansion, it should be resized after node publish and hence we should be covered there too.

msau42 · 2019-03-04T23:17:51Z

test/e2e/testing-manifests/storage-csi/external-resizer/rbac.yaml

+  name: external-resizer-cfg
+rules:
+- apiGroups: [""]
+  resources: ["endpoints"]


we're trying to get all new components to use the Lease mechanism for leader election, not endpoints. cc @andrewsykim

We don't use this rbac permission in e2e btw because our e2es run without leader election in external-resizer. The same applies to other sidecar containers too. I have kept it here just to have same version between here and external-resizer repo.

test/e2e/storage/csi_mock_volume.go

thockin · 2019-03-07T20:56:52Z

The vast majority of changes here are pkg/volume or test/e2e/storage - i think it's flyable.

vendor change LGTM

saad-ali

There are few changes to common code. The majority of functional changes are adding functionality to the CSI package and that functionality flag gated and off by default since this is an alpha feature.

/lgtm
/approve

saad-ali · 2019-03-07T21:33:46Z

pkg/volume/util/operationexecutor/operation_generator.go

@@ -676,15 +700,15 @@ func (og *operationGenerator) GenerateMountVolumeFunc(
 	}
 }

-func (og *operationGenerator) resizeFileSystem(volumeToMount VolumeToMount, devicePath, deviceMountPath, pluginName string) (simpleErr, detailedErr error) {
+func (og *operationGenerator) resizeFileSystem(volumeToMount VolumeToMount, rsOpts volume.NodeResizeOptions, pluginName string) (bool, error) {


nit: put a comment explaining what the expected values are that are returned here.

I will follow up this in a follow up.

msau42 · 2019-03-07T22:18:30Z

/hold cancel
This feature is already planned and tracked for 1.14

gnufied · 2019-03-07T22:20:23Z

/assign @smarterclayton

smarterclayton · 2019-03-07T22:46:46Z

approving vendor change as per sig and other leads

/approve

k8s-ci-robot · 2019-03-07T22:47:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied, msau42, saad-ali, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~Godeps/OWNERS~~ [smarterclayton]
~~pkg/features/OWNERS~~ [saad-ali,smarterclayton]
~~pkg/kubelet/volumemanager/OWNERS~~ [saad-ali,smarterclayton]
~~pkg/volume/OWNERS~~ [gnufied,msau42,saad-ali,smarterclayton]
~~test/e2e/storage/OWNERS~~ [gnufied,msau42,saad-ali,smarterclayton]
~~test/e2e/testing-manifests/storage-csi/OWNERS~~ [gnufied,msau42,saad-ali,smarterclayton]
~~vendor/OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Handle resize error in online resizing Use NodeExpandable plugin to mark volumes that require node expansion

Update CSI library version

This enables us to pass CSI volume phase

Add some tests for checking node expansion Add new tests for expander

gnufied · 2019-03-08T14:26:48Z

I had to rebase because of a conflict. @jsafrane @tsmetana can you add lgtm again?

jsafrane · 2019-03-08T14:30:13Z

/lgtm

tedyu · 2019-03-08T18:25:23Z

pkg/volume/csi/csi_client.go

+	}
+
+	if newSize.Value() < 0 {
+		return newSize, errors.New("size can not be less than 0")


It would be helpful to include the negative value in the message

tedyu · 2019-03-08T18:26:23Z

pkg/volume/csi/csi_client.go

+	if err != nil {
+		return newSize, err
+	}
+	updatedQuantity := resource.NewQuantity(resp.CapacityBytes, resource.BinarySI)


Do we need to check that updatedQuantity is not nil?

tedyu · 2019-03-08T18:28:40Z

pkg/volume/csi/csi_client.go

+		}
+		for _, capability := range capabilities {
+			if capability.GetRpc().GetType() == csipbv1.NodeServiceCapability_RPC_EXPAND_VOLUME {
+				nodeExpandSet = true


We can break out of the loop here

tedyu · 2019-03-08T18:37:13Z

Created #75203 for the last comment above.

This cherry-picks the [74863](kubernetes#74863) and allows us to use CSI volume expansion This involves the following commits: ``` 63eb25e Introduce feature gate for volume expansion 89d1de9 Rename ExandFS to NodeExpand 529cd71 Move resize function parameters to a new ResizeOptions type a8f3187 Add CSI volume resizing tests 1bd9ed0 Update csi library version ```

k8s-ci-robot requested review from anguslees and brendandburns March 3, 2019 21:41

This was referenced Mar 3, 2019

Add e2e for resizing of flexvolumes that don't implement attach/detach #74864

Closed

Flexvolume volume resizing doesn't work #72131

Closed

mlmhl reviewed Mar 4, 2019

View reviewed changes

pkg/volume/csi/expander.go Outdated Show resolved Hide resolved

mlmhl reviewed Mar 4, 2019

View reviewed changes

pkg/volume/csi/expander.go Outdated Show resolved Hide resolved

gnufied mentioned this pull request Mar 4, 2019

When checking for pod info version, make volume attachable #74811

Merged

gnufied force-pushed the csi-volume-expansion branch from 6f3b3cc to 5c502ee Compare March 4, 2019 21:22

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 4, 2019

gnufied mentioned this pull request Mar 4, 2019

Add resizing support to CSI volumes kubernetes/enhancements#556

Open

gnufied changed the title ~~Csi volume expansion~~ CSI volume expansion Mar 4, 2019

gnufied force-pushed the csi-volume-expansion branch from 5c502ee to 159b6b9 Compare March 4, 2019 22:20

msau42 reviewed Mar 4, 2019

View reviewed changes

saad-ali approved these changes Mar 7, 2019

View reviewed changes

k8s-ci-robot assigned saad-ali Mar 7, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 7, 2019

k8s-ci-robot assigned smarterclayton Mar 7, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 7, 2019

gnufied added 5 commits March 8, 2019 09:21

Rename ExandFS to NodeExpand

89d1de9

Handle resize error in online resizing Use NodeExpandable plugin to mark volumes that require node expansion

Introduce feature gate for volume expansion

63eb25e

Update CSI library version

Move resize function parameters to a new ResizeOptions type

529cd71

This enables us to pass CSI volume phase

Add CSI volume resizing tests

a8f3187

Add some tests for checking node expansion Add new tests for expander

Update csi library version

1bd9ed0

gnufied force-pushed the csi-volume-expansion branch from 4582623 to 1bd9ed0 Compare March 8, 2019 14:23

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 8, 2019

k8s-ci-robot assigned jsafrane Mar 8, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 8, 2019

k8s-ci-robot merged commit 3624c74 into kubernetes:master Mar 8, 2019

gnufied mentioned this pull request Mar 8, 2019

Add support for NodeExpand kubernetes-csi/external-resizer#10

Closed

tedyu reviewed Mar 8, 2019

View reviewed changes

adisky mentioned this pull request Mar 26, 2019

Add Support for Cinder CSI Volume Resize kubernetes/cloud-provider-openstack#565

Closed

chrischdi mentioned this pull request Jun 26, 2019

Cherry Pick PR-74863 CSI Volume expansion c445/kubernetes#1

Closed

nabokihms mentioned this pull request Aug 30, 2019

ceph: Add dynamic flexvolume expansion rook/rook#3417

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CSI volume expansion #74863

CSI volume expansion #74863

gnufied commented Mar 3, 2019 •

edited

Loading

gnufied commented Mar 4, 2019

msau42 Mar 4, 2019

gnufied Mar 5, 2019 •

edited

Loading

msau42 Mar 5, 2019

msau42 Mar 4, 2019

msau42 Mar 4, 2019

gnufied Mar 5, 2019

gnufied Mar 5, 2019

msau42 Mar 4, 2019

gnufied Mar 5, 2019

thockin commented Mar 7, 2019

saad-ali left a comment

saad-ali Mar 7, 2019

gnufied Mar 7, 2019

msau42 commented Mar 7, 2019

gnufied commented Mar 7, 2019

smarterclayton commented Mar 7, 2019

k8s-ci-robot commented Mar 7, 2019

gnufied commented Mar 8, 2019

jsafrane commented Mar 8, 2019

tedyu Mar 8, 2019

tedyu Mar 8, 2019

tedyu Mar 8, 2019

tedyu commented Mar 8, 2019

CSI volume expansion #74863

CSI volume expansion #74863

Conversation

gnufied commented Mar 3, 2019 • edited Loading

gnufied commented Mar 4, 2019

Choose a reason for hiding this comment

gnufied Mar 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Mar 7, 2019

saad-ali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 commented Mar 7, 2019

gnufied commented Mar 7, 2019

smarterclayton commented Mar 7, 2019

k8s-ci-robot commented Mar 7, 2019

gnufied commented Mar 8, 2019

jsafrane commented Mar 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tedyu commented Mar 8, 2019

gnufied commented Mar 3, 2019 •

edited

Loading

gnufied Mar 5, 2019 •

edited

Loading