Only retrieve relevant volumes #41785

jamiehannaford · 2017-02-21T11:05:15Z

What this PR does / why we need it:

Improves performance for Cinder volume attach/detach calls.

Currently when Cinder volumes are attached or detached, functions try to retrieve details about the volume from the Nova API. Because some only have the volume name not its UUID, they use the list function in gophercloud to iterate over all volumes to find a match. This incurs severe performance problems on OpenStack projects with lots of volumes (sometimes thousands) since it needs to send a new request when the current page does not contain a match. A better way of doing this is use the ?name=XXX query parameter to refine the results.

Which issue this PR fixes:

#26404

Special notes for your reviewer:

There were 2 ways of addressing this problem:

Use the name query parameter
Instead of using the list function, switch to using volume UUIDs and use the GET function instead. You'd need to change the signature of a few functions though, such as DeleteVolume, so I'm not sure how backwards compatible that is.

Since #1 does effectively the same as #2, I went with it because it ensures BC.

One assumption that is made is that the volumeName being retrieved matches exactly the name of the volume in Cinder. I'm not sure how accurate that is, but I see no reason why cloud providers would want to append/prefix things arbitrarily.

Release note:

Improves performance of Cinder volume attach/detach operations

k8s-ci-robot · 2017-02-21T11:05:16Z

Hi @jamiehannaford. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-reviewable · 2017-02-21T11:06:22Z

This change is

jingxu97 · 2017-02-22T06:44:47Z

pkg/cloudprovider/providers/openstack/openstack_volumes.go

@@ -115,7 +115,8 @@ func (os *OpenStack) getVolume(diskName string) (volumes.Volume, error) {
 		return volume, err
 	}

-	err = volumes.List(sClient, nil).EachPage(func(page pagination.Page) (bool, error) {
+	opts := &volumes.ListOpts{Name: diskName}
+	err = volumes.List(sClient, opts).EachPage(func(page pagination.Page) (bool, error) {


What is the difference of using opts vs. nil?

The Cinder API allows you to provide query parameters in the URL when retrieving a list of volumes. Filtering by name for example allows you to restrict the amount of volumes down considerably. Otherwise, the K8s code will traverse the entire collection of volumes in an OpenStack project to find a match (some project could have 10s of thousands). Filtering by name reduces/eliminates the need for subsequent pagination calls.

jamiehannaford · 2017-03-06T10:42:25Z

Any status update for this?

jamiehannaford · 2017-03-14T14:33:23Z

@k8s-sig-openstack-pr-reviews Would somebody mind reviewing this?

xsgordon · 2017-03-14T20:26:50Z

@idvoretskyi can you add the sig/openstack label please?

hogepodge · 2017-03-14T22:36:41Z

lgtm

jamiehannaford · 2017-04-03T09:05:16Z

@hogepodge Would you mind re-running tests? There seems to be a flaky test case failing

jamiehannaford · 2017-04-12T11:26:02Z

@jingxu97 @xsgordon @grodrigues3 @k8s-sig-openstack-api-reviews Would somebody mind commenting with ok to test so I can fix flaky tests?

jingxu97 · 2017-04-12T20:43:25Z

/approve

jingxu97 · 2017-04-12T20:44:12Z

@k8s-bot ok to test

jamiehannaford · 2017-04-25T10:18:59Z

@jingxu97 @anguslees Would somebody mind commenting with /approve and /lgtm?

anguslees · 2017-05-01T01:48:02Z

pkg/cloudprovider/providers/openstack/openstack_volumes.go

@@ -111,7 +111,9 @@ func (volumes *VolumesV2) createVolume(opts VolumeCreateOpts) (string, error) {
 func (volumes *VolumesV1) getVolume(diskName string) (Volume, error) {
 	var volume_v1 volumes_v1.Volume
 	var volume Volume
-	err := volumes_v1.List(volumes.blockstorage, nil).EachPage(func(page pagination.Page) (bool, error) {
+
+	opts := &volumes_v1.ListOpts{Name: diskName}


The logic below looks for either name == diskName or the ID contains diskName. That latter case sounds amazingly terrible, but are we confident we don't need it? (and if so, can we remove it in this PR to make it clear that that logic isn't going to trigger anymore?)

anguslees · 2017-05-01T01:49:03Z

pkg/cloudprovider/providers/openstack/openstack_volumes.go

@@ -149,7 +151,9 @@ func (volumes *VolumesV1) getVolume(diskName string) (Volume, error) {
 func (volumes *VolumesV2) getVolume(diskName string) (Volume, error) {
 	var volume_v2 volumes_v2.Volume
 	var volume Volume
-	err := volumes_v2.List(volumes.blockstorage, nil).EachPage(func(page pagination.Page) (bool, error) {
+
+	opts := &volumes_v2.ListOpts{Name: diskName}


Ditto - clarify the "ID contains diskName" case below

jamiehannaford · 2017-05-02T13:33:52Z

@anguslees I've refactored a bunch of signatures to make the volume ID stuff clearer. I don't why it was implemented to handle both names and IDs, but I've switched it to the latter. It also seems that the current volume implementation uses IDs too, so I think this is backwards compatible.

jamiehannaford · 2017-05-09T19:48:13Z

@mikedanese Would you mind weighing in here and if necessary approving this change?

jsafrane · 2017-05-10T08:49:56Z

Since the behaviour change interacted with this ID/name confusion, the PR seeks to also simplify it. I don't see a reason why this PR can't do both.

I am not against the PR, I am trying to find out what it tries to do, as it is quite a long one and I do not want to miss some important spot hidden between renames.

So far it looks good, please rebase and I'll mark it as lgtm.

jamiehannaford · 2017-05-16T10:39:02Z

@jsafrane Thanks for the feedback. I've made those changes - would you mind taking another quick look and adding /lgtm?

jsafrane · 2017-05-16T11:21:15Z

/lgtm

jamiehannaford · 2017-05-16T11:38:32Z

@mikedanese Can you /approve this PR?

mikedanese · 2017-05-16T17:54:31Z

/approve

jamiehannaford · 2017-05-19T09:17:49Z

@jsafrane Can you add /lgtm again? sorry for the pestering 😛

jamiehannaford · 2017-05-22T10:06:47Z

Bump cc/ @jsafrane

jsafrane · 2017-05-24T12:01:41Z

/lgtm

k8s-github-robot · 2017-05-24T12:01:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jamiehannaford, jingxu97, jsafrane, mikedanese

Needs approval from an approver in each of these OWNERS Files:

~~pkg/cloudprovider/OWNERS~~ [mikedanese]
~~pkg/volume/cinder/OWNERS~~ [jsafrane]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-05-24T13:46:58Z

Automatic merge from submit-queue (batch tested with PRs 38505, 41785, 46315)

k8s-ci-robot · 2017-05-24T14:03:02Z

@jamiehannaford: The following test(s) failed:

Test name	Commit	Details	Rerun command
pull-kubernetes-federation-e2e-gce	`4bd71a3`	link	`@k8s-bot pull-kubernetes-federation-e2e-gce test this`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…mance Automatic merge from submit-queue (batch tested with PRs 38505, 41785, 46315) Only retrieve relevant volumes **What this PR does / why we need it**: Improves performance for Cinder volume attach/detach calls. Currently when Cinder volumes are attached or detached, functions try to retrieve details about the volume from the Nova API. Because some only have the volume name not its UUID, they use the list function in gophercloud to iterate over all volumes to find a match. This incurs severe performance problems on OpenStack projects with lots of volumes (sometimes thousands) since it needs to send a new request when the current page does not contain a match. A better way of doing this is use the `?name=XXX` query parameter to refine the results. **Which issue this PR fixes**: kubernetes#26404 **Special notes for your reviewer**: There were 2 ways of addressing this problem: 1. Use the `name` query parameter 2. Instead of using the list function, switch to using volume UUIDs and use the GET function instead. You'd need to change the signature of a few functions though, such as [`DeleteVolume`](https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cinder/cinder.go#L49), so I'm not sure how backwards compatible that is. Since #1 does effectively the same as #2, I went with it because it ensures BC. One assumption that is made is that the `volumeName` being retrieved matches exactly the name of the volume in Cinder. I'm not sure how accurate that is, but I see no reason why cloud providers would want to append/prefix things arbitrarily. **Release note**: ```release-note Improves performance of Cinder volume attach/detach operations ```

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 21, 2017

k8s-github-robot assigned thockin Feb 21, 2017

k8s-github-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Feb 21, 2017

thockin assigned jingxu97 and unassigned thockin Feb 21, 2017

jingxu97 reviewed Feb 22, 2017

View reviewed changes

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 30, 2017

jamiehannaford force-pushed the cinder-performance branch from aa14ff0 to 343ce66 Compare March 31, 2017 09:30

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 31, 2017

grodrigues3 added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 5, 2017

jingxu97 removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 12, 2017

jamiehannaford force-pushed the cinder-performance branch from 343ce66 to 85be278 Compare April 13, 2017 10:17

anguslees suggested changes May 1, 2017

View reviewed changes

jamiehannaford force-pushed the cinder-performance branch 2 times, most recently from 25d7dc3 to ba33b16 Compare May 2, 2017 12:49

k8s-github-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 2, 2017

mikedanese assigned saad-ali and unassigned mikedanese May 11, 2017

jamiehannaford force-pushed the cinder-performance branch from ba33b16 to 5588515 Compare May 12, 2017 12:12

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 12, 2017

jamiehannaford force-pushed the cinder-performance branch from 5588515 to ccb3bb1 Compare May 12, 2017 14:30

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 16, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 16, 2017

jamiehannaford force-pushed the cinder-performance branch from ccb3bb1 to cfa8371 Compare May 17, 2017 09:18

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 17, 2017

jamiehannaford force-pushed the cinder-performance branch 3 times, most recently from 9520b26 to 8fe7860 Compare May 17, 2017 12:37

jamiehannaford force-pushed the cinder-performance branch from 8fe7860 to 05373e9 Compare May 24, 2017 10:09

Refactor to use Volume IDs and remove ambiguity

4bd71a3

jamiehannaford force-pushed the cinder-performance branch from 05373e9 to 4bd71a3 Compare May 24, 2017 10:59

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 24, 2017

k8s-github-robot merged commit 70dd10c into kubernetes:master May 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Only retrieve relevant volumes #41785

Only retrieve relevant volumes #41785

jamiehannaford commented Feb 21, 2017 •

edited

Loading

k8s-ci-robot commented Feb 21, 2017

k8s-reviewable commented Feb 21, 2017

jingxu97 Feb 22, 2017

jamiehannaford Feb 22, 2017

jamiehannaford commented Mar 6, 2017

jamiehannaford commented Mar 14, 2017

xsgordon commented Mar 14, 2017

hogepodge commented Mar 14, 2017

jamiehannaford commented Apr 3, 2017

jamiehannaford commented Apr 12, 2017

jingxu97 commented Apr 12, 2017

jingxu97 commented Apr 12, 2017

jamiehannaford commented Apr 25, 2017

anguslees May 1, 2017

anguslees May 1, 2017

jamiehannaford commented May 2, 2017

jamiehannaford commented May 9, 2017

jsafrane commented May 10, 2017

jamiehannaford commented May 16, 2017

jsafrane commented May 16, 2017

jamiehannaford commented May 16, 2017

mikedanese commented May 16, 2017

jamiehannaford commented May 19, 2017

jamiehannaford commented May 22, 2017

jsafrane commented May 24, 2017

k8s-github-robot commented May 24, 2017

k8s-github-robot commented May 24, 2017

k8s-ci-robot commented May 24, 2017 •

edited

Loading

Only retrieve relevant volumes #41785

Only retrieve relevant volumes #41785

Conversation

jamiehannaford commented Feb 21, 2017 • edited Loading

k8s-ci-robot commented Feb 21, 2017

k8s-reviewable commented Feb 21, 2017

jingxu97 Feb 22, 2017

Choose a reason for hiding this comment

jamiehannaford Feb 22, 2017

Choose a reason for hiding this comment

jamiehannaford commented Mar 6, 2017

jamiehannaford commented Mar 14, 2017

xsgordon commented Mar 14, 2017

hogepodge commented Mar 14, 2017

jamiehannaford commented Apr 3, 2017

jamiehannaford commented Apr 12, 2017

jingxu97 commented Apr 12, 2017

jingxu97 commented Apr 12, 2017

jamiehannaford commented Apr 25, 2017

anguslees May 1, 2017

Choose a reason for hiding this comment

anguslees May 1, 2017

Choose a reason for hiding this comment

jamiehannaford commented May 2, 2017

jamiehannaford commented May 9, 2017

jsafrane commented May 10, 2017

jamiehannaford commented May 16, 2017

jsafrane commented May 16, 2017

jamiehannaford commented May 16, 2017

mikedanese commented May 16, 2017

jamiehannaford commented May 19, 2017

jamiehannaford commented May 22, 2017

jsafrane commented May 24, 2017

k8s-github-robot commented May 24, 2017

k8s-github-robot commented May 24, 2017

k8s-ci-robot commented May 24, 2017 • edited Loading

jamiehannaford commented Feb 21, 2017 •

edited

Loading

k8s-ci-robot commented May 24, 2017 •

edited

Loading