Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1974598: OpenStack: Optimize cluster deletion #5004

Merged

Conversation

mandre
Copy link
Member

@mandre mandre commented Jun 16, 2021

Previously each destroy function would exit on the first conflict and
it was expected to retry on a next iteration, hoping that in the mean
time the conflict that prevented removal of the resource was fixed.
While this strategy works, it is also slower then necessary. A better
strategy is to try deleting all resources and ignore the ones that have
conflicts. On the next iteration there will be less conflicts.

This patch was tested on a openstack cluster with openshiftSDN, and
we observed a 37% faster cluster deletion. I expect the performance
boost to be significantly higher for clusters using kuryr.

@mandre
Copy link
Member Author

mandre commented Jun 16, 2021

/test e2e-openstack-ipi

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 19, 2021

@mandre: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 19, 2021
@staebler
Copy link
Contributor

Nice. This matches more closely the desired destroy behavior that we am for with cloud platforms.

@mandre mandre force-pushed the openstack-optimize-destroy branch from 7814284 to 3cb8ed4 Compare June 21, 2021 08:30
@mandre
Copy link
Member Author

mandre commented Jun 21, 2021

/hold
I'd like to get numbers for kuryr cluster deletion. If the performance gain is really amazing we might consider backporting the change, in which case I'd like a BZ for it.

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 21, 2021
@Gal-Zaidman
Copy link
Contributor

Gal-Zaidman commented Jun 21, 2021

/test e2e-ovirt

Retriggering, failure due to infrastructure change

@mandre mandre force-pushed the openstack-optimize-destroy branch from 3cb8ed4 to 7875786 Compare June 21, 2021 12:50
@mandre
Copy link
Member Author

mandre commented Jun 21, 2021

My test on a kuryr cluster shows a 24% improvement on the time it takes to delete a cluster, 14m37s vs 19m14s. Not as impressive as I hoped it would be but still nice to take.

@EmilienM
Copy link
Member

INFO Time elapsed: 8m22s

Wow !!

/lgtm
/approve

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 21, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 21, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: EmilienM

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 21, 2021
@mandre mandre changed the title OpenStack: Optimize cluster deletion Bug 1974598: OpenStack: Optimize cluster deletion Jun 22, 2021
@openshift-ci openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jun 22, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 22, 2021

@mandre: This pull request references Bugzilla bug 1974598, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @eurijon

In response to this:

Bug 1974598: OpenStack: Optimize cluster deletion

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested a review from eurijon June 22, 2021 07:07
@mandre
Copy link
Member Author

mandre commented Jun 22, 2021

/retest
e2e-openstack should be working now, thanks to openshift/release#19272

Previously each destroy function would exit on the first conflict and
it was expected to retry on a next iteration, hoping that in the mean
time the conflict that prevented removal of the resource was fixed.
While this strategy works, it is also slower then necessary. A better
strategy is to try deleting all resources and ignore the ones that have
conflicts. On the next iteration there will be less conflicts.

This patch was tested on a openstack cluster with openshiftSDN, and
we observed a 37% faster cluster deletion. I expect the performance
boost to be significantly higher for clusters using kuryr.
@mandre mandre force-pushed the openstack-optimize-destroy branch from 7875786 to 375fe6f Compare June 22, 2021 07:10
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 22, 2021
@pierreprinetti
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 22, 2021
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 22, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 22, 2021

@mandre: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-aws-fips 375fe6f link /test e2e-aws-fips
ci/prow/e2e-libvirt 375fe6f link /test e2e-libvirt
ci/prow/e2e-aws-workers-rhel7 375fe6f link /test e2e-aws-workers-rhel7
ci/prow/e2e-crc 375fe6f link /test e2e-crc

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@mandre
Copy link
Member Author

mandre commented Jun 22, 2021

/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 22, 2021
@openshift-merge-robot openshift-merge-robot merged commit c88955c into openshift:master Jun 22, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 22, 2021

@mandre: All pull requests linked via external trackers have merged:

Bugzilla bug 1974598 has been moved to the MODIFIED state.

In response to this:

Bug 1974598: OpenStack: Optimize cluster deletion

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@pierreprinetti pierreprinetti deleted the openstack-optimize-destroy branch June 22, 2021 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants