Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add isGarbageCollected to perfscale-load-generator ObjListWatcher #8027

Merged
merged 4 commits into from Jul 28, 2022
Merged

Add isGarbageCollected to perfscale-load-generator ObjListWatcher #8027

merged 4 commits into from Jul 28, 2022

Conversation

alaypatel07
Copy link
Contributor

@alaypatel07 alaypatel07 commented Jul 1, 2022 •
edited

What this PR does / why we need it:

This helps in asserting the Delete semantics of the api during
performance testing. It is required that when Delete request is processed
against kubevirt api objects (VMI, VM and VirtualMachineReplicaSet),
kubevirt system is able to observe the complete garbage collection
(from delete request all the way to finalizer being removed and object being
removed from the kube api)

In order to avoid stomping apiserver with get requests from the e2e test suite
a further optimization of introducing a cache is required

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #8025

Special notes for your reviewer:

Release note:

Wait deletion to succeed all the way till objects are finalized in perfscale tests

@kubevirt-bot kubevirt-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: no Indicates the PR's author has not DCO signed all their commits. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/M needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 1, 2022
@kubevirt-bot
Copy link
Contributor

Hi @alaypatel07. Thanks for your PR.

I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubevirt-bot kubevirt-bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Jul 1, 2022
@alaypatel07 alaypatel07 force-pushed the fix-delete-perf-tests branch 2 times, most recently from e9d39e5 to 1b4d9f5 Compare July 1, 2022 04:03
@rthallisey
Copy link
Contributor

/ok-to-test

@kubevirt-bot kubevirt-bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 21, 2022
Alay Patel added 3 commits July 21, 2022 13:19
Add isGarbageCollected to perfscale-load-generator ObjListWatcher
1bf63ed 
This helps in asserting the Delete semantics of the api during
performance testing. It is required that when Delete request is processed
against kubevirt api objects (VMI, VM and VirtualMachineReplicaSet),
kubevirt system is able to observe the complete garbage collection
(from delete request all the way to finalizer being removed and object being
removed from the kube api)

In order to avoid stomping apiserver with get requests from the e2e test suite
a further optimisation of introducing a cache is required

Signed-off-by: Alay Patel <alayp@nvidia.com>
fix imports for watcher.
1a5d770 
Signed-off-by: Alay Patel <alayp@nvidia.com>
update get to use cache instead of live apiserver
0b93da0 
Signed-off-by: Alay Patel <alayp@nvidia.com>
@alaypatel07 alaypatel07 marked this pull request as ready for review July 21, 2022 17:21
@kubevirt-bot kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 21, 2022
@kubevirt-bot kubevirt-bot requested review from EdDev and rmohr July 21, 2022 17:21
@alaypatel07
Copy link
Contributor Author

/test pull-kubevirt-build

selecting docker as container runtime
dial tcp [::1]:8080: connect: connection refused
Build is not clean:
selecting docker as container runtime
Client Version: version.Info{GitVersion:"v0.55.0-124+df43661c4bc810-dirty", GitCommit:"df43661c4bc81015dd920e5747cce2d2c1a24ca6", GitTreeState:"dirty", BuildDate:"2022-07-21T17:34:41Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
dial tcp [::1]:8080: connect: connection refused
dial tcp [::1]:8080: connect: connection refused
On branch main
Changes not staged for commit:

Docker did not come up for some reason. trying to see if this is a flake..

@alaypatel07
Copy link
Contributor Author

https://asciinema.org/a/MNURPsH2alH3EzQ6JIJv6eJBt

I was able to make a local run wait for all the deletions with this.

@alaypatel07
Copy link
Contributor Author

/test periodic-kubevirt-performance-cluster-100-density-test

trying to see if I can trigger the performance test on this PR

@kubevirt-bot
Copy link
Contributor

@alaypatel07: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test pull-kubevirt-apidocs
  • /test pull-kubevirt-build
  • /test pull-kubevirt-build-arm64
  • /test pull-kubevirt-check-unassigned-tests
  • /test pull-kubevirt-client-python
  • /test pull-kubevirt-e2e-k8s-1.22-operator
  • /test pull-kubevirt-e2e-k8s-1.22-sig-compute
  • /test pull-kubevirt-e2e-k8s-1.22-sig-compute-migrations
  • /test pull-kubevirt-e2e-k8s-1.22-sig-monitoring
  • /test pull-kubevirt-e2e-k8s-1.22-sig-network
  • /test pull-kubevirt-e2e-k8s-1.22-sig-storage
  • /test pull-kubevirt-e2e-k8s-1.23-operator
  • /test pull-kubevirt-e2e-k8s-1.23-operator-nonroot
  • /test pull-kubevirt-e2e-k8s-1.23-sig-compute
  • /test pull-kubevirt-e2e-k8s-1.23-sig-compute-cgroupsv2
  • /test pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations
  • /test pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations-nonroot
  • /test pull-kubevirt-e2e-k8s-1.23-sig-compute-nonroot
  • /test pull-kubevirt-e2e-k8s-1.23-sig-network
  • /test pull-kubevirt-e2e-k8s-1.23-sig-network-nonroot
  • /test pull-kubevirt-e2e-k8s-1.23-sig-storage
  • /test pull-kubevirt-e2e-k8s-1.23-sig-storage-cgroupsv2
  • /test pull-kubevirt-e2e-k8s-1.23-sig-storage-nonroot
  • /test pull-kubevirt-e2e-k8s-1.24-ipv6-sig-network
  • /test pull-kubevirt-e2e-k8s-1.24-operator
  • /test pull-kubevirt-e2e-k8s-1.24-sig-compute
  • /test pull-kubevirt-e2e-k8s-1.24-sig-network
  • /test pull-kubevirt-e2e-k8s-1.24-sig-storage
  • /test pull-kubevirt-e2e-kind-1.22-sriov
  • /test pull-kubevirt-e2e-kind-1.23-vgpu
  • /test pull-kubevirt-e2e-kind-1.23-vgpu-nonroot
  • /test pull-kubevirt-e2e-windows2016
  • /test pull-kubevirt-generate
  • /test pull-kubevirt-manifests
  • /test pull-kubevirt-prom-rules-verify
  • /test pull-kubevirt-unit-test
  • /test pull-kubevirt-verify-go-mod
  • /test pull-kubevirtci-bump-kubevirt

The following commands are available to trigger optional jobs:

  • /test build-kubevirt-builder
  • /test pull-kubevirt-check-tests-for-flakes
  • /test pull-kubevirt-code-lint
  • /test pull-kubevirt-e2e-arm64
  • /test pull-kubevirt-e2e-k8s-1.22-sig-compute-realtime
  • /test pull-kubevirt-e2e-k8s-1.22-sig-performance
  • /test pull-kubevirt-e2e-k8s-1.23-single-node
  • /test pull-kubevirt-e2e-k8s-1.23-swap-enabled
  • /test pull-kubevirt-e2e-k8s-1.24-sig-storage-dv-gc
  • /test pull-kubevirt-e2e-kind-1.22-sriov-nonroot
  • /test pull-kubevirt-fossa
  • /test pull-kubevirt-gosec
  • /test pull-kubevirt-goveralls
  • /test pull-kubevirt-verify-rpms

Use /test all to run the following jobs that were automatically triggered:

  • pull-kubevirt-apidocs
  • pull-kubevirt-build
  • pull-kubevirt-build-arm64
  • pull-kubevirt-check-tests-for-flakes
  • pull-kubevirt-check-unassigned-tests
  • pull-kubevirt-client-python
  • pull-kubevirt-code-lint
  • pull-kubevirt-e2e-k8s-1.22-operator
  • pull-kubevirt-e2e-k8s-1.22-sig-compute
  • pull-kubevirt-e2e-k8s-1.22-sig-compute-migrations
  • pull-kubevirt-e2e-k8s-1.22-sig-network
  • pull-kubevirt-e2e-k8s-1.22-sig-storage
  • pull-kubevirt-e2e-k8s-1.23-operator
  • pull-kubevirt-e2e-k8s-1.23-operator-nonroot
  • pull-kubevirt-e2e-k8s-1.23-sig-compute
  • pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations
  • pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations-nonroot
  • pull-kubevirt-e2e-k8s-1.23-sig-compute-nonroot
  • pull-kubevirt-e2e-k8s-1.23-sig-network
  • pull-kubevirt-e2e-k8s-1.23-sig-network-nonroot
  • pull-kubevirt-e2e-k8s-1.23-sig-storage
  • pull-kubevirt-e2e-k8s-1.23-sig-storage-nonroot
  • pull-kubevirt-e2e-k8s-1.24-ipv6-sig-network
  • pull-kubevirt-e2e-k8s-1.24-operator
  • pull-kubevirt-e2e-k8s-1.24-sig-compute
  • pull-kubevirt-e2e-k8s-1.24-sig-network
  • pull-kubevirt-e2e-k8s-1.24-sig-storage
  • pull-kubevirt-e2e-kind-1.22-sriov
  • pull-kubevirt-e2e-kind-1.23-vgpu
  • pull-kubevirt-e2e-kind-1.23-vgpu-nonroot
  • pull-kubevirt-e2e-windows2016
  • pull-kubevirt-fossa
  • pull-kubevirt-generate
  • pull-kubevirt-goveralls
  • pull-kubevirt-manifests
  • pull-kubevirt-prom-rules-verify
  • pull-kubevirt-unit-test

In response to this:

/test periodic-kubevirt-performance-cluster-100-density-test

trying to see if I can trigger the performance test on this PR

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@alaypatel07
Copy link
Contributor Author

/test pull-kubevirt-e2e-k8s-1.22-sig-performance

@alaypatel07
Copy link
Contributor Author

/test pull-kubevirt-e2e-k8s-1.22-sig-performance

@alaypatel07
Copy link
Contributor Author

/test pull-kubevirt-fossa
/test pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations-nonroot
/test pull-kubevirt-e2e-k8s-1.24-sig-storage

rthallisey
rthallisey reviewed Jul 22, 2022
View reviewed changes
Copy link
Contributor

@rthallisey rthallisey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit

tools/perfscale-load-generator/watcher/watcher.go Outdated
if vmi.Status.Phase == kvv1.Succeeded {
w.deleteObj(string(vmi.GetUID()))
if vmi.Status.Phase == kvv1.Succeeded || vmi.Status.Phase == kvv1.Failed {
log.Log.V(4).Errorf("VirtualMachineInstance obj: %v Succeeded, waiting for garbage collection", fmt.Sprintf("%s/%s", vmi.Namespace, vmi.Name))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically we could be in either Succeeded or failed. I think the right way to say this is, " VMI obj: %s is in a final state,.."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, thanks for the catch @rthallisey

check running count is 0 in WaitDeletion and add logs
8e37903 
Signed-off-by: Alay Patel <alayp@nvidia.com>
@rthallisey
Copy link
Contributor

/lgtm

@kubevirt-bot kubevirt-bot added lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 22, 2022
@alaypatel07
Copy link
Contributor Author

/retest-required

@rthallisey
Copy link
Contributor

/approve

@kubevirt-bot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rthallisey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 28, 2022
@kubevirt-bot
Copy link
Contributor

kubevirt-bot commented Jul 28, 2022 •
edited

@alaypatel07: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubevirt-fossa 8e37903 link false /test pull-kubevirt-fossa
pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations-nonroot 8e37903 link unknown /test pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations-nonroot

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@alaypatel07
Copy link
Contributor Author 

19:13:21:   Unexpected error:
19:13:21:       <*url.Error | 0xc001e415c0>: {
19:13:21:           Op: "Get",
19:13:21:           URL: "https://api.github.com/repos/kubevirt/kubevirt/releases?per_page=10000",
19:13:21:           Err: <http.tlsHandshakeTimeoutError>{},
19:13:21:       }
19:13:21:       Get "https://api.github.com/repos/kubevirt/kubevirt/releases?per_page=10000": net/http: TLS handshake timeout
19:13:21:   occurred

seems like a flak in getting releases https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/8027/pull-kubevirt-e2e-k8s-1.23-operator/1552675673892458496

/test pull-kubevirt-e2e-k8s-1.23-operator

@alaypatel07
Copy link
Contributor Author

reported the failure for this test here #7802 (comment), retrying...

/test pull-kubevirt-e2e-kind-1.23-vgpu-nonroot

@alaypatel07 alaypatel07 mentioned this pull request Jul 28, 2022
Closed
@alaypatel07
Copy link
Contributor Author

reported here the failure here: #7168 (comment)

/test pull-kubevirt-e2e-k8s-1.23-sig-compute-migrations-nonroot

@alaypatel07
Copy link
Contributor Author

/override pull-kubevirt-fossa

this failure is well-known and being discussed in the mailing list

@kubevirt-bot
Copy link
Contributor

@alaypatel07: alaypatel07 unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file.

In response to this:

/override pull-kubevirt-fossa

this failure is well-known and being discussed in the mailing list

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubevirt-bot kubevirt-bot merged commit 5ed0962 into kubevirt:main Jul 28, 2022
@alaypatel07 alaypatel07 deleted the fix-delete-perf-tests branch August 1, 2022 17:03
@kubevirt-bot kubevirt-bot mentioned this pull request Aug 4, 2022
Closed
@dhiller dhiller mentioned this pull request May 17, 2023
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rthallisey rthallisey rthallisey left review comments

@stu-gott stu-gott Awaiting requested review from stu-gott

@vladikr vladikr Awaiting requested review from vladikr

@EdDev EdDev Awaiting requested review from EdDev

@rmohr rmohr Awaiting requested review from rmohr

Assignees

@rthallisey rthallisey

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Handle VMI cleanup in the load-generator tool
3 participants
@alaypatel07 @kubevirt-bot @rthallisey