-
Notifications
You must be signed in to change notification settings - Fork 39.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce flakiness of density test #66239
Comments
Yes, I definitely agree we should improve our deletions logic - it's not very robust currently. I can remember some RC deletion timeout flakes even from our smaller jobs.
I was suggesting such an idea (in a very rough sense) to you sometime ago if you remember - but in a more general sense, solving for all such flakes instead of doing point-by-point. My idea's roughly the following:
Pros of that approach:
Cons of that approach:
Wdyt? |
cc @mborsz (who's currently looking into addressing our test flakiness) Maciej - Would you be willing to look into this? |
@shyamjvs - yes I know we were talking about that. So if you can do that in a way that we will be able to reuse with ClusterLoader - I'm fine with that. |
IMO isolating flakes is independent problem from moving stuff to cluster-loader :)
Yes - I'm trying to make some changes in test-framework that'll also be reusable by cluster-loader. |
Automatic merge from submit-queue (batch tested with PRs 66296, 66382). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Add flake-reporting utility to testing framework One step towards #66239 /cc @wojtek-t @mborsz (whoever can review first) ```release-note NONE ```
So with #66296 in, we now have a library available for reporting flakes. The next steps to do are:
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale /assign @mborsz |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
WHile a lot of work has happened here, I will still leave it open since some flakes are not yet fully understood. /remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle rotten |
Density test has been merged with load test in the meantime. There are also pretty stable now. |
Based on this run:
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/179/build-log.txt
there are a couple things that went wrong:
Since failure of deleting RC shouldn't break the whole test:
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/scalability/density.go#L365
[That should just cause an error, but don't break the whole test]
If one of a node failed due to some reason, we should force deletion of pods from that, to avoid deleting a namespace to take 1h and fail after that.
[It's probably enough to:
/assign @shyamjvs
Shyam - please take a look or delegate.
The text was updated successfully, but these errors were encountered: