Reduce flakiness of density test

[wojtek-t]

Based on this run:
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/179/build-log.txt
there are a couple things that went wrong:

1. Since failure of deleting RC shouldn't break the whole test:
   https://github.com/kubernetes/kubernetes/blob/master/test/e2e/scalability/density.go#L365
   [That should just cause an error, but don't break the whole test]

2. If one of a node failed due to some reason, we should force deletion of pods from that, to avoid deleting a namespace to take 1h and fail after that.
   [It's probably enough to:

• when deleting RC fails, list all pods with that selector and force deletiong of them]

3. Probably reduce timeout for namespace deletion.

/assign @shyamjvs

Shyam - please take a look or delegate.

[shyamjvs]

Yes, I definitely agree we should improve our deletions logic - it's not very robust currently. I can remember some RC deletion timeout flakes even from our smaller jobs.

Since failure of deleting RC shouldn't break the whole test:

I was suggesting such an idea (in a very rough sense) to you sometime ago if you remember - but in a more general sense, solving for all such flakes instead of doing point-by-point. My idea's roughly the following:

• whenever some operation in the test (that's not super-critical) fails, we log it to some flakes.txt file and proceed without causing failures in our tests
• at the end, as part of the CI job we detect if any flakes happened by looking into that file (this would be at test-infra level)
• if yes (and more than some allowed threshold) we will just mark a separate 'flakes' step in the job as red (instead of making our tests red)
• we can later process offline those flakes and try to fix those

Pros of that approach:

• we can clearly separate flakes from genuine test failures (so it'll avoid giving wrong signal to community about scalability tests)
• tests can continue to run until end and give some useful info instead of failing in between
• it will allow us to capture flakes systematically - helping plan/prioritize their fixes

Cons of that approach:

• Sometimes genuine scalability issues may be hidden as 'flakes'. While this can happen, my feeling is we should be able to spot that from the flakes.txt file just as we currently do from build-log. Also, we can limit the flakes threshold to a very low value (for e.g <5) to avoid masking serious issues

Wdyt?

[shyamjvs]

cc @mborsz (who's currently looking into addressing our test flakiness)

Maciej - Would you be willing to look into this?

[wojtek-t]

@shyamjvs - yes I know we were talking about that.
But given the ClusterLoader effort I don't want to spend too much effort on fixing that.

So if you can do that in a way that we will be able to reuse with ClusterLoader - I'm fine with that.
If not - do the simplest possible thing without fixing all possible problems.

[shyamjvs]

But given the ClusterLoader effort I don't want to spend too much effort on fixing that.

IMO isolating flakes is independent problem from moving stuff to cluster-loader :)

So if you can do that in a way that we will be able to reuse with ClusterLoader - I'm fine with that.

Yes - I'm trying to make some changes in test-framework that'll also be reusable by cluster-loader.

[shyamjvs]

So with #66296 in, we now have a library available for reporting flakes. The next steps to do are:

• Move our tests from using framework.ExpectNoError() to framework.RecordFlakeIfError() wherever it makes sense. This should be done for non-critical operations, e.g creating/updating/deleting individual RCs, services, etc
• Fail the test at the end if flakeCount > {some small threshold}

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wojtek-t]

/remove-lifecycle stale

/assign @mborsz

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[wojtek-t]

WHile a lot of work has happened here, I will still leave it open since some flakes are not yet fully understood.

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[wojtek-t]

/remove-lifecycle rotten
/lifecycle frozen

[wojtek-t]

Density test has been merged with load test in the meantime. There are also pretty stable now.
Closing.