Rolling-update validation harmonization

[justinsb]

This is a follow-on to #8868; I believe the intent of that was to
expose the option to do more (or fewer) retries.

We previously had a single retry to prevent flapping; this basically
unifies the previous behaviour with the idea of making it
configurable.

• validate-count=0 effectively turns off validation.

• validate-count=1 will do a single validation, without flapping
  detection.

• validate-count>=2 will require N succesful validations in a row,
  waiting ValidateSuccessDuration in between.

A nice side-effect of this is that the tests now explicitly specify
ValidateCount=1 instead of setting ValidateSuccessDuration=0, which
had the side effect of doing the equivalent to ValidateCount=1.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

cc @johngmyers @hakman @zetaab

We talked in the last office hours about trying to not change the behaviour while supporting configurable retries per #8868; I believe this does that. I didn't see any other PRs so sent one to get the ball rolling.

I think the piece we're still missing is the idea of detecting flapping and making it an error at least in e2e tests. I think the logic is a little simpler now and so we could relatively easily create e.g. a feature flag to do so which we would set in our tests.

This is one of the bigger 1.18 blockers, I believe.

[zetaab]

for me this looks good. Could you @justinsb check #8897 also, because in my opinion it affects to behaviour pretty much

/lgtm