Fix handling of "leader changed" errors

[cenkalti]

What this PR does / why we need it:
This PR aims to fix temporary "etcdserver: leader changed" errors from kube-apiserver that is previously attempted by #11401 by adding a single retry when this kind of error is detected.

/cc @dims

Special notes for your reviewer:
I also reverted the previous fix that didn't solve the issue. One commit is the revert, the other one is the new fix. It's better if you review commits separately.

If applicable:

• this PR contains documentation
• this PR contains unit tests
• this PR has been tested for backwards compatibility

[dims]

@hickeyma @technosophos @mattfarina Folks, @cenkalti tested a bunch of scenarios recreating the problem(s) with etcd with this patch.

[cenkalti]

Hey @technosophos @hickeyma. Bumping up this for visibility. It's a small fix. Can you take a look please?

[hickeyma]

@cenkalti thanks for pushing the PR. The code looks good in principle. However, we don't have any easy way of testing this and this is the 2nd fix following #11401. Would it be possible to provide some tests for this?

[cenkalti]

@hickeyma I tested this fix manually by setting up a custom k8s cluster with 3 etcd nodes and running etcdctl move-leader with random member ID continuously to trigger the error case. Even with this special setup, the error does not occur every time when I run helm install command as the timing of API calls are very critical that they must overlap with the leader change event.

I checked https://github.com/helm/acceptance-testing project to see If I can write this scenario as an acceptance test but I couldn't find a solution as the clusters are created with kind and there is no easy way to trigger a leader change event.

[hickeyma]

LGTM, thanks @cenkalti

[hickeyma]

Needs another maintainer approval for merge.

[dims]

LGTM

[dims]

thanks @cenkalti @hickeyma

[cenkalti]

@technosophos @mattfarina Can you take a look please?

[technosophos]

Looks good.

[technosophos]

@hickeyma feel free to merge whenever you feel good about it.

[sruthiwander]

Hi, Any chance this could be patched to the 3.2.x release.