allow the restclient to contact multiple apiservers

[mikedanese]

WIP
cc @krousey

fixes #18174

[mikedanese]

this is turning out to be a pretty big change. I will invest more time into fixing tests if we decide this is an okay thing to do.

cc @liggitt @deads2k @aaronlevy @lavalamp

[liggitt]

Is the intent to make clients know about multiple identical back ends? As a client I find that harder to use than an actual HA/load-balanced server

[mikedanese]

Forgot to link the issue: #18174, the desire is to use this specifically for kube-proxy and kubelet. Everything else should communicate to the apiserver through the service vip.

As a client I find that harder to use than an actual HA/load-balanced server

This approach requires one less outside dependency and is thus more portable

[aaronlevy]

Overall this approach seems good to me.

[smarterclayton]

Needs doc about how server overrides this. It might be better to make this alternateServers so that it's clear which one should be first

[aaronlevy]

Any thoughts if this is a reasonable approach? I selfishly would love to have this in the 1.3 release, but I know the freeze is end of week. If it helps, I could try and take a look at updating the tests.

[mikedanese]

@aaronlevy I will fix this up today. I'd appreciate you guys verifying the change on your deployment once it's ready. Currently we don't have any multimaster e2e ci setup.

[smarterclayton]

When this lands we will test it out in the Openshift multimaster setups. My primary concern is the signature change for servers []string - that was ugly in etcd and led to lots of bugs. The only option I could suggest would be "server" and "alternateServers" which is more complex to deal with. I'm also concerned with client cert flows, since some of the patterns we've described might want multiple certs.

[aaronlevy]

@mikedanese sure, happy to help with multi-master testing.

[k8s-github-robot]

@mikedanese PR needs rebase

[aaronlevy]

I've been playing around with this change a bit, and noticed a couple things (also added some minor modifications: 1cdb969).

• The error rate limiting is far too high (in my minimal tests). At the current setting kubelet/kube-proxy running a couple pods will never roll over to the next host. In this current approach, would there be a reason we couldn't roll-over to next host on first failure?
• It is somewhat odd behavior with kubectl - where as a user I might expect it to log failure and retry, but instead it randomly picks a host and fails/exits.

[therc]

Is this still intended for 1.3?

[lavalamp]

This has missed the cut-off date for 1.3.

[k8s-bot]

GCE e2e build/test passed for commit 1c2cff6.

• Test Results
• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-github-robot]

This PR hasn't been active in 54 days. Will be closed in 35 days.

You can add 'keep-open' label to prevent this from happening.

[dshulyak]

@mikedanese do you plan to work on this in nearest time? Asking because we can not rely on multiple dns records and i could try to fix it myself