Multi master patch

[chuckbutler]

What this PR does / why we need it: Corrects a sync files issue present when running in a HA Master configuration. This PR adds logic to syncronize on first deployment for /etc/kubernetes/serviceaccount.key which will cause cypto verification failure if not 1:1 on each master unit. Additionally syncs basic_auth and additional files in /srv/kubernetes.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #41019

Special notes for your reviewer: This requires PR #41251 as a dependency before merging.

Release note:

Juju - K8s master charm now properly keeps distributed master files in sync for an HA control plane.

[k8s-reviewable]

This change is 

[ivan4th]

I don't see any problems with this patch, but I can't test juju-based clusters at the moment. I already mentioned this in another PR, but maybe I'm should be removed from the reviewer list of this directory?

[ivan4th]

/lgtm

[chuckbutler]

@k8s-bot gce etcd3 e2e test this

@k8s-bot non-cri e2e test this

[chuckbutler]

@ivan4th - I'll make sure to follow up and issue a PR removing you from the reviewers list. Thanks for letting us know you're not comfortable reviewing these submissions. I'm happy to remove you and should you ever feel comfortable moving forward just issue an incoming PR and let me know so I can +1 :)

[chuckbutler]

@k8s-bot gce etcd3 e2e test this

[chuckbutler]

@k8s-bot bazel test this

[chuckbutler]

Fantastic this looks good now. I don't see a follow up that it's on it's way to the submit queue however, and based on PR #41256 it seems like our contributions are stuck in a limbo.

[chuckbutler]

Limbo averted. things are progressing 🎉

[chuckbutler]

@ivan4th - I got ninja'd by another branches lint fixes (hence the rebase). Once I've got the tests settled I'll re-ping for re LGTM label. Thanks for supporting this branch until landing.

[chuckbutler]

Minor flake here -

 I0222 09:47:26.605] FAILED   hack/make-rules/../../hack/verify-staging-godeps.sh	211s

@k8s-bot verify test this

-- edit --

this doesn't appear to have an open flake against it, but is also the first time I've seen this particular test failure. Will open a bug if it continues to be a problem.

[chuckbutler]

upstream issue seems resolved. giving this another go

@k8s-bot verify test this

[chuckbutler]

@ivan4th - ok this looks ready to re-review/LGTM again. all the tests have passed and the merge conflict is resolved. Thanks!

[ivan4th]

/lgtm

[k8s-merge-robot]

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: chuckbutler, ivan4th

Needs approval from an approver in each of these OWNERS Files:

• cluster/juju/OWNERS [chuckbutler]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[chuckbutler]

xref for test flakes: #40295 #39975

[chuckbutler]

@k8s-bot non-cri node e2e test this
@k8s-bot non-cri e2e test this
@k8s-bot kops aws e2e test this
@k8s-bot kubemark e2e test this

[mbruzek]

This change has been reviewed and approved days ago. Now this is getting"Jenkins overloaded" messages. We do want to get this change in before code freeze as it is an important fix for our cluster.

@k8s-bot gce etcd3 e2e test this

[mbruzek]

@k8s-bot node e2e test this

[mbruzek]

All the required tests have passed. I see this PR in the submit queue! Yeah!

[k8s-merge-robot]

Automatic merge from submit-queue (batch tested with PRs 40665, 41094, 41351, 41721, 41843)