feat: azure kms provider as static pod

[aramase]

Reason for Change:

• Configures azure kms provider to be run as a static pod. The following change which was introduced prior to 1.13 ensures the api server waits for the kms plugin to be up and running - kube-apiserver should wait/block for kms-plugin to start. kubernetes/kubernetes#68585

Things to complete as part of this PR:

• Update tests
• Update docs
• Check cluster upgrade scenario

Issue Fixed:

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:
Tested cluster upgrade:

1. Created an aks-engine cluster with enableEncryptionWithExternalKms: true.
2. Validated secret in etcd is encrypted after cluster is deployed and secret is created.
3. Upgraded the cluster to a newer version using custom aks-engine build with the static pod changes.
4. Validated the secret is still encrypted after upgrade, the new kms provider static pod is running and kubectl get secrets returns the decoded content.

➜ kubectl get pods -n kube-system -l component=azure-kms-provider
NAME                                       READY   STATUS    RESTARTS   AGE
azure-kms-provider-k8s-master-14691719-0   1/1     Running   0          33m

[codecov]

Codecov Report

Merging #3667 into master will increase coverage by 0.00%.
The diff coverage is 83.58%.

@@           Coverage Diff           @@
##           master    #3667   +/-   ##
=======================================
  Coverage   73.19%   73.20%           
=======================================
  Files         148      148           
  Lines       25394    25428   +34     
=======================================
+ Hits        18587    18614   +27     
- Misses       5671     5679    +8     
+ Partials     1136     1135    -1     

Impacted Files	Coverage Δ
pkg/api/common/const.go	40.00% <ø> (ø)
pkg/engine/templates_generated.go	52.91% <38.88%> (-0.51%)	⬇️
pkg/api/components.go	97.15% <100.00%> (+0.20%)	⬆️
pkg/api/defaults-kubelet.go	96.82% <100.00%> (+0.01%)	⬆️
pkg/api/k8s_versions.go	100.00% <100.00%> (ø)
pkg/engine/armvariables.go	86.44% <100.00%> (-0.06%)	⬇️
pkg/engine/artifacts.go	99.15% <100.00%> (+0.01%)	⬆️
pkg/engine/template_generator.go	82.40% <100.00%> (+0.20%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c670571...d74f373. Read the comment docs.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aramase]

This PR is ready for review.

cc @ritazh

[jackfrancis]

So basically we are deprecating all systemd foo in favor of everything being deliverd via daemonset?

[jackfrancis]

Actually nevermind I see it's a system Pod

[jackfrancis]

We should be able to delete the parts/k8s/cloud-init/artifacts/kms.service file entirely from the codebase now

[aramase]

deleted parts/k8s/cloud-init/artifacts/kms.service

[jackfrancis]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aramase, jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment