chore: use Azure CCM for k8s 1.16 and later

[mboersma]

Reason for Change:
Switches to the out-of-tree cloud provider components if useCloudControllerManager is enabled and the Kubernetes version is 1.16.x or later. These components are considered alpha status currently but intended to become the default by 1.17 final release (early December!).

Issue Fixed:
Fixes #1915

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:
The cloud-controller-manager is deployed as a static pod manifest as it already was, and the cloud-node-manager is deployed as an addon. This is odd but I would argue the best we can do, but I'm open to feedback.

[mboersma]

Let's revisit this after #2191 merges because I anticipate some (small) refactorings needed.

[codecov]

Codecov Report

Merging #2161 into master will increase coverage by <.01%.
The diff coverage is 77.58%.

@@            Coverage Diff            @@
##           master   #2161      +/-   ##
=========================================
+ Coverage    71.6%   71.6%   +<.01%     
=========================================
  Files         142     142              
  Lines       24817   24863      +46     
=========================================
+ Hits        17770   17804      +34     
- Misses       5911    5923      +12     
  Partials     1136    1136

[feiskyer]

/lgtm

Thanks a lot.

[feiskyer]

@mboersma any updates on this?

[mboersma]

@feiskyer I think it's ready to merge, just trying to get CI to agree and looking for reviews from the AKS Engine team.

[mboersma]

CI reports that we may be exceeding custom data max length for 1.17 clusters on this PR. :-(

[jackfrancis]

We want:

• addons_test.go UT to validate the expected cloud-node-manager addon default behaviors
• validate.go foo to throw an error if cloud-node-manager addon is explicitly enabled, but isn't supported by an appropriate cluster config (CCM disabled, running the wrong version of k8s, etc)

[jackfrancis]

lgtm pending the requested UT

code looks good, bon courage!

[mboersma]

This was a duplicated pair of entries introduced in an earlier PR.

[mboersma]

Same as above: duplicate entries removed.

[feiskyer]

/lgtm

[jackfrancis]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, jackfrancis, mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis,mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment