🐛: Use external CCM in default template

[nrb]

What type of PR is this?

What this PR does / why we need it:

The default cluster template uses the external cloud provider, but does not provide the necessary information to install the cloud provider.

Copies the structure of the
cluster-template-external-cloud-provider.yaml into the default template, with the folloiwng changes.

• The following version updates were made to keep up-to-date:
  aws-cloud-controller-manager: v1.28.3
  CSI driver: v1.25.0
  CSI provisioner: v3.6.2
  CSI attacher: v4.4.2
  CSI resizer: v1.9.2
  CSI livenessprobe: v2.11.0
  CSI register: v2.9.2

• The CNI resource references were removed, since the Quickstart lets the user install their own CNI.

• Updated the contents of the aws-ccm-external.yaml file based on files in https://github.com/kubernetes/cloud-provider-aws/tree/master/examples/existing-cluster/base.

  • Changed --configure-cloud-routes=false from the default
  • Changed node selector from node-role.kubernetes.io/master to node-role.kubernetes.io/controle-plane. I was not able to see the CCM pods get scheduled until this was changed.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4677

Special notes for your reviewer:

NOTE: Because this change requires the use of ClusterResourceSet, the Quickstart will need to be updated to include export EXP_CLUSTER_RESOURCE_SET=true prior to initializing the management cluster.

Checklist:

• squashed commits
• includes emojis

Release note:

Updated the default clusterctl template to use external cloud-controller-manager

[nrb]

Given this requires an experimental feature, is this an acceptable solution?

We could perhaps instruct the user to manually install the CCM, though my initial attempt of installing the CCM via Helm needed to have the --config-cloud-routes flag changed at the least; I'm not sure if the node selector change also needs to be made, too.

[vincepri]

/lgtm
/assign @richardcase

[nrb]

Double checking this, I can't get it to trigger the admission webhook warnings about CRSes, so I'm going to remove the breaking change warnings.

[richardcase]

Given this requires an experimental feature, is this an acceptable solution?

I think its fine. We can update the quick start guide to say CRS needs to be enabled and the webhook should also reject the CRS if its not enabled.

[nrb]

@richardcase Removed the extra network stanza.

[richardcase]

Lets make sure we change the upstream quickstart as well.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardcase

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [richardcase]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[richardcase]

Adding the lgtm in that @vincepri did a while back:

/lgtm

[richardcase]

/cherrypick release-2.3

[k8s-infra-cherrypick-robot]

@richardcase: once the present PR merges, I will cherry-pick it on top of release-2.3 in a new PR and assign it to you.

In response to this:

/cherrypick release-2.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-infra-cherrypick-robot]

@richardcase: new pull request created: #4695

In response to this:

/cherrypick release-2.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.