kube-controller-manager: Add configure-cloud-routes option

[justinsb]

This allows kube-controller-manager to allocate CIDRs to nodes (with
allocate-node-cidrs=true), but will not try to configure them on the
cloud provider, even if the cloud provider supports Routes.

The default is configure-cloud-routes=true, and it will only try to
configure routes if allocate-node-cidrs is also configured, so the
default behaviour is unchanged.

This is useful because on AWS the cloud provider configures routes by
setting up VPC routing table entries, but there is a limit of 50
entries. So setting configure-cloud-routes on AWS would allow us to
continue to allocate node CIDRs as today, but replace the VPC
route-table mechanism with something not limited to 50 nodes.

We can't just turn off the cloud-provider entirely because it also
controls other things - node discovery, load balancer creation etc.

Fix #25602

[bprashanth]

./federation/apis/federation/types.generated.go up to date.
FAILED   ./hack/../hack/verify-codecgen.sh  447s

[justinsb]

Finally passed the integration tests - thanks for the heads-up @bprashanth :-)

Going to leave the e2e flake until I hear whether the PR is something we'll consider - not least because I suspect even if we accept the idea, we'll likely change the name.

[justinsb]

@thockin any thoughts? Could support the general disaggregation of KCM; perhaps in that light it should be "enable-route-controller" (default true)?

[thockin]

LGTM

@mikedanese @cjcullen @Q-Lee FYI

if you rebase and get green before Monday, you're in.

[thockin]

also permission to self-lgtm when rebased and green

[mikedanese]

LGTM

[cjcullen]

Looks fine to me.

[justinsb]

Re-adding lgtm after rebase.

[k8s-github-robot]

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

[k8s-bot]

GCE e2e build/test passed for commit 6c66764.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-github-robot]

Automatic merge from submit-queue

[colemickens]

Is there any reason to not allow the case where:

• I want to allocate Pod CIDRs myself, but,
• I want cloudprovider to write the route table myself?

It's a use-case that we ran into today, it's mostly to unblock some testing of some half-baked functionality, but if I squint I could imagine someone wanting that configuration in a real cluster.