Investigate alternatives to CPEM IP Address provisioning

[displague]

In issue #55, we seek to make Cloud Provider: Equinix Metal (CPEM) optional during installation.

In lieu of CPEM, users will need to provision IP addresses in some other way (manually, using another controller, using Terraform) and assign these addresses statically on all subsequent LoadBalancer resources.

What solutions can we provide here? What are the obstacles to those approaches?

• Use Crossplane to provision IP ranges? Implement IP Reservation Requests crossplane-contrib/provider-equinix-metal#41
  Would we preinstall and configure Crossplane?
  How would the provisioned addresses update the Service records? How would crossplane detect the need for more IP addresses from Service records without addresses?
• Create a custom controller for this purpose?
• Build this functionality into Kube-VIP or MetalLB (are plugins supported)?
• Let Terraform pre-provision ranges and make these ranges available to kube-vip or metallb or the user? Is this a configmap? Is this arguments to a controller?

[displague]

@deitch suggests another approach to #55 and #56 would be to introduce CPEM options to reduce the functionality, allowing for the CPEM to act as the custom controller that provisions addresses but does not operate on nodes in the other Cloud Provider defined ways:

https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/#running-cloud-controller-manager

The cloud controller manager can implement:

• Node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
• Service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
• Route controller - responsible for setting up network routes on your cloud
• any other features you would like to implement if you are running an out-of-tree provider.

We would effectively introduce a flag to disable the Node Controller in CPEM.