Use External Subnets for Service Controller

[sudomateo]

The service controller currently uses floating IPs as the implementation for LoadBalancer services. This presents the following issues.

• Floating IPs are transparent to the instance—traffic arrives on the instance's internal IP address instead of the floating IP address. To work around this we add the instance's internal IP address to the load balancer status, which is technically incorrect.
• Due to the above point, we cannot create two different load balancers using the same service port on the same Kubernetes node. Otherwise, there will be a port conflict.

Let's explore using external subnets instead of floating IPs as the implementation for LoadBalancer services. This would solve the limitations with floating IPs, and create some new limitations, described below.

• The subnet pool that the external subnet is allocated from must already exist and be provided to the LoadBalancer service. This could presumably be done using annotations (e.g., oxide.computer/subnet-pool).
• Attaching an external subnet to an instance does not automatically make the instance listen on any of the IP addresses in that subnet. That is, there would need to be some mechanism to tell the Kubernetes node to listen on an IP address from the attached external subnet. I'm thinking the implementation would allocate /32 IP addresses from the external subnet and use that as the external address in the LoadBalancer status. That should configure the Kubernetes node to listen on that IP address but perhaps I'm mistaken and further discovery is needed here.

Assuming further discovery proves fruitful, switching to external subnets for LoadBalancer services could significantly improve the current state of Kubernetes on Oxide by lifting the IP:PORT conflict limitation that floating IPs currently have.