Want support for multiple external IP addresses for a Port

[bnaecker]

[FelixMcFelix]

Ben and I talked this over to try and figure out some of the semantics needed on the OPTE side. We discussed 3 interpretations:

1. Several (concurrent) ephemeral/floating IPs assigned to a port.
2. Multiple v4/v6 IPs per interface, possibly over > 1 VPC subnet.
3. Multiple guest vNICs, each with 1 v4/v6 VPC subnet address and a single external IP.

I'd originally assumed something like 1. was intended, but 3./2. seem the closest match to the RFD 21's future directions.

3. in particular seems best – this makes routing obvious on the guest side, while other interpretations require some information passing along layers to guide, e.g., outbound source-IP selection based on traffic destination (firewall/reachability). There are other minor concerns around needing to assign priorities to addresses in 1., and removing UFT entries on address withdrawal, but I think these come up regardless.

[rmustacc]

For some attempted clarification here are the assumptions that I had on this:

• A given interface may only have a single VPC Subnet associated with it. I think we'd want to be careful with loosening this up.
• Eventually that interface may have multiple IP addresses from that VPC subnet assigned to it.
• An instance with multiple interfaces must have one of them designated as "primary". The primary interface has several constraints that are mentioned at https://rfd.shared.oxide.computer/rfd/021#_the_primary_and_multiple_interfaces.
• There may be multiple floating IPs and an ephemeral IP on an instance. They all point to the primary interface's IP addresses.

From there, my assumption for OPTE is that there is a 1:1 mapping in OPTE between its port and an interface in the guest. Multiple floating IPs will show up long before multiple addresses on an interface I believe/expect. Happy to discuss further and clarify more if that helps.

[FelixMcFelix]

I see, thanks a lot for the extra clarification -- I'd missed the details there on multiple floating IPs. My (re-)reading of what we need then is to load-balance outbound across the preferred IP set (priority: 1x SNAT < (1 or $m$?)x ephemeral < $n$ x floating IPs), maintaining affinity on flows.

[rmustacc]

When we're in the world of having lots of outbound IP addresses, I'm not sure how we would to utilize those quite yet in terms of outbound initiated connections. I think for inbound flows we'll need the affinity.