Add external APIs to reserve IP Pools for Oxide services

[bnaecker]

Today, there's a single IP Pool for each IP version delegated to us to run Oxide services like Nexus or NTP. This isn't flexible enough, since it doesn't provide a way for operators to run those services on multiple disjoint networks. We need to provide APIs for operators to delegate one or more IP Pools for Oxide services.

We should support basic CRUD on these delegated pools as well, so operators can see them, list their ranges, add ranges, and also remove them from this set of delegated pools.

Note that this relies on some database work to more carefully track which IP Pools are associated with the internal Oxide silo, which will be tracked separately.

[bnaecker]

As part of this, I think we need to rework the IP Pool endpoints more broadly. We have a bunch of inconsistent stuff going on here, like the old "project-scoped" operations (GET /v1/ip-pools), the "system-level" operations not having "system" anywhere in the name, and so on. Here's my proposal to simplify this, along with the actual APIs for delegating IP Pools to Oxide.

Today

As of 1fb71d1, we have the following Nexus endpoints for managing IP Pools:

bnaecker@shale : ~/omicron $ rg ip_pool nexus/external-api/output/nexus_tags.txt
131:project_ip_pool_list                     GET      /v1/ip-pools
132:project_ip_pool_view                     GET      /v1/ip-pools/{pool}
208:ip_pool_create                           POST     /v1/system/ip-pools
209:ip_pool_delete                           DELETE   /v1/system/ip-pools/{pool}
210:ip_pool_list                             GET      /v1/system/ip-pools
211:ip_pool_range_add                        POST     /v1/system/ip-pools/{pool}/ranges/add
212:ip_pool_range_list                       GET      /v1/system/ip-pools/{pool}/ranges
213:ip_pool_range_remove                     POST     /v1/system/ip-pools/{pool}/ranges/remove
214:ip_pool_service_range_add                POST     /v1/system/ip-pools-service/ranges/add
215:ip_pool_service_range_list               GET      /v1/system/ip-pools-service/ranges
216:ip_pool_service_range_remove             POST     /v1/system/ip-pools-service/ranges/remove
217:ip_pool_service_view                     GET      /v1/system/ip-pools-service
218:ip_pool_silo_link                        POST     /v1/system/ip-pools/{pool}/silos
219:ip_pool_silo_list                        GET      /v1/system/ip-pools/{pool}/silos
220:ip_pool_silo_unlink                      DELETE   /v1/system/ip-pools/{pool}/silos/{silo}
221:ip_pool_silo_update                      PUT      /v1/system/ip-pools/{pool}/silos/{silo}
222:ip_pool_update                           PUT      /v1/system/ip-pools/{pool}
223:ip_pool_utilization_view                 GET      /v1/system/ip-pools/{pool}/utilization
224:ip_pool_view                             GET      /v1/system/ip-pools/{pool}
278:silo_ip_pool_list                        GET      /v1/system/silos/{silo}/ip-pools
320:internet_gateway_ip_pool_create          POST     /v1/internet-gateway-ip-pools
321:internet_gateway_ip_pool_delete          DELETE   /v1/internet-gateway-ip-pools/{pool}
322:internet_gateway_ip_pool_list            GET      /v1/internet-gateway-ip-pools

I'm going to ignore the Internet Gateway endpoints for now. Those might need some work too, but I don't think my proposal changes them now.

The first two, prefixed with project_, are the normal Silo user endpoints. These are needed so a developer knows which IP Pools are available to them, say when spawning an instance.

The next chunk, with /v1/system in the path, are the operator-focused APIs. These are used for CRUD on pools, adding ranges, linking them to silos, etc.

Prefixing

The /v1/system/ operator-focused APIs don't have system_ in their operation ID. These are truly system-level things, but we haven't prefixed them that way to date, to avoid a conflict with the developer-focused /v1/ip-pools endpoints. Most of the actual operations don't conflict, but listing and viewing do. So I'm proposing that we change something about the prefixing here to make it more clear.

We could prefix the developer-focused endpoints with silo_. That is accurate, but confusing, because no other developer endpoints are like that. They may not even be aware that they're in a silo. No other endpoints have silo_ as a prefix, so I think it's unfair and inconsistent to do that here.

I think we should instead put system_* in front of all the operator API operations. Yes, we then have a bunch of things prefixed, which is annoying. But it's also accurate and useful.

Concretely, here are the current -> proposed operations:

project_ip_pool_list -> ip_pool_list
project_ip_pool_view -> ip_pool_view
ip_pool_create -> system_ip_pool_create
ip_pool_delete -> system_ip_pool_delete
ip_pool_list -> system_ip_pool_list # Lists all pools and their delegation state
ip_pool_range_add -> system_ip_pool_range_add
ip_pool_range_list -> system_ip_pool_range_list
ip_pool_range_remove -> system_ip_pool_range_remove
ip_pool_service_range_add -> GONE # Ranges can be added and removed from delegated pools like any other
ip_pool_service_range_list -> GONE
ip_pool_service_range_remove -> GONE
ip_pool_service_view -> GONE
ip_pool_silo_link -> system_ip_pool_silo_link
ip_pool_silo_list -> system_ip_pool_silo_list
ip_pool_silo_unlink -> system_ip_pool_silo_unlink
ip_pool_silo_update -> system_ip_pool_silo_update # Just updates the is_default, which seems like overkill
ip_pool_update -> system_ip_pool_update # Just updates metadata
ip_pool_utilization_view -> system_ip_pool_utilization_view
ip_pool_view -> system_ip_pool_view
silo_ip_pool_list -> system_silo_ip_pool_list

Delegating pools

As part of this change, we need APIs for operators to delegate IP Pools to Oxide. That's functionally equivalent to the existing notion of linking a Silo and an IP Pool, it's just linking it to our hidden, internal Silo. We don't want to expose that Silo name or ID via the API, which means it can't show up when the operator lists silos attached to the pool, with the /v1/system/ip-pools/{pool}/silos endpoint. But we need to communicate that it's linked somehow, so operators can see that state and change it if they need.

So what I'm proposing is that we add a new type, SystemIpPool. This includes a delegated field, which could be a bool or an enum, and would communicate if the pool is delegated to Oxide for internal usage. We need a separate type here because the developer API for listing pools shouldn't include this information. This can be set at create time, or as part of an update to an existing pool. We could also consider having this return the actual linked Silo, if it exists, too. I don't see a reason the operator shouldn't see that information.

All the existing operations for listing the IP Pools or Silos linked to each other would be specifically reserved for links to customer silos. The Oxide Silo will never show up here, either when listing IP Pools linked to a Silo, or Silos linked to an IP Pool.

As far as APIs for changing this delegation state, we have a few options. We could co-opt the existing operations for linking / unlinking a pool, adding a way to mean "link this to Oxide's internal Silo", but without spelling its name or ID. We could also add new APIs specifically for doing the delegation or clawing the pool back. Something like system_ip_pool_delegate and system_ip_pool_undelegate.

I'm inclined to use the existing link / unlink endpoints, but it does mean changing the types involved. Instead of a NameOrId, we'd need something like:

enum IpPoolLinkage {
    Silo { silo: NameOrId, is_default: bool },
    DelegatedToOxide,
}

And then accept an Option<IpPoolLinkage> in the request. This is nice because it's the same type we'd use in a modified version of the system_ip_pool_list operation.

@david-crespo @ahl This is a pretty big change, so I'd love some input.

[david-crespo]

So is the idea that all the system_ IP pool endpoints would deal in SystemIpPool objects? We don't really do that anywhere else, but I guess we don't really have anything else like this. One place we've talked about a similar structure is with fleet-level images that could then be made visible within silos by the operator, which would save the operator from having to upload the same images to every silo. In that situation I guess we would have a SystemImage distinct from an Image?

[bnaecker]

That's the idea, yeah.

We don't really do that anywhere else, but I guess we don't really have anything else like this.

Yeah, we don't do that elsewhere. There also aren't many other resources that are so visible at to the operator and developer, and have different contents (here, the delegation state). I don't love it, but as you pointed out on #8894, there isn't an unambiguously good answer I can see.

[david-crespo]

My main concern with that is that it's easy to confuse the distinction between system IP pool and regular IP pool and between a delegated and non-delegated system IP pool. In other words, if I saw SystemIpPool without knowing anything, I might think that a system IP pool was delegated to oxide and a IP pool was not. Doesn't mean we shouldn't do it, just something to watch out for and make extra clear.

[bnaecker]

That's a good point. I think it'll be alright, since each class of endpoints returns a homogeneous collection of one type. So ip_pool_list returns only IpPools and system_ip_pool_list only SystemIpPools. But you're right that documentation will be important here too.