IPv6 address assignment for sled services

[smklein]

Quoting @rcgoodfellow :

I think it's still an open question how stable addresses will find their way onto servers. The approach I've been taking in my testing setups is the following: Each router running on a rack switch is started with an IPv6 /56 prefix, who provides that? ... not sure. When routers running on servers peer with a rack-level router, the rack-level router delegates a /64 to them and then the router running on the server automatically assigns the first address in that /64 to the server.

It was also discussed, this assigned IP address should be advertised during the IPv6 UDP multicast discovery phase. As implemented, the "self address" during this advertisement is currently set to "UNSPECIFIED", which could certainly be more precise.

Without the actual presence of a switch/router which can assign an IPv6 prefix, this is a little tricky to implement in a test network.

IMO, we should take the following next steps:

• Proceed with the IPv6 advertisement using a combination of hard-coded + "Unspecified" addresses over link-local. This is a placeholder, but it works in our existing lab network.
• Virtualize our switch/router, so we can begin a more realistic process of allocating these prefixes. @rcgoodfellow , I'd be interested in your perspective here - IMO this should be something that is easily integrated with the developer workflow, such that "starting omicron" also "starts the virtual router" (like we basically do for CockroachDB on our Github Actions). Is there a way we could do this with Maghemite - by running it on one of the Sleds, such that it acts as the router?

Related to: #332

[rcgoodfellow]

Currently, the rack-level router delegates a /64 from its /56 to a server-level router based on the index of the physical port the server-level router is attached to. This is not conducive to a local testing setup without additional virtual network machinery.

There is at least one alternative that comes to mind that is more local-development friendly. Having the rack-level router assign prefixes based on some sort of seed that comes from the sever-level router. For example, if the server-level router includes a router discovery or peering option that says "i'm sled number 4" it will get prefix number 4. Currently Maghemite is doing router discovery within a link-local multicast scope (within ff02::/16) and tracks link-layer peering relationships based on link-local unicast source addresses within that scope. I think I'd need to bump that up to global scope and then track peers on ULAs instead of link-local addresses for this to work. This is totally doable. The trade-off that is being made is depending on the sleds to seed the network by configuration rather than the routers sorting it out automatically by link-level geometry. It's also worth noting that being able to test Maghemite locally for things that can be tested locally, without having to use a testbed environment like Falcon is highly appealing - and this design alternative would achieve that.

That being said, we are talking about the bootstrap network here. So this is a network that does not leave the rack and we can simply statically assign a well known IPv6 ULA prefix for it to operate in. Say we choose fd00:a:b::/56 as this network. Then sled 4 will always have the prefix fd00:a:b:4::/64 and can just self assign fd00:a:b:4::1/64 (or anything else within that /64) as its self address.

[smklein]

From a chat conversation with @rcgoodfellow , the following was mentioned, which seems like a great idea in both a development and "real" environment:

1. A rack level Maghemite instance is started.
2. For each "server instance" the developer is running for testing, a server-level Maghemite instance is started.
3. The server-level Maghemite instances will peer with the rack-level one, this will result in the assignment of a ULA.
4. The elements participating in UDP multicast discovery will use their respective assigned ULA address as a source address for multicast communication (this implies that the developers local machine will have a number of disjoint ULA prefixes, one for each "server instance" that is running, and the their host machine will implicitly "route" between the addresses because in actuality they are all local)

This extra step would presumably let everyone build and run the software representing the rack switch (such as assigning ULA addresses) without actually having that hardware yet. Seems like a good compromise - works for developers now, but is aligned with our long-term hardware plans.

[smklein]

#904 provides self-derived bootstrap addresses, and assignment of sled /64s.

#945 describes some follow-up work, but I think it's safe to mark this current issue done.