Omicron Live Migration Support

[smklein]

Tracking issue for Omicron changes necessary to support and manage live migration.

Nexus

• Durable storage of live migration state (likely tightly coupled with the question "which sled does an instance live on")
• Client APIs for accessing Sled Agent API
• Decision-making engine for "when should we actually initiate this process"? Likely influenced by system load, system updates (see Update System: Tracking Issue #250), but we can make something super simple to start (or explicitly triggered by an internal API).
• (???) Any networking-related work to ensure that the source/destination propolis instances can actually talk to each other
• Validation of source/destination versions, to ensure that live migration between the two sleds is actually possible
• API + expanded instance state for "getting updates from the propolis instances that are actually doing the work of the updates". Nexus probably wants to be aware of "what phase of the migration are we in", and should be notified by source/destination.

Sled Agent

• Interface to initiate live migration (source propolis instance)
• Interface to initiate live migration (destination propolis instance). Presumably, the source/dest will talk between each other + send notifications up to Nexus as Propolis moves between stages of live migration, but interfaces will be necessary here for bootstrapping.

[gjcolombo]

I've been working on this for the last few weeks, but have been too quiet and insufficiently transparent about it. To fix that (and to help organize my own thoughts), here are my notes on the roadmap I'm following, progress so far, and next steps, with apologies for both the length of this comment and for any errors below.

---

My current mental roadmap for this issue looks like this:

1. Ensure we can migrate between two Propolis servers running without a control plane
2. Implement all the Nexus/sled agent commands needed to kick off such a migration
3. Make sure the end of migration is handled robustly
4. Add logic to tell clients (broadly construed) that a VM has migrated
5. Decide when migrations should start and with what parameters

(1) is mostly done, modulo some changes needed to make sure that Propolis correctly migrates OPTE device state and virtual timers/timestamp counters. That work is being, or should be, tracked in the Propolis repo.

(2) requires a live migration saga and sled agent logic to initiate migrations in response to that saga's actions. The saga is more or less in place today. The two main problems with it are:

• The saga's last step sends an "instance ensure" command to sled agent that's missing most of the instance hardware description that's needed to initialize Propolis correctly. This is Gather and send instance information on migration #1783.
• Registering and starting an instance in a single saga step creates a synchronization problem. This is Race in instance creation saga can cause an instance to be running when it's supposed to be stopped #2209 (equally applicable to instance creation and migration).

(3) needs a good bit of work. Ending migration robustly requires sled agent to know whether an instance is a migration source or migration target. Sled agent also has to reason about the migration states Propolis reports to know whether migrations succeeded or failed. These data combine to determine what InstanceRuntimeState updates sleds have to send to Nexus to keep an instance from getting stuck in a "migrating" state. RFD 361 explains this in more (probably excruciating) detail.

(4) consists of at least two work items: updating rack-level networking configuration when an instance migrates (e.g. the V2P mappings described in #2002) and signaling to any serial console clients that they need to connect to a new Propolis server.

(5) needs more discussion. Currently, there's an external API to migrate an instance to a specific caller-identified sled. Not having discussed with anyone or asked any users, I think that might not be quite the abstraction we want--for example, if I were an operator and wanted to update a sled, I would hope the control plane could drain the sled for me (and find appropriate homes for the instances thereon) without my having to bin-pack instances onto sleds manually. But again, n = 1, not totally up to me, I haven't operated a rack before, this might not really be the right approach.

---

Some notes on work scheduling:

• The remaining work for (1) can be parallelized with (2) and (3).
• (2) and (3) are interrelated enough that I wouldn't recommend trying to parallelize them (any interface/design changes needed for one have to be consumable by the other).
• I'd say (2) is done if the "happy path" through migration is working--i.e. you ask to migrate an instance and, barring any interruptions/crashes/hiccups, the instance moves to its new sled and begins to run there. Once we're at that point, we can make progress on (3) and (4) in parallel.
• We can discuss (5) at any time, and probably start implementing it at any time too, since I doubt the interface to request migrations is going to change very much at this point.

At this point I think we're working according to this plan--there are hypervisor folks working on the various pieces of Propolis migration that are covered by item 1, and I'm slogging ahead on items 2 and 3, about which more in a future comment.

[iximeow]

for late-comers to this issue, at the very least on

and signaling to any serial console clients that they need to connect to a new Propolis server.

this is handled these days. propolis_client::support::InstanceSerialConsoleHelper and its use in Omicron are some magic words to look for, but it's pretty much what Greg described here