Live Migration Feature Tracking and Collaboration

[ljrcore]

This issue is created to track the development and discussion of live migration features and related improvements.

Feel free to comment below, claim features you are interested in, and suggest updates to the checklist as needed.

• Pre-copy downtime limit (Jinrong, Songqian)
  vm-migration: Add support for downtime limits #7033
• Asynchronous migration (Jinrong, Songqian)
  Add support for live migration asynchronization #7039
• Migration cancellation (Jinrong, Songqian)
• Migration information query (Jinrong, Songqian)
• Bandwidth limit (Jinrong, Songqian)
• Pre-copy supports vCPU throttling (Philipp)
• Support postcopy (Philipp)
• Parallel network connections to transfer VM state and VM memory faster(Philipp)
• Selectable CPU profiles (Philipp)
• Support migration network devices with network FDs (Philipp)
  [WIP] migration: Persist Nested Virtualisation State #7074
• Support nested virtualization (Philipp)
  [WIP] migration: Persist Nested Virtualisation State #7074
• TLS encryption of the migration (Philipp)
• Smarter downtime calculation: take VM stop and final VM state transfer (devices, vcpus) into account
  vm-migration: Add support for downtime limits #7033 (review))
• Block VM lifecycle API calls during on-going live-migration. sending pause() and resume() or doing hot-detach or hot-attach during live-migration is cursed and we should simply reject it
• Implement abort-migration API call

How to contribute:

• Comment below to discuss implementation details, share ideas, or suggest changes.
• Assign yourself to features you want to work on.
• If you have suggestions for updating the checklist or description, please leave a comment with your proposed changes, or send the revised text to me directly.
• I will update the issue accordingly as progress is made.

[phip1611]

Thanks for creating the issue! Some context: This is a curated list @ljrcore from @Tencent and I from @cyberus-technology discussed by email. It is what we consider to be the necessary steps in order to make Live Migration a battle-ready First-Class-Citizen.

---

We at Cyberus are especially interested in:

• pre-copy with
  • parallel TCP connections and
  • auto-converging (vCPU throttling)
    • currently a PoC is in the works
• possibly post-copy next to pre-copy if our benchmarking data for pre-copy show that memory-intensive workloads wont perform properly during migration
• Migrate guests using nested virtualization
  • ongoing, see [WIP] migration: Persist Nested Virtualisation State #7074
• Selectable CPU profiles
  • see Support for CPU templates/CPU types #7068
• TLS encryption of the migration (currently missing in above's list)

We at Cyberus Technology have a strategic interest and have a pool of capable engineers to work on these things: @blitz, @hertrste, @snue, @tpressure, me + 2 more

[russell-islam]

Will it be configurable if an user wants to post copy or not? Also for post copy Hypervisor needs to support guest memfd if I am not mistaken.

[phip1611]

@ljrcore Could you update the issue? Some things are already WIP.

• Support nested virtualization
  • [WIP] migration: Persist Nested Virtualisation State #7074
• Support migration network devices with network FDs
  • [WIP] add support for live-migration of virtio-net devices & network FDs #7150

[phip1611]

Will it be configurable if an user wants to post copy or not? Also for post copy Hypervisor needs to support guest memfd if I am not mistaken.

If we add post-copy (which is on our roadmap), it will be configurable. We don't know all the details yet but there will be at least this mode:

• post-after-pre: do post-copy after a small amount of pre-copy iterations

[ljrcore]

Welcome Songqian @lisongqian to join the contribution of live migration.

[phip1611]

Can someone with write rights please add

• "smarter downtime calculation: take VM stop and final VM state transfer (devices, vcpus) into account" (context)
• block VM lifecycle API calls during on-going live-migration. sending pause() and resume() or doing hot-detach or hot-attach during live-migration is cursed and we should simply reject it
• implement abort-migration API call.