Track dirty page writes from the VMM for live migration

[rbradford]

The live migration code currently handles writes by the guest into the guest memory through the KVM dirty page tracking (and sends those writes over the wire to the destination) however for completeness writes by the VMM to the guest memory (such as VIRTIO devices) should also be tracked and likewise transmitted over the wire.

The existing infrastructure for communicating the changed pages can be reused (e.g. Vm::send_memory_regions) and protocol changes should not be needed however a mechanism for tracking the writes to the guest memory is needed.

This PR rust-vmm/vm-memory#125 by @alexandruag provides a mechanism for tracking writes made to the guest memory through the GuestMemory API.

[rbradford]

@likebreath Here is the setup I used for moving one VM from one VM to another (i.e. the moved VM is nested and uses a shared disk that is exposed into both via virtio-block)

VM 1

target/debug/cloud-hypervisor --kernel ~/src/linux/vmlinux --pmem file=~/workloads/focal1.raw --cpus boot=1 --memory size=512M --cmdline "root=/dev/pmem0p1 console=ttyS0"  --serial tty --console off --api-socket=/tmp/api1 --seccomp false --net ip=192.168.101.1 --disk path=~/workloads/focal-nested.raw

VM 2

target/debug/cloud-hypervisor --kernel ~/src/linux/vmlinux --pmem file=~/workloads/focal2.raw --cpus boot=1 --memory size=512M --cmdline "root=/dev/pmem0p1 console=ttyS0"  --serial tty --console off --api-socket=/tmp/api1 --seccomp false --net ip=192.168.102.1 --disk path=~/workloads/focal-nested.raw

Nested VM on VM 1

sudo ./cloud-hypervisor --kernel vmlinux --serial tty --console off --cmdline "console=ttyS0 root=/dev/vda1" --disk path=/dev/vda --api-socket=/tmp/api1 --net ip=192.168.100.1 --memory size=128M --seccomp false

On VM2

sudo ./cloud-hypervisor --api-socket=/tmp/api2 --seccomp false

On VM2:

sudo ./ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/sock2
sudo socat TCP-LISTEN:6000,reuseaddr UNIX-CLIENT:/tmp/sock2

On VM1:

sudo socat UNIX-LISTEN:/tmp/sock1,reuseaddr TCP:192.168.102.2:6000
sudo ./ch-remote --api-socket=/tmp/api1 send-migration unix:/tmp/sock1

[rbradford]

I think the three areas that need to be looked at:

• Specifying the 'Bitmap implementation to be used for the tracking. All the GuestRegionMmap are created in MemoryManager::create_ram_region.
• Modifying MemoryManager::dirty_memory_range_table to also look for dirty pages in the different regions of the GuestMemoryMmap
• Testing by doing lots of VIRTIO work in the guest and through tracing check that we have dirty pages reported where the VMM is touching the guest memory

[likebreath]

@rbradford When following your setup above to try the live migration, I encountered a (basic) network setup issue: how do I make sure two L1 VMs are reachable from each other?

I did the following setup:

VM1 (whose vmtap0 IP is 192.168.101.1):

sudo ip addr add 192.168.101.2/24 dev ens2
sudo ip link set up dev ens2
sudo ip r add default via 192.168.101.1

VM2 (whose vmtap1 IP is 192.168.102.1):

sudo ip addr add 192.168.102.2/24 dev ens2
sudo ip link set up dev ens2
sudo ip r add default via 192.168.102.1

The access between guests and host are fine (e.g. ping 192.168.102.1 from VM1, and ping 192.168.101.2 from the hos). However, I won't be able to get access between two guests, say ping 192.168.102.2 from VM1 won't work. (I have ip_forward enabled on the host). Any quick pointers on how to get it work? Pardon me on my very limited networking knowledge. Thank you.

[rbradford]

Use L3 routing. Setup a route on each VM that points to the other subnet via the host. Make sure you have IP forwarding turned on there.

[rbradford]

I tested your setup and it worked fine. Did you make sure to disable any iptables/firewall rules on the host?

[likebreath]

I tested your setup and it worked fine. Did you make sure to disable any iptables/firewall rules on the host?

You are right. Problem resolved.

I found/learnt my host iptables was using the default rule that always drops packets on FORWARD: -P FORWARD DROP. I now have two guest VMs reachable from each other, with the following setup on the host iptables:

-A FORWARD -s 192.168.0.0/16 -i vmtap0 -j ACCEPT
-A FORWARD -s 192.168.0.0/16 -i vmtap1 -j ACCEPT

[rbradford]

You can also test without any nesting (but you cannot have a network interface on the VM)

target/debug/cloud-hypervisor --kernel ~/src/linux/vmlinux --pmem file=~/workloads/focal.raw --cpus boot=1 --memory size=1G --cmdline "root=/dev/pmem0p1 console=ttyS0"  --serial tty --console off --api-socket=/tmp/api1 --seccomp false
target/debug/cloud-hypervisor --api-socket=/tmp/api2
rm /tmp/sock; target/debug/ch-remote --api-socket=/tmp/api2 receive-migration unix:/tmp/sock
target/debug/ch-remote --api-socket=/tmp/api1 send-migration unix:/tmp/sock

[rbradford]

FYI, I had issues with an error setting the KVM_SET_SREGS ioctl using ch-5.10.6 as the L1 kernel but that is resolved if you use ch-5.12-rc6 from https://github.com/cloud-hypervisor/linux