Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Live migrating VMs between AARCH64 hosts fails #77

Open
abufrejoval opened this issue Apr 11, 2024 · 9 comments
Open

Live migrating VMs between AARCH64 hosts fails #77

abufrejoval opened this issue Apr 11, 2024 · 9 comments

Comments

@abufrejoval
Copy link

abufrejoval commented Apr 11, 2024 •
edited
Loading

I've got two hosts, one RK3588 based Orange PI 5+ and a Raspberry PI 5 with (hopefully) identical ISAs (A76), and am trying to live migrate a VM, that runs just fine on either machine when restarted after an offline migration.

On x86 this is well-established and I've done it for nearly two decades now, but it can be tricky with hosts from different vendors. The more recent versions of KVM/QEMU have abstracted generic CPU types with the greatest common denominators on x86 to enable migration between "cousins" like AMD and Intel CPUs (e.g. x86-64-v2-AES). But downgrading to the lowest common denominator has been a common practice in the past.

When I try to specify a CPU type like A76, A57, A55 or A53 on the VMs, they won't start at all, QEMU reporting a CPU type not supported error.

That is actually an error on its own, I guess, you should be able to downgrade CPUs, although I don't know just how transparently ARM kernels are ready to handle that... (they way that ISA extensions are exploding in the ARM ecoverse, that may soon become interesting)

I wouldn't specify "host" as a CPU type for a VM that I want to live-migrate, because that's logically supposed to inhibit live migrations.

When I specify "max" as a CPU type for that VM, it can be started and I'd half-expect it to be migratable, too, when you're lucky and your hosts are "similar" enough or just identical...

But I'm afraid that is what ultimately causes the migration to fail.

The initial stages work just fine, RAM content is copied (since my VM is using Ceph storage, ony RAM contents and registers need to be carried over), but the final switch fails and the VM is resumed on the source.

I see mostly "migration failed" messages in the GUI and the pvedaemon logs, but this error message seems to be the real error description and since there is no value 258 anywhere else, it migh represent "high" for the CPU type (no, I didn't check yet):

QEMU
kvm: Invalid value 258 expecting positive value <= 242

PRIORITY
    3
SYSLOG_IDENTIFIER
    QEMU
_AUDIT_LOGINUID
    0
_AUDIT_SESSION
    283
_BOOT_ID
    6194508de54d4471ad19da9fea050ae3
_CAP_EFFECTIVE
    1ffffffffff
_CMDLINE
    /usr/bin/kvm -id 502 -name rp5jammy,debug-threads=on -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/502.qmp,server=on,wait=off -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/502.pid -daemonize -smbios type=1,uuid=ee7be9ec-bf71-42e8-b42c-a87977d23cdd -drive if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//AAVMF_CODE.ms.fd -drive if=pflash,unit=1,id=drive-efidisk0,cache=writeback,format=raw,file=rbd:nucpool/vm-502-disk-1:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/nucpool.keyring:rbd_cache_policy=writeback,size=67108864 -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/502.vnc,password=on -cpu max -m 4096 -object iothread,id=iothread-virtioscsi0 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pcie.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pcie.0,addr=0x1f -device pci-bridge,id=pci.3,chassis_nr=3,bus=pcie.0,addr=0x5 -device qemu-xhci,id=qemu-xhci -device usb-tablet,id=tablet,bus=qemu-xhci.0,port=1 -device usb-kbd,id=keyboard,bus=qemu-xhci.0,port=2 -readconfig /usr/share/qemu-server/pve-aarch64.cfg -device virtio-gpu,id=vga,bus=pcie.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/502.qga,server=on,wait=off,id=qga0 -device virtio-serial,id=qga0,bus=pcie.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-serial,id=spice,bus=pcie.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice tls-port=61000,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x3,free-page-reporting=on -iscsi initiator-name=iqn.1993-08.org.debian:01:5c614eedc38 -device virtio-scsi-pci,id=virtioscsi0,bus=pcie.3,addr=0x1,iothread=iothread-virtioscsi0 -drive file=rbd:nucpool/vm-502-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/nucpool.keyring,if=none,id=drive-scsi0,cache=writeback,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap -device scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100 -device virtio-scsi-pci,id=virtioscsi2,bus=pcie.3,addr=0x3 -drive if=none,id=drive-scsi2,media=cdrom,aio=io_uring -device scsi-cd,bus=virtioscsi2.0,channel=0,scsi-id=0,lun=2,drive=drive-scsi2,id=scsi2,bootindex=101 -netdev type=tap,id=net0,ifname=tap502i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device virtio-net-pci,mac=BC:24:11:5E:A5:08,netdev=net0,bus=pcie.0,addr=0xb,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102 -machine type=virt-8.1+pve0,gic-version=host -incoming unix:/run/qemu-server/502.migrate -S
_COMM
    kvm
_EXE
    /usr/bin/qemu-system-aarch64
_GID
    0
_HOSTNAME
    orange
_MACHINE_ID
    cbb96e1db46545c09b14a716d16e0011
_PID
    1341257
_RUNTIME_SCOPE
    system
_STREAM_ID
    684447e77d59485b8fd57104482b9dfb
_SYSTEMD_CGROUP
    /qemu.slice/502.scope
_SYSTEMD_INVOCATION_ID
    0babb797535f47f49dad585ef3faf646
_SYSTEMD_SLICE
    qemu.slice
_SYSTEMD_UNIT
    502.scope
_TRANSPORT
    stdout
_UID
    0
__CURSOR
    s=8812ed8108994e4785a0f03126f9e210;i=51539;b=6194508de54d4471ad19da9fea050ae3;m=122e2e8448;t=615d9e56f1c94;x=c9250a834be89554
__MONOTONIC_TIMESTAMP
    78084211784
__REALTIME_TIMESTAMP
    1712875461614740
@jiangcuo
Copy link
Owner

Live migrate only works on the same cpu。
eg.

rk3588 -> rk3588    work
rk3588 -> rk3588s       failed

@abufrejoval
Copy link
Author

abufrejoval commented Apr 18, 2024 •
edited
Loading

Live migrate only works on the same cpu。 eg.

rk3588 -> rk3588    work
rk3588 -> rk3588s       failed

CPUs between those two should really be identical, even if the SoC isn't. But then the subtle differenes even between the A55 and the A76 make the Alder-Lake E/P core differences with respect to AVS-512 look harmless.

I've been trying to find the equivalent to CPUID for AARCH64 and where within KVM the differences is actually determined, just to see if I can put in a hard override for my systems. The only reason KVM should care is if there is a risk that VMs would actually try to execute instructions that don't exist on one or the other, and in the RK3588 vs RK3588S case that clearly won't happen, in fact all the A76 ISA extensions over the A55 should be disabled by any code running on RK3588* CPUs.

Anyway, this is a whole big can of worms within KVM and nothing you can fix within this project.

Yet without live migration, Proxmox is pretty near pointless.

Well perhaps not Proxmox, because it doesn't really have the level auto automation that oVirt/RHV have, which is what I have been using most. With those it's the management engine which initiates quite a few of the live-migrations to satisfy load and quality rules established by the admin.

Would you prefer to keep it open until we can potentially find a hard override and document it here in case someone needs it?

Or should it be closed, since it's not strictly related to your project?

@jiangcuo
Copy link
Owner

Currently, aarch64 can only use host-passthrough cpu-model.

https://opendev.org/openstack/nova/src/branch/master/releasenotes/notes/aarch64-set-proper-cpu-mode-8455bad7d69dc6fd.yaml

I guess oVirt/RHV can't do live migration between different socs either.

@jiangcuo
Copy link
Owner

LXC is exempt from this restriction

@abufrejoval
Copy link
Author

Currently, aarch64 can only use host-passthrough cpu-model.

https://opendev.org/openstack/nova/src/branch/master/releasenotes/notes/aarch64-set-proper-cpu-mode-8455bad7d69dc6fd.yaml

I guess oVirt/RHV can't do live migration between different socs either.

It's a KVM thing, which all use, nothing specific to Proxmox vs. ovirt/RHV. And on the x86 world that works rather well, when you use the proper CPU definitions, either a really old CPU (e.g. Nehalem) or one of the artificial ones (e.g. x86-64-v2-AES) that have been defined to make this possible.

It allows me to migrate VMs easily between hosts that are as different as an Intel Pentium Silver J5005 Atom, a AMD Ryzen 9 5950X, an Intel Tiger Lake i7-1165G7, an Intel Alder-Lake i7-12700H, and a Xeon E5-2696 v4.

And it works, because Linux itself respects these CPU definitions, compilers know them and use them to build the KVM kernels and the user lands to match them.

All that type of work isn't currently done for AARCH or RISC-V I guess, or only within the proprietary confines of the cloud hyperscalers, where the variations are non-existant or very small.

Could be one of the reasons, why the Proxmox guys themselves aborted their project, but that's pure guesswork.

@abufrejoval
Copy link
Author

And host-passthrough CPU VMs normally cannot be migrated by definition, because there is no fixed subset of instructions defined before the machine is launched. You tend to use those only on machines that really need access to the latest ISA extensions and I've used that also on machines which have pass-through GPUs e.g. for ML. Since those can't be live-migrated, there is no harm in allowing the host cpu type to allow for the lastet and greatest AVX512 or similar extensions.

@abufrejoval
Copy link
Author

OpenVZ allowed container live migration using the OpenVZ kernel. One of the main OpenVZ contributors (Kir Kolyshkin) tried to generalize this under the moniker CRIU, something like a container runtime in user space migration layer, so it could go upstream. But even if he switched to Redhat, they didn't seem to want to see that project proceed and I fear it's dead for now.

Migrating containers like with LXC instead of VMs in many ways can be come even more complex that migrating VMs. I've tried run CUDA on OpenVZ containers and it conflicted with the CUDA runtime loading modules etc. Also tried running CUDA Docker containers nested on OpenVZ containers, which sort of work, ...unless CUDA gets involved.

The lack of good abstractions makes live migrations much harder than they should be and with a new architecture like AARCH64 there would have been a chance to make cleaner interfacers. But evidently that opportunity was missed, when I look at how not even within a single SoC these big/little designs support harmonized ISA extensions as far as I can tell. Stuff like pointer authentication and some low precision floating point extensions are missing from the A55 yet present on the A76? How is that supposed to work?

@SuperKali
Copy link

@jiangcuo there's a problem with live migration, only with QEMU servers, seems like not working but the error not tell anything:

2024-04-20 19:01:18 starting migration of VM 102 to node 't6-2' (192.168.179.15)
2024-04-20 19:01:18 found generated disk 'local:102/vm-102-cloudinit.qcow2' (in current VM config)
2024-04-20 19:01:18 found local disk 'local:102/vm-102-disk-0.qcow2' (attached)
2024-04-20 19:01:18 found local disk 'local:102/vm-102-disk-1.qcow2' (attached)
2024-04-20 19:01:18 drive 'scsi1': size of disk 'local:102/vm-102-cloudinit.qcow2' updated from 0T to 4M
2024-04-20 19:01:18 copying local disk images
2024-04-20 19:01:21 Formatting '/var/lib/vz/images/102/vm-102-cloudinit.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=4194304 lazy_refcounts=off refcount_bits=16
2024-04-20 19:01:21 1104+0 records in
2024-04-20 19:01:21 1104+0 records out
2024-04-20 19:01:21 4521984 bytes (4.5 MB, 4.3 MiB) copied, 0.826864 s, 5.5 MB/s
2024-04-20 19:01:21 59+20 records in
2024-04-20 19:01:21 59+20 records out
2024-04-20 19:01:21 4521984 bytes (4.5 MB, 4.3 MiB) copied, 0.154846 s, 29.2 MB/s
2024-04-20 19:01:21 successfully imported 'local:102/vm-102-cloudinit.qcow2'
2024-04-20 19:01:21 volume 'local:102/vm-102-cloudinit.qcow2' is 'local:102/vm-102-cloudinit.qcow2' on the target
2024-04-20 19:01:21 starting VM 102 on remote node 't6-2'
2024-04-20 19:01:26 volume 'local:102/vm-102-disk-0.qcow2' is 'local:102/vm-102-disk-0.qcow2' on the target
2024-04-20 19:01:26 volume 'local:102/vm-102-disk-1.qcow2' is 'local:102/vm-102-disk-1.qcow2' on the target
2024-04-20 19:01:26 start remote tunnel
2024-04-20 19:01:27 ssh tunnel ver 1
2024-04-20 19:01:27 starting storage migration
2024-04-20 19:01:27 scsi0: start migration to nbd:unix:/run/qemu-server/102_nbd.migrate:exportname=drive-scsi0
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 0.0 B of 30.0 GiB (0.00%) in 0s
drive-scsi0: transferred 233.0 MiB of 30.0 GiB (0.76%) in 1s
drive-scsi0: transferred 508.0 MiB of 30.0 GiB (1.65%) in 2s
drive-scsi0: transferred 748.0 MiB of 30.0 GiB (2.43%) in 3s
drive-scsi0: transferred 923.0 MiB of 30.0 GiB (3.00%) in 4s
drive-scsi0: transferred 1.1 GiB of 30.0 GiB (3.73%) in 5s
drive-scsi0: transferred 1.3 GiB of 30.0 GiB (4.46%) in 6s
drive-scsi0: transferred 1.6 GiB of 30.0 GiB (5.20%) in 7s
drive-scsi0: transferred 1.7 GiB of 30.0 GiB (5.73%) in 8s
drive-scsi0: transferred 2.0 GiB of 30.0 GiB (6.51%) in 9s
drive-scsi0: transferred 2.1 GiB of 30.0 GiB (7.15%) in 10s
drive-scsi0: transferred 2.3 GiB of 30.0 GiB (7.80%) in 11s
drive-scsi0: transferred 2.5 GiB of 30.0 GiB (8.47%) in 12s
drive-scsi0: transferred 2.8 GiB of 30.0 GiB (9.31%) in 13s
drive-scsi0: transferred 3.0 GiB of 30.0 GiB (10.11%) in 14s
drive-scsi0: transferred 3.3 GiB of 30.0 GiB (10.86%) in 15s
drive-scsi0: transferred 3.5 GiB of 30.0 GiB (11.51%) in 16s
drive-scsi0: transferred 3.6 GiB of 30.0 GiB (12.15%) in 17s
drive-scsi0: transferred 3.8 GiB of 30.0 GiB (12.53%) in 18s
drive-scsi0: transferred 4.0 GiB of 30.0 GiB (13.29%) in 19s
drive-scsi0: transferred 4.1 GiB of 30.0 GiB (13.68%) in 20s
drive-scsi0: transferred 4.2 GiB of 30.0 GiB (13.96%) in 22s
drive-scsi0: transferred 4.4 GiB of 30.0 GiB (14.62%) in 23s
drive-scsi0: transferred 4.5 GiB of 30.0 GiB (15.04%) in 24s
drive-scsi0: transferred 4.7 GiB of 30.0 GiB (15.60%) in 25s
drive-scsi0: transferred 4.9 GiB of 30.0 GiB (16.30%) in 26s
drive-scsi0: transferred 5.1 GiB of 30.0 GiB (16.97%) in 27s
drive-scsi0: transferred 5.3 GiB of 30.0 GiB (17.59%) in 28s
drive-scsi0: transferred 5.5 GiB of 30.0 GiB (18.46%) in 29s
drive-scsi0: transferred 5.8 GiB of 30.0 GiB (19.37%) in 30s
drive-scsi0: transferred 6.1 GiB of 30.0 GiB (20.29%) in 31s
drive-scsi0: transferred 6.3 GiB of 30.0 GiB (21.07%) in 32s
drive-scsi0: transferred 6.6 GiB of 30.0 GiB (21.97%) in 33s
drive-scsi0: transferred 6.7 GiB of 30.0 GiB (22.42%) in 34s
drive-scsi0: transferred 7.0 GiB of 30.0 GiB (23.32%) in 35s
drive-scsi0: transferred 7.2 GiB of 30.0 GiB (23.95%) in 36s
drive-scsi0: transferred 7.4 GiB of 30.0 GiB (24.54%) in 37s
drive-scsi0: transferred 7.5 GiB of 30.0 GiB (25.01%) in 38s
drive-scsi0: transferred 7.7 GiB of 30.0 GiB (25.65%) in 39s
drive-scsi0: transferred 7.9 GiB of 30.0 GiB (26.47%) in 40s
drive-scsi0: transferred 8.1 GiB of 30.0 GiB (27.06%) in 41s
drive-scsi0: transferred 8.3 GiB of 30.0 GiB (27.63%) in 42s
drive-scsi0: transferred 8.4 GiB of 30.0 GiB (28.14%) in 43s
drive-scsi0: transferred 8.6 GiB of 30.0 GiB (28.66%) in 44s
drive-scsi0: transferred 8.8 GiB of 30.0 GiB (29.31%) in 45s
drive-scsi0: transferred 9.0 GiB of 30.0 GiB (29.92%) in 46s
drive-scsi0: transferred 9.1 GiB of 30.0 GiB (30.37%) in 47s
drive-scsi0: transferred 9.2 GiB of 30.0 GiB (30.67%) in 48s
drive-scsi0: transferred 9.3 GiB of 30.0 GiB (31.16%) in 49s
drive-scsi0: transferred 9.5 GiB of 30.0 GiB (31.70%) in 50s
drive-scsi0: transferred 9.6 GiB of 30.0 GiB (32.10%) in 51s
drive-scsi0: transferred 9.7 GiB of 30.0 GiB (32.48%) in 52s
drive-scsi0: transferred 9.9 GiB of 30.0 GiB (32.98%) in 53s
drive-scsi0: transferred 10.1 GiB of 30.0 GiB (33.55%) in 54s
drive-scsi0: transferred 10.3 GiB of 30.0 GiB (34.25%) in 55s
drive-scsi0: transferred 10.4 GiB of 30.0 GiB (34.63%) in 56s
drive-scsi0: transferred 10.5 GiB of 30.0 GiB (35.07%) in 57s
drive-scsi0: transferred 10.7 GiB of 30.0 GiB (35.72%) in 58s
drive-scsi0: transferred 10.9 GiB of 30.0 GiB (36.40%) in 59s
drive-scsi0: transferred 11.2 GiB of 30.0 GiB (37.21%) in 1m
drive-scsi0: transferred 11.4 GiB of 30.0 GiB (37.87%) in 1m 1s
drive-scsi0: transferred 11.5 GiB of 30.0 GiB (38.38%) in 1m 2s
drive-scsi0: transferred 11.7 GiB of 30.0 GiB (39.04%) in 1m 3s
drive-scsi0: transferred 11.9 GiB of 30.0 GiB (39.78%) in 1m 4s
drive-scsi0: transferred 12.2 GiB of 30.0 GiB (40.56%) in 1m 5s
drive-scsi0: transferred 12.3 GiB of 30.0 GiB (40.90%) in 1m 6s
drive-scsi0: transferred 12.4 GiB of 30.0 GiB (41.45%) in 1m 7s
drive-scsi0: transferred 12.6 GiB of 30.0 GiB (42.01%) in 1m 8s
drive-scsi0: transferred 12.8 GiB of 30.0 GiB (42.53%) in 1m 9s
drive-scsi0: transferred 13.0 GiB of 30.0 GiB (43.32%) in 1m 10s
drive-scsi0: transferred 13.2 GiB of 30.0 GiB (43.91%) in 1m 11s
drive-scsi0: transferred 13.3 GiB of 30.0 GiB (44.38%) in 1m 12s
drive-scsi0: transferred 13.5 GiB of 30.0 GiB (44.91%) in 1m 13s
drive-scsi0: transferred 13.7 GiB of 30.0 GiB (45.53%) in 1m 14s
drive-scsi0: transferred 13.9 GiB of 30.0 GiB (46.18%) in 1m 15s
drive-scsi0: transferred 14.0 GiB of 30.0 GiB (46.83%) in 1m 16s
drive-scsi0: transferred 14.2 GiB of 30.0 GiB (47.34%) in 1m 17s
drive-scsi0: transferred 14.5 GiB of 30.0 GiB (48.23%) in 1m 18s
drive-scsi0: transferred 14.6 GiB of 30.0 GiB (48.78%) in 1m 19s
drive-scsi0: transferred 14.8 GiB of 30.0 GiB (49.21%) in 1m 20s
drive-scsi0: transferred 14.9 GiB of 30.0 GiB (49.79%) in 1m 21s
drive-scsi0: transferred 15.1 GiB of 30.0 GiB (50.20%) in 1m 22s
drive-scsi0: transferred 15.2 GiB of 30.0 GiB (50.80%) in 1m 23s
drive-scsi0: transferred 15.4 GiB of 30.0 GiB (51.37%) in 1m 24s
drive-scsi0: transferred 15.6 GiB of 30.0 GiB (51.89%) in 1m 25s
drive-scsi0: transferred 15.7 GiB of 30.0 GiB (52.44%) in 1m 26s
drive-scsi0: transferred 15.9 GiB of 30.0 GiB (53.10%) in 1m 27s
drive-scsi0: transferred 16.2 GiB of 30.0 GiB (53.93%) in 1m 28s
drive-scsi0: transferred 16.4 GiB of 30.0 GiB (54.73%) in 1m 29s
drive-scsi0: transferred 16.6 GiB of 30.0 GiB (55.26%) in 1m 30s
drive-scsi0: transferred 16.8 GiB of 30.0 GiB (55.85%) in 1m 31s
drive-scsi0: transferred 16.9 GiB of 30.0 GiB (56.49%) in 1m 32s
drive-scsi0: transferred 17.1 GiB of 30.0 GiB (57.03%) in 1m 33s
drive-scsi0: transferred 17.3 GiB of 30.0 GiB (57.81%) in 1m 34s
drive-scsi0: transferred 17.6 GiB of 30.0 GiB (58.59%) in 1m 35s
drive-scsi0: transferred 17.8 GiB of 30.0 GiB (59.36%) in 1m 37s
drive-scsi0: transferred 18.1 GiB of 30.0 GiB (60.29%) in 1m 38s
drive-scsi0: transferred 18.3 GiB of 30.0 GiB (61.04%) in 1m 39s
drive-scsi0: transferred 18.6 GiB of 30.0 GiB (61.94%) in 1m 40s
drive-scsi0: transferred 18.8 GiB of 30.0 GiB (62.68%) in 1m 41s
drive-scsi0: transferred 19.0 GiB of 30.0 GiB (63.44%) in 1m 42s
drive-scsi0: transferred 19.3 GiB of 30.0 GiB (64.31%) in 1m 43s
drive-scsi0: transferred 19.4 GiB of 30.0 GiB (64.75%) in 1m 44s
drive-scsi0: transferred 19.6 GiB of 30.0 GiB (65.34%) in 1m 45s
drive-scsi0: transferred 19.9 GiB of 30.0 GiB (66.25%) in 1m 46s
drive-scsi0: transferred 20.1 GiB of 30.0 GiB (66.95%) in 1m 47s
drive-scsi0: transferred 20.3 GiB of 30.0 GiB (67.55%) in 1m 48s
drive-scsi0: transferred 20.5 GiB of 30.0 GiB (68.25%) in 1m 49s
drive-scsi0: transferred 20.7 GiB of 30.0 GiB (68.85%) in 1m 50s
drive-scsi0: transferred 20.8 GiB of 30.0 GiB (69.36%) in 1m 51s
drive-scsi0: transferred 20.9 GiB of 30.0 GiB (69.75%) in 1m 52s
drive-scsi0: transferred 21.1 GiB of 30.0 GiB (70.38%) in 1m 53s
drive-scsi0: transferred 21.2 GiB of 30.0 GiB (70.82%) in 1m 54s
drive-scsi0: transferred 21.5 GiB of 30.0 GiB (71.70%) in 1m 55s
drive-scsi0: transferred 21.7 GiB of 30.0 GiB (72.28%) in 1m 56s
drive-scsi0: transferred 21.9 GiB of 30.0 GiB (72.92%) in 1m 57s
drive-scsi0: transferred 22.1 GiB of 30.0 GiB (73.51%) in 1m 58s
drive-scsi0: transferred 22.3 GiB of 30.0 GiB (74.39%) in 1m 59s
drive-scsi0: transferred 22.5 GiB of 30.0 GiB (74.93%) in 2m
drive-scsi0: transferred 22.6 GiB of 30.0 GiB (75.38%) in 2m 1s
drive-scsi0: transferred 22.7 GiB of 30.0 GiB (75.73%) in 2m 2s
drive-scsi0: transferred 22.8 GiB of 30.0 GiB (76.15%) in 2m 3s
drive-scsi0: transferred 23.0 GiB of 30.0 GiB (76.80%) in 2m 4s
drive-scsi0: transferred 23.2 GiB of 30.0 GiB (77.32%) in 2m 5s
drive-scsi0: transferred 23.4 GiB of 30.0 GiB (78.00%) in 2m 6s
drive-scsi0: transferred 23.6 GiB of 30.0 GiB (78.71%) in 2m 7s
drive-scsi0: transferred 23.8 GiB of 30.0 GiB (79.45%) in 2m 8s
drive-scsi0: transferred 24.0 GiB of 30.0 GiB (79.95%) in 2m 9s
drive-scsi0: transferred 24.3 GiB of 30.0 GiB (80.86%) in 2m 10s
drive-scsi0: transferred 24.4 GiB of 30.0 GiB (81.45%) in 2m 11s
drive-scsi0: transferred 24.7 GiB of 30.0 GiB (82.25%) in 2m 12s
drive-scsi0: transferred 24.9 GiB of 30.0 GiB (83.12%) in 2m 13s
drive-scsi0: transferred 25.2 GiB of 30.0 GiB (83.93%) in 2m 14s
drive-scsi0: transferred 25.4 GiB of 30.0 GiB (84.77%) in 2m 15s
drive-scsi0: transferred 25.7 GiB of 30.0 GiB (85.56%) in 2m 16s
drive-scsi0: transferred 25.9 GiB of 30.0 GiB (86.45%) in 2m 17s
drive-scsi0: transferred 26.2 GiB of 30.0 GiB (87.36%) in 2m 18s
drive-scsi0: transferred 26.5 GiB of 30.0 GiB (88.29%) in 2m 19s
drive-scsi0: transferred 26.8 GiB of 30.0 GiB (89.15%) in 2m 20s
drive-scsi0: transferred 27.0 GiB of 30.0 GiB (89.82%) in 2m 21s
drive-scsi0: transferred 27.1 GiB of 30.0 GiB (90.32%) in 2m 22s
drive-scsi0: transferred 27.3 GiB of 30.0 GiB (90.86%) in 2m 23s
drive-scsi0: transferred 27.4 GiB of 30.0 GiB (91.29%) in 2m 24s
drive-scsi0: transferred 27.6 GiB of 30.0 GiB (91.89%) in 2m 25s
drive-scsi0: transferred 27.7 GiB of 30.0 GiB (92.31%) in 2m 26s
drive-scsi0: transferred 27.8 GiB of 30.0 GiB (92.64%) in 2m 27s
drive-scsi0: transferred 27.9 GiB of 30.0 GiB (93.03%) in 2m 28s
drive-scsi0: transferred 28.1 GiB of 30.0 GiB (93.48%) in 2m 29s
drive-scsi0: transferred 28.1 GiB of 30.0 GiB (93.79%) in 2m 30s
drive-scsi0: transferred 28.4 GiB of 30.0 GiB (94.61%) in 2m 31s
drive-scsi0: transferred 28.6 GiB of 30.0 GiB (95.34%) in 2m 32s
drive-scsi0: transferred 28.8 GiB of 30.0 GiB (96.09%) in 2m 33s
drive-scsi0: transferred 29.1 GiB of 30.0 GiB (96.83%) in 2m 34s
drive-scsi0: transferred 29.3 GiB of 30.0 GiB (97.75%) in 2m 35s
drive-scsi0: transferred 29.6 GiB of 30.0 GiB (98.49%) in 2m 36s
drive-scsi0: transferred 29.8 GiB of 30.0 GiB (99.38%) in 2m 37s
drive-scsi0: transferred 30.0 GiB of 30.0 GiB (100.00%) in 2m 38s, ready
all 'mirror' jobs are ready
2024-04-20 19:04:05 efidisk0: start migration to nbd:unix:/run/qemu-server/102_nbd.migrate:exportname=drive-efidisk0
drive mirror is starting for drive-efidisk0
drive-efidisk0: transferred 0.0 B of 64.0 MiB (0.00%) in 0s
drive-efidisk0: transferred 64.0 MiB of 64.0 MiB (100.00%) in 1s, ready
all 'mirror' jobs are ready
2024-04-20 19:04:06 starting online/live migration on unix:/run/qemu-server/102.migrate
2024-04-20 19:04:06 set migration capabilities
2024-04-20 19:04:07 migration downtime limit: 100 ms
2024-04-20 19:04:07 migration cachesize: 256.0 MiB
2024-04-20 19:04:07 set migration parameters
2024-04-20 19:04:07 start migrate command to unix:/run/qemu-server/102.migrate
2024-04-20 19:04:08 migration active, transferred 196.6 MiB of 2.1 GiB VM-state, 326.5 MiB/s
2024-04-20 19:04:09 migration active, transferred 397.7 MiB of 2.1 GiB VM-state, 173.9 MiB/s
2024-04-20 19:04:10 migration active, transferred 568.5 MiB of 2.1 GiB VM-state, 235.9 MiB/s
2024-04-20 19:04:11 migration active, transferred 768.1 MiB of 2.1 GiB VM-state, 223.9 MiB/s
2024-04-20 19:04:12 migration active, transferred 1.0 GiB of 2.1 GiB VM-state, 317.4 MiB/s
2024-04-20 19:04:13 migration active, transferred 1.3 GiB of 2.1 GiB VM-state, 165.2 MiB/s
2024-04-20 19:04:14 migration active, transferred 1.5 GiB of 2.1 GiB VM-state, 190.5 MiB/s
2024-04-20 19:04:15 migration status error: failed
2024-04-20 19:04:15 ERROR: online migrate failure - aborting
2024-04-20 19:04:15 aborting phase 2 - cleanup resources
2024-04-20 19:04:15 migrate_cancel
drive-scsi0: Cancelling block job
drive-efidisk0: Cancelling block job
drive-scsi0: Done.
drive-efidisk0: Done.
2024-04-20 19:04:23 ERROR: migration finished with problems (duration 00:03:05)
TASK ERROR: migration problems

The same hardware two nanopc t6 with the same disks but different kernel.

Host 1: Linux 6.8.7-edge-rockchip-rk3588
Host 2: Linux 6.1.43-vendor-rk35xx

@SuperKali
Copy link

Additional comment: if both node have the same kernel version work, but my question is one, if i put a thirt node with orange pi 5 plus? It works?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SuperKali @jiangcuo @abufrejoval