Improve migration of external VMware VMs into KVM cluster #8815

sureshanaparti · 2024-03-20T17:06:45Z

Description

This PR improves/optimises the migration/import performance of external VMware VMs into KVM cluster.

The improved migration process still uses the 'virt-v2v' for importing the VMware VMs, and uses exported OVF (of the VMware VM) as the source for the conversion to KVM instance, instead of the cloned VM (in earlier migration, accessing vCenter using 'virt-v2v' from KVM host). For this,

CloudStack would try to pick the supported host for conversion if one exists, when conversion host is not specified, Otherwise uses any random Up & Enabled host as conversion host. It first checks if the host supports conversion (with virt-v2v, and nbdkit for Ubuntu host) or not.
CloudStack would then export OVF template files of the source VMware VM (or Cloned VM - for Powered-On VMs, it creates a clone of the source VM on VMware) to the temporary conversion location (uses primary or secondary storage - only NFS pools are supported, defaulted to secondary storage). Any cloned VM and OVF template files are cleaned up after the migration, or if any errors.
CloudStack then delegates the conversion to a KVM host. The KVM host accesses/uses the exported OVF on the temporary conversion location as the source for the migration using 'virt-v2v' ova input. It creates the temporary disks and xml on the temporary conversion location, and later these are moved to the destination storage pools to import them. Any errors during virt-v2v conversion due to unsupported guest OS, or other reasons are handled, and the conversion operation is failed.

Noticed 70-90% decrease in the time taken for the migration with this approach (Some earlier linux VM migrations with 3-5 GB disks which took 30-35 mins, now takes 2-3 mins with this improvement).

There are no changes in the VMs supported on vCenter - Stopped & Running Linux VMs, and Stopped Windows VMs are supported as earlier. Additionally, Stopped VMs on standalone hosts are supported (can be imported by specifying its default datacenter name - ha-datacenter along with host and credentails).

Note: All 'virt-v2v' limitations are still applicable. CloudStack will not perform any checks about the guest OS compatibility for the virt-v2v library as indicated on: https://access.redhat.com/articles/1351473.

Doc PR: apache/cloudstack-documentation#388

Other improvements in the migration process:

MS checks the host for instance conversion support (with virt-v2v, and nbdkit for Ubuntu host), & windows guest conversion support (virtio-win package) before attempting migration (using CheckConvertInstanceCommand / CheckConvertInstanceAnswer).
MS checks ovftool available in the host to export OVF from vCenter directly, if not MS is used to export OVF
Auto selects any existing host with instance conversion capability if conversion host not specified (uses detail 'host.instance.conversion').
New Parameter: forcemstoimportvmfiles in importVM API, to force MS to import VM files/OVF if required. Added UI support for it.
New detail 'host.instance.conversion' with value 'true' is added to host_details table if the host supports instance conversion (needs agent restart after installing virt-v2v).
New host response parameter 'instanceconversionsupported' - true, indicating the instance conversion support.
UI shows Instance Conversion Supported - true in Host details if host supports conversion (with virt-v2v, and nbdkit for Ubuntu host).
UI shows Supported KVM hosts (with virt-v2v) for conversion on Import VM dialog.
Support for parallel import/download of OVF disk files on MS or KVM Host, using threads that are configurable using the global settings: threads.on.ms.to.import.vmware.vm.files or threads.on.kvm.host.to.import.vmware.vm.files respectively.

Also, fixes: #8632

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

Host details (Instance Conversion Supported for KVM hosts with virt-v2v):

Import VM from VMware (Supported KVM hosts with virt-v2v):

Flag - to Force MS to import/download VM files (OVF):

VMwareMigrationToKVM_ForceMSFlag_Updated

Threads config - to import/download VM files

How Has This Been Tested?

Manually tested importing external VMware VMs with single and multiple disks(on different datastore) to KVM cluster in CloudStack.

How did you try to break this feature and the system with this change?

sureshanaparti · 2024-03-20T17:07:35Z

@blueorangutan package

blueorangutan · 2024-03-20T17:08:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2024-03-20T17:15:09Z

Codecov Report

Attention: Patch coverage is 26.35135% with 109 lines in your changes are missing coverage. Please review.

Project coverage is 30.97%. Comparing base (308ed13) to head (3ebee84).
Report is 4 commits behind head on 4.19.

Files	Patch %	Lines
...ain/java/com/cloud/hypervisor/guru/VMwareGuru.java	0.00%	66 Missing ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	0.00%	14 Missing ⚠️
.../apache/cloudstack/vm/UnmanagedVMsManagerImpl.java	66.66%	7 Missing and 5 partials ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java	0.00%	7 Missing ⚠️
.../wrapper/LibvirtConvertInstanceCommandWrapper.java	70.58%	3 Missing and 2 partials ⚠️
.../java/com/cloud/hypervisor/HypervisorGuruBase.java	0.00%	2 Missing ⚠️
...m/cloud/hypervisor/vmware/mo/VirtualMachineMO.java	0.00%	2 Missing ⚠️
...va/com/cloud/agent/api/ConvertInstanceCommand.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.19    #8815      +/-   ##
============================================
+ Coverage     30.92%   30.97%   +0.05%     
- Complexity    34290    34372      +82     
============================================
  Files          5355     5355              
  Lines        376634   376724      +90     
  Branches      54808    54823      +15     
============================================
+ Hits         116480   116705     +225     
+ Misses       244820   244661     -159     
- Partials      15334    15358      +24

Flag	Coverage Δ
simulator-marvin-tests	`24.85% <0.00%> (+0.10%)`	⬆️
uitests	`4.39% <ø> (ø)`
unit-tests	`16.56% <26.35%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-03-20T18:09:01Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8985

sureshanaparti · 2024-03-20T18:14:30Z

@blueorangutan test matrix

blueorangutan · 2024-03-20T18:16:03Z

@sureshanaparti a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-03-21T06:15:00Z

[SF] Trillian test result (tid-9541)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41640 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9541-kvm-centos7.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-03-21T07:25:13Z

[SF] Trillian test result (tid-9539)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45973 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9539-xenserver-71.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.63	test_safe_shutdown.py

blueorangutan · 2024-03-21T07:39:32Z

[SF] Trillian test result (tid-9540)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 46712 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9540-vmware-67u3.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

nvazquez

Code LGTM

sureshanaparti · 2024-03-22T06:32:01Z

@blueorangutan package

blueorangutan · 2024-03-22T06:34:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-03-22T07:31:06Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9003

sureshanaparti · 2024-03-22T07:45:18Z

@blueorangutan test matrix

blueorangutan · 2024-03-22T07:46:03Z

@sureshanaparti a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-03-22T19:34:53Z

[SF] Trillian test result (tid-9553)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41092 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9553-xenserver-71.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.82	test_safe_shutdown.py

blueorangutan · 2024-03-22T21:20:44Z

[SF] Trillian test result (tid-9554)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 47384 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t9554-vmware-67u3.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_balanced_drs_algorithm	`Failure`	128.96	test_cluster_drs.py

DaanHoogland

clgtm

…alled in KVM host)

…mfiles', to be generic

…windows guest vms

…om VMware, instead of OVA file - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

… to 3 hrs

…g OVF from MS, and other changes below. - Skip clone for powered off VMs - Fixes to support standalone host (with its default datacenter) - Some code improvements

…to import vm files

sureshanaparti · 2024-06-24T20:16:33Z

@blueorangutan package

blueorangutan · 2024-06-24T20:18:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-24T21:22:04Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10100

sureshanaparti · 2024-06-24T21:31:48Z

@blueorangutan test matrix

blueorangutan · 2024-06-24T21:34:05Z

@sureshanaparti a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-06-25T11:24:56Z

[SF] Trillian test result (tid-10603)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42055 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t10603-xenserver-71.zip
Smoke tests completed. 131 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-06-25T15:58:03Z

[SF] Trillian test result (tid-10605)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46262 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t10605-kvm-centos7.zip
Smoke tests completed. 131 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-06-25T22:35:57Z

[SF] Trillian test result (tid-10604)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 79916 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8815-t10604-vmware-67u3.zip
Smoke tests completed. 128 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_restore_vm	`Error`	3618.07	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	6.73	test_restore_vm.py
test_01_deploy_vm_on_specific_host	`Error`	13.64	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3602.65	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	4.44	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	2.47	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	2.36	test_vm_deployment_planner.py
test_09_expunge_vm	`Failure`	425.69	test_vm_life_cycle.py

sureshanaparti · 2024-06-26T03:47:59Z

@blueorangutan test rocky8 vmware-70u3

blueorangutan · 2024-06-26T03:50:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (rocky8 mgmt + vmware-70u3) has been kicked to run smoke tests

github-actions · 2024-06-26T12:33:37Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

boring-cyborg bot added component:api component:kvm component:vmware labels Mar 20, 2024

sureshanaparti self-assigned this Mar 20, 2024

sureshanaparti added this to the 4.19.1.0 milestone Mar 20, 2024

sureshanaparti requested a review from nvazquez March 20, 2024 17:45

sureshanaparti changed the title ~~Improve migration of VMware VMs into KVM cluster~~ Improve migration of external VMware VMs into KVM cluster Mar 20, 2024

sureshanaparti mentioned this pull request Mar 21, 2024

Migrate VMware VM to KVM improvement documentation apache/cloudstack-documentation#388

Open

nvazquez approved these changes Mar 22, 2024

View reviewed changes

boring-cyborg bot added component:orchestration component:storage component:UI labels Mar 22, 2024

DaanHoogland approved these changes Mar 25, 2024

View reviewed changes

DaanHoogland added the status:needs-testing label Mar 25, 2024

sureshanaparti added 16 commits June 25, 2024 00:39

Skip persistence map update if network doesn't exist

ff8427c

Added support to export OVA from KVM host, through ovftool (when inst…

0260c65

…alled in KVM host)

Updated importvm api param 'usemsforovaexport' to 'forcemstodownloadv…

2d0b7d6

…mfiles', to be generic

Updated hardcoded UI messages with message labels

4ae0b42

Updated UI to support importvm api param - forcemstodownloadvmfiles

5adf353

Improved instance conversion support checks on ubuntu hosts, and for …

b14525b

…windows guest vms

Use OVF template (VM disks and spec files) for instance conversion fr…

d0e45b4

…om VMware, instead of OVA file - this would further increase the migration performance (as it reduces the time for OVA preparation / archiving of the VM files into a single file)

OVF export tool parallel threads code improvements

cddf38e

Updated 'convert.vmware.instance.to.kvm.timeout' config default value…

7519006

… to 3 hrs

Config values check & code improvements

089f743

Updated import log, with time taken and vm details

0c52c28

Support for parallel downloads of VMware VM disk files while exportin…

ee6720e

…g OVF from MS, and other changes below. - Skip clone for powered off VMs - Fixes to support standalone host (with its default datacenter) - Some code improvements

rebase fixes

d324a7e

rebase fixes

8aa2973

minor improvement

a0634e4

code improvements - threads configuration, and api parameter changes …

3cb750d

…to import vm files

sureshanaparti force-pushed the vmware-to-kvm-migration-improvements branch from 449c158 to 3cb750d Compare June 24, 2024 20:13

sureshanaparti assigned vladimirpetrov Jun 26, 2024

github-actions bot added the status:has-conflicts label Jun 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve migration of external VMware VMs into KVM cluster #8815

Improve migration of external VMware VMs into KVM cluster #8815

sureshanaparti commented Mar 20, 2024 •

edited

Loading

sureshanaparti commented Mar 20, 2024

blueorangutan commented Mar 20, 2024

codecov bot commented Mar 20, 2024 •

edited

Loading

blueorangutan commented Mar 20, 2024

sureshanaparti commented Mar 20, 2024

blueorangutan commented Mar 20, 2024

blueorangutan commented Mar 21, 2024

blueorangutan commented Mar 21, 2024

blueorangutan commented Mar 21, 2024

nvazquez left a comment

sureshanaparti commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

sureshanaparti commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

DaanHoogland left a comment

sureshanaparti commented Jun 24, 2024

blueorangutan commented Jun 24, 2024

blueorangutan commented Jun 24, 2024

sureshanaparti commented Jun 24, 2024

blueorangutan commented Jun 24, 2024

blueorangutan commented Jun 25, 2024

blueorangutan commented Jun 25, 2024

blueorangutan commented Jun 25, 2024

sureshanaparti commented Jun 26, 2024

blueorangutan commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

Improve migration of external VMware VMs into KVM cluster #8815

Are you sure you want to change the base?

Improve migration of external VMware VMs into KVM cluster #8815

Conversation

sureshanaparti commented Mar 20, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sureshanaparti commented Mar 20, 2024

blueorangutan commented Mar 20, 2024

codecov bot commented Mar 20, 2024 • edited Loading

Codecov Report

blueorangutan commented Mar 20, 2024

sureshanaparti commented Mar 20, 2024

blueorangutan commented Mar 20, 2024

blueorangutan commented Mar 21, 2024

blueorangutan commented Mar 21, 2024

blueorangutan commented Mar 21, 2024

nvazquez left a comment

Choose a reason for hiding this comment

sureshanaparti commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

sureshanaparti commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

blueorangutan commented Mar 22, 2024

DaanHoogland left a comment

Choose a reason for hiding this comment

sureshanaparti commented Jun 24, 2024

blueorangutan commented Jun 24, 2024

blueorangutan commented Jun 24, 2024

sureshanaparti commented Jun 24, 2024

blueorangutan commented Jun 24, 2024

blueorangutan commented Jun 25, 2024

blueorangutan commented Jun 25, 2024

blueorangutan commented Jun 25, 2024

sureshanaparti commented Jun 26, 2024

blueorangutan commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

sureshanaparti commented Mar 20, 2024 •

edited

Loading

codecov bot commented Mar 20, 2024 •

edited

Loading