api,server,ui: snapshot copy, multi-zone replica

[shwstppr]

Description

This PR adds new functionality to copy snapshots across zones and take snapshots for multiple zones.

Copy functionality is similar to template copy. The source zone acts as the web server from where the destination zone(s) can download the snapshot files. For this purpose, a new API - copySnapshot has been added. The response for copySnapshot will be returning zone and download details from the first destination zone of the request. This behaviour is similar to the copyTemplate API.

In a similar manner, multiple zones can be selected while taking the snapshots or creating snapshot policies. For this snapshot will be taken in the base zone(in which volume is present) and then copied to the additional zones. A new parameter - zoneids has been added to createSnapshot and createSnapshotPolicy APIs.

As snapshots can be present on multiple zones (secondary stores), a new parameter zoneid has been added to delete the snapshot copy on a specific zone.

listSnapshots API has been updated to allow listing snapshot entries for different zones/datastores. New parameters - showUnique, locationType have been added.

Events generated during snapshot operations will now be linked to the snapshot itself rather than the volume of the snapshot.

listSnapshotPolicies and createSnapshotPolicy APIs will return zone details of the zones in which backup will be scheduled for the policy.

---

New API added
copySnapshot

Request and response params updated for APIs

- listSnapshots
- deleteSnapshot
- createTemplate
- listZones
- listSnapshotPolicies
- createSnapshotPolicy

UI updated for

• Snapshot detail view
• Create snapshot form
• Create snapshot policy form
• Create volume (from snapshot) form
• Create template (from snapshot) form

Doc PR: apache/cloudstack-documentation#344

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Manual snapshot operations in a multi-zone environment

Smoke test added:

[root@ref-trl-5659-k-M7-abhishek-kumar-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./ref-trl-5659-k-M7-abhishek-kumar-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/component/test_snapshot_copy.py
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
  from cryptography.hazmat.backends import default_backend

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/Sep_13_2023_22_14_19_WGW1DW All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
=== TestName: test_01_take_snapshot_multi_zone | Status : SUCCESS ===

=== TestName: test_02_copy_snapshot_multi_zone | Status : SUCCESS ===

=== TestName: test_03_take_snapshot_multi_zone_delete_single_zone | Status : SUCCESS ===

=== TestName: test_04_copy_snapshot_multi_zone_delete_all | Status : SUCCESS ===

=== TestName: test_05_take_snapshot_multi_zone_create_volume_additional_zone | Status : SUCCESS ===

=== TestName: test_06_take_snapshot_multi_zone_create_template_additional_zone | Status : SUCCESS ===

=== Final results are now copied to: /marvin//MarvinLogs/test_snapshot_copy_WS0393 ===

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6972

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[codecov]

Codecov Report

Merging #7873 (ba05e7c) into main (c9d2303) will increase coverage by 10.10%.
The diff coverage is 22.67%.

@@              Coverage Diff              @@
##               main    #7873       +/-   ##
=============================================
+ Coverage     19.07%   29.17%   +10.10%     
- Complexity    18250    30616    +12366     
=============================================
  Files          4967     5111      +144     
  Lines        336878   360624    +23746     
  Branches      48395    52697     +4302     
=============================================
+ Hits          64246   105222    +40976     
+ Misses       263345   240946    -22399     
- Partials       9287    14456     +5169     

Flag	Coverage Δ
simulator-marvin-tests	25.13% <17.53%> (+4.88%)	⬆️
uitests	4.79% <0.00%> (-0.06%)	⬇️
unit-tests	14.52% <15.17%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
api/src/main/java/com/cloud/event/EventTypes.java	95.88% <ø> (ø)
.../main/java/com/cloud/storage/VolumeApiService.java	100.00% <ø> (ø)
...n/java/org/apache/cloudstack/api/ApiConstants.java	40.90% <ø> (ø)
...k/api/command/user/snapshot/CreateSnapshotCmd.java	45.67% <100.00%> (+3.17%)	⬆️
...k/api/command/user/snapshot/DeleteSnapshotCmd.java	80.00% <100.00%> (+1.05%)	⬆️
...k/api/command/user/template/CreateTemplateCmd.java	59.74% <100.00%> (+7.10%)	⬆️
...cloudstack/api/command/user/zone/ListZonesCmd.java	93.75% <100.00%> (+0.41%)	⬆️
...g/apache/cloudstack/api/response/ZoneResponse.java	92.04% <100.00%> (-0.98%)	⬇️
...java/org/apache/cloudstack/query/QueryService.java	100.00% <ø> (ø)
...ack/engine/subsystem/api/storage/SnapshotInfo.java	100.00% <ø> (ø)
... and 75 more

... and 1638 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6974

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7395

[shwstppr]

@blueorangutan test matrix

[blueorangutan]

@shwstppr a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-7986)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42340 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t7986-kvm-centos7.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_vm_wrong_checksum	Error	41.60	test_templates.py
test_09_list_templates_download_details	Failure	0.04	test_templates.py

[blueorangutan]

[SF] Trillian test result (tid-7984)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47955 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t7984-xenserver-71.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_10_vpc_tier_kubernetes_cluster	Error	83.21	test_kubernetes_clusters.py

[blueorangutan]

[SF] Trillian test result (tid-7985)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 60893 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t7985-vmware-67u3.zip
Smoke tests completed. 110 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_update_vmprofile_and_vmgroup	Error	62.91	test_vm_autoscaling.py
test_03_scale_down_verify	Failure	309.76	test_vm_autoscaling.py
test_04_stop_remove_vm_in_vmgroup	Failure	17.87	test_vm_autoscaling.py
test_05_remove_vmgroup	Failure	3.14	test_vm_autoscaling.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Error	465.84	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	Failure	291.52	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	Error	291.54	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Failure	593.83	test_vpc_redundant.py
test_01_redundant_vpc_site2site_vpn	Error	452.42	test_vpc_vpn.py

[slavkap]

@DaanHoogland, I have to run a few more tests with the latest changes

[DaanHoogland]

@blueorangutan test rocky8 vmware-70u3

[blueorangutan]

@DaanHoogland a [LL] Trillian-Jenkins test job (rocky8 mgmt + vmware-70u3) has been kicked to run smoke tests

[blueorangutan]

[LL]Trillian test result (tid-6861)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48294 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t6861-vmware-70u3.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@DaanHoogland, I have to run a few more tests with the latest changes

ok, we have an interest in putting this in 4.19. Can you give an E.T.A. on these, @slavkap ?

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7456

[shwstppr]

@blueorangutan test matrix

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-8046)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39837 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8046-kvm-centos7.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-8045)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 48666 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8045-vmware-67u3.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	Failure	588.34	test_kubernetes_clusters.py

[shwstppr]

@blueorangutan test centos7 xs71

[blueorangutan]

@shwstppr [SL] unsupported parameters provided. Supported mgmt server os are: centos7, centos6, suse15, alma8, ubuntu18, ubuntu22, ubuntu20, rocky8, alma9. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-rocky8, kvm-alma8, kvm-alma9, kvm-ubuntu18, kvm-ubuntu20, kvm-ubuntu22, kvm-suse15, vmware-55u3, vmware-60u2, vmware-65u2, vmware-67u3, vmware-70u1, vmware-70u2, vmware-70u3, vmware-80, vmware-80u1, xenserver-65sp1, xenserver-71, xenserver-74, xcpng74, xcpng76, xcpng80, xcpng81, xcpng82

[shwstppr]

@blueorangutan test centos7 xenserver-71

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + xenserver-71) has been kicked to run smoke tests

[slavkap]

CLGTM, tested with StorPool and NFS primary storage

[blueorangutan]

[SF] Trillian test result (tid-8051)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39412 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7873-t8051-xenserver-71.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[weizhouapache]

Merging based on approvals, QA testing.

great work @shwstppr !