New feature: VNF templates and appliances integration #8022

weizhouapache · 2023-09-29T12:10:31Z

Description

This feature introduce a new type of template in CloudStack: "VNF"

Virtualized Network Functions (a.k.a VNF) commonly refers to the software form of network appliances such as a router, firewall, load balancer, etc.

VNF appliances are mostly deployed as virtual machines (VMs) on Linux KVM or VMware vSphere hypervisors on commercial off-the-shelf hardware (COTS).

It supports the life cycle of VNF templates:

register
update
list
delete

It also supports the management of VNF templates and deployment of VNF appliances (from VNF templates) on GUI

Design:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/VNF+Appliance+Integration

Doc PR: apache/cloudstack-documentation#354

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

This feature introduce a new type of template in CloudStack: "VNF" It supports the life cycle of VNF templates: - register - update - list - delete It also supports the management of VNF templates and deployment of VNF appliances (from VNF templates) on GUI Design: https://cwiki.apache.org/confluence/display/CLOUDSTACK/VNF+Appliance+Integration

weizhouapache · 2023-09-29T12:10:45Z

@blueorangutan package

blueorangutan · 2023-09-29T12:12:03Z

@weizhouapache a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2023-09-29T12:21:49Z

Codecov Report

Merging #8022 (cf562c4) into main (587d1d7) will increase coverage by 24.39%.
Report is 1 commits behind head on main.
The diff coverage is 26.64%.

@@             Coverage Diff              @@
##              main    #8022       +/-   ##
============================================
+ Coverage     4.78%   29.18%   +24.39%     
- Complexity       0    30862    +30862     
============================================
  Files          346     5150     +4804     
  Lines        26150   363420   +337270     
  Branches      4497    53230    +48733     
============================================
+ Hits          1252   106071   +104819     
- Misses       24763   242774   +218011     
- Partials       135    14575    +14440

Flag	Coverage Δ
simulator-marvin-tests	`25.22% <12.41%> (?)`
uitests	`4.55% <0.48%> (-0.24%)`	⬇️
unit-tests	`14.69% <60.10%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...pi/src/main/java/com/cloud/server/ResourceTag.java	`98.27% <100.00%> (ø)`
api/src/main/java/com/cloud/storage/Storage.java	`76.76% <100.00%> (ø)`
...n/java/org/apache/cloudstack/api/ApiConstants.java	`40.90% <100.00%> (ø)`
...c/main/java/org/apache/cloudstack/api/BaseCmd.java	`70.33% <ø> (ø)`
...ck/api/command/user/template/ListTemplatesCmd.java	`46.80% <100.00%> (ø)`
...k/api/command/user/template/UpdateTemplateCmd.java	`77.27% <ø> (ø)`
...che/cloudstack/api/command/user/vm/ListVMsCmd.java	`68.33% <100.00%> (ø)`
...cloudstack/api/response/ChildTemplateResponse.java	`0.00% <ø> (ø)`
...ache/cloudstack/api/response/TemplateResponse.java	`68.54% <ø> (ø)`
...loudstack/storage/template/VnfTemplateManager.java	`100.00% <100.00%> (ø)`
... and 51 more

... and 4756 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

blueorangutan · 2023-09-29T13:11:48Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7173

tools/marvin/marvin/lib/base.py

engine/components-api/src/main/java/com/cloud/template/TemplateManager.java

server/src/main/java/com/cloud/api/query/dao/UserVmJoinDaoImpl.java

test/integration/smoke/test_vnf_templates.py

DaanHoogland · 2023-10-02T05:43:25Z

test/integration/smoke/test_vnf_templates.py

+        self.vnf_template = VnfTemplate.register(self.user_apiclient,
+                                                 self.vnf_template_config,
+                                                 zoneid=self.zone.id,
+                                                 hypervisor=self.hypervisor,
+                                                 vnfnics=VNF_NICS,
+                                                 vnfdetails=VNF_DETAILS)


should this be unregistered after the test?

template will be removed in test_05_delete_vnf_template

I think these (tests 1 and 5) should be part of setup and teardown. In this way the order of the tests matters, which can lead to issues of intermittend failures. Tests, even integration tests should be atomic.

weizhouapache · 2023-10-02T07:28:06Z

@blueorangutan package

blueorangutan · 2023-10-02T07:30:03Z

@weizhouapache a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-10-02T07:36:09Z

@blueorangutan package

blueorangutan · 2023-10-02T07:38:04Z

@weizhouapache a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-10-02T07:45:18Z

@blueorangutan LLpackage

blueorangutan · 2023-10-02T07:46:02Z

@weizhouapache a [LL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-10-02T08:31:54Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7189

blueorangutan · 2023-10-02T08:38:40Z

Packaging result [LL]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6188

blueorangutan · 2023-10-02T08:38:41Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7190

weizhouapache · 2023-10-02T09:09:45Z

@blueorangutan LLtest

blueorangutan · 2023-10-02T09:10:03Z

@weizhouapache a [LL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-10-02T09:14:31Z

[LL] Trillian Build Failed (tid-6758)

weizhouapache · 2023-10-02T10:38:34Z

@blueorangutan LLtest rocky8 kvm-rocky8

blueorangutan · 2023-10-02T10:40:03Z

@weizhouapache a [LL] Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

blueorangutan · 2023-10-03T01:13:10Z

[LL]Trillian test result (tid-6759)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 50769 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t6759-kvm-rocky8.zip
Smoke tests completed. 60 look OK, 21 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_deploy_vm_start_failure	`Error`	67.39	test_deploy_vm.py
test_deploy_vm_volume_creation_failure	`Error`	70.98	test_deploy_vm.py
test_vm_ha	`Error`	63.26	test_vm_ha.py
test_vm_sync	`Error`	132.78	test_vm_sync.py
test_07_project_resources_account_delete	`Error`	5.60	test_projects.py
test_08_cleanup_after_project_delete	`Error`	6.80	test_projects.py
ContextSuite context=TestProjectResources>:teardown	`Error`	8.93	test_projects.py
ContextSuite context=TestProjectSuspendActivate>:teardown	`Error`	67.24	test_projects.py
ContextSuite context=TestDeployVmWithAffinityGroup>:teardown	`Error`	99.92	test_affinity_groups_projects.py
test_UpdateConfigParamWithScope	`Error`	0.56	test_global_settings.py
test_03_deploy_vm_wrong_checksum	`Error`	46.72	test_templates.py
test_dedicatePublicIpRange	`Error`	0.03	test_public_ip_range.py
test_create_pvlan_network	`Error`	0.04	test_pvlan.py
test_03_user_role_dont_see_annotations	`Failure`	2.44	test_host_annotations.py
test_03_RVR_Network_check_router_state	`Failure`	100.57	test_routers_network_ops.py
test_02_deploy_vm_root_resize	`Failure`	0.08	test_deploy_vm_root_resize.py
test_02_vpc_privategw_static_routes	`Failure`	140.42	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	136.53	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	207.98	test_privategw_acl.py
test_02_unsecure_vm_migration	`Error`	124.09	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	79.97	test_vm_life_cycle.py
test_03_download_attached_volume	`Error`	19.54	test_volumes.py
test_04_delete_attached_volume	`Error`	0.06	test_volumes.py
test_05_detach_volume	`Error`	1.05	test_volumes.py
test_06_download_detached_volume	`Error`	1.06	test_volumes.py
test_07_resize_fail	`Error`	2.17	test_volumes.py
test_08_resize_volume	`Error`	0.04	test_volumes.py
test_09_delete_detached_volume	`Error`	19.59	test_volumes.py
test_11_attach_volume_with_unstarted_vm	`Error`	2.24	test_volumes.py
ContextSuite context=TestVolumes>:teardown	`Error`	262.06	test_volumes.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	230.88	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Failure`	227.74	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	142.74	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	107.50	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Failure`	202.97	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	230.89	test_vpc_redundant.py
test_oobm_background_powerstate_sync	`Failure`	21.55	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	`Error`	27.40	test_outofbandmanagement.py
test_oobm_issue_power_cycle	`Error`	19.59	test_outofbandmanagement.py
test_oobm_issue_power_off	`Error`	18.57	test_outofbandmanagement.py
test_oobm_issue_power_on	`Error`	19.63	test_outofbandmanagement.py
test_oobm_issue_power_reset	`Error`	19.56	test_outofbandmanagement.py
test_oobm_issue_power_soft	`Error`	19.61	test_outofbandmanagement.py
test_oobm_issue_power_status	`Error`	19.59	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	`Failure`	27.66	test_outofbandmanagement.py
test_oobm_zchange_password	`Error`	7.27	test_outofbandmanagement.py
test_createPortablePublicIPAcquire	`Error`	0.02	test_portable_publicip.py
test_createPortablePublicIPRange	`Error`	0.02	test_portable_publicip.py
test_01_redundant_vpc_site2site_vpn	`Failure`	459.17	test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options	`Error`	1119.98	test_vpc_vpn.py
test_01_vpc_site2site_vpn	`Error`	406.77	test_vpc_vpn.py
test_02_cancel_host_maintenace_with_migration_jobs	`Failure`	172.68	test_host_maintenance.py
test_hostha_kvm_host_degraded	`Error`	7.60	test_hostha_kvm.py
test_hostha_kvm_host_fencing	`Error`	8.57	test_hostha_kvm.py
test_hostha_kvm_host_recovering	`Error`	6.55	test_hostha_kvm.py

weizhouapache · 2023-10-23T11:03:32Z

@blueorangutan test matrix

blueorangutan · 2023-10-23T11:04:03Z

@weizhouapache a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-10-23T23:14:30Z

[SF] Trillian test result (tid-8056)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42389 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t8056-kvm-centos7.zip
Smoke tests completed. 113 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	587.97	test_kubernetes_clusters.py

blueorangutan · 2023-10-23T23:25:14Z

[SF] Trillian test result (tid-8054)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43094 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t8054-xenserver-71.zip
Smoke tests completed. 114 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-10-24T02:21:20Z

[SF] Trillian test result (tid-8055)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 53599 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t8055-vmware-67u3.zip
Smoke tests completed. 113 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_trigger_shutdown	`Failure`	336.57	test_safe_shutdown.py

alexandremattioli · 2023-10-25T19:20:30Z

LGTM

shwstppr

Code LGTM

github-actions · 2023-10-26T06:20:43Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

rajujith

LGTM verified the feature in all the use cases.

weizhouapache · 2023-10-26T10:33:30Z

@blueorangutan package

blueorangutan · 2023-10-26T10:34:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-10-26T10:43:29Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 7522

… Query APIs

weizhouapache · 2023-10-26T11:22:45Z

@blueorangutan package

blueorangutan · 2023-10-26T11:24:03Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-10-26T14:49:15Z

@blueorangutan test matrix

blueorangutan · 2023-10-26T14:50:03Z

@weizhouapache a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-10-27T02:56:29Z

[SF] Trillian test result (tid-8115)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42089 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t8115-kvm-centos7.zip
Smoke tests completed. 115 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-10-27T03:05:47Z

[SF] Trillian test result (tid-8113)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42707 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t8113-xenserver-71.zip
Smoke tests completed. 115 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-10-27T08:55:55Z

[LL] Trillian Build Failed (tid-6868)

blueorangutan · 2023-10-27T12:01:12Z

[SF] Trillian test result (tid-8114)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 74831 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8022-t8114-vmware-67u3.zip
Smoke tests completed. 111 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_balanced_drs_algorithm	`Failure`	128.97	test_cluster_drs.py
test_list_system_vms_metrics_history	`Failure`	0.26	test_metrics_api.py
test_list_vms_metrics_admin	`Error`	3607.36	test_metrics_api.py
test_list_vms_metrics_history	`Error`	4.67	test_metrics_api.py
test_list_volumes_metrics_history	`Error`	3605.47	test_metrics_api.py
test_01_deploy_vm_on_specific_host	`Error`	18.72	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3603.75	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	4.38	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	4.39	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	4.37	test_vm_deployment_planner.py
test_09_expunge_vm	`Failure`	424.59	test_vm_life_cycle.py

boring-cyborg bot added component:api component:storage labels Sep 29, 2023

VNF: fix pre-commit lint

82aa44c

DaanHoogland added this to the 4.19.0.0 milestone Sep 29, 2023

DaanHoogland reviewed Sep 29, 2023

View reviewed changes

tools/marvin/marvin/lib/base.py Show resolved Hide resolved

DaanHoogland reviewed Sep 29, 2023

View reviewed changes

engine/components-api/src/main/java/com/cloud/template/TemplateManager.java Outdated Show resolved Hide resolved

DaanHoogland reviewed Oct 2, 2023

View reviewed changes

VNF: add management property to VNF template nics

047b5da

VNF: changes as per Daan's comments

100307e

VNF: add unit tests

b2038f7

shwstppr marked this pull request as ready for review October 25, 2023 19:25

shwstppr approved these changes Oct 25, 2023

View reviewed changes

shwstppr requested a review from rajujith October 25, 2023 19:28

github-actions bot added the status:has-conflicts label Oct 26, 2023

rajujith approved these changes Oct 26, 2023

View reviewed changes

github-actions bot removed the status:has-conflicts label Oct 26, 2023

weizhouapache added 2 commits October 26, 2023 13:17

Merge remote-tracking branch 'apache/main' into 4.19-vnf and refactor…

b69e02c

… Query APIs

Apply PR8150 to fix build errors

cf562c4

weizhouapache force-pushed the 4.19-vnf branch from 22a649f to cf562c4 Compare October 26, 2023 11:22

DaanHoogland merged commit bd52fa8 into apache:main Oct 27, 2023
26 checks passed

DaanHoogland deleted the 4.19-vnf branch October 27, 2023 08:23

NuxRo mentioned this pull request Nov 10, 2023

In an SG zone a VM can't be connected to an L2 network only #8217

Closed

New feature: VNF templates and appliances integration #8022

New feature: VNF templates and appliances integration #8022

Conversation

weizhouapache commented Sep 29, 2023 • edited

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

weizhouapache commented Sep 29, 2023

blueorangutan commented Sep 29, 2023

codecov bot commented Sep 29, 2023 • edited

Codecov Report

blueorangutan commented Sep 29, 2023

DaanHoogland Oct 2, 2023

Choose a reason for hiding this comment

weizhouapache Oct 2, 2023

Choose a reason for hiding this comment

DaanHoogland Oct 3, 2023

Choose a reason for hiding this comment

weizhouapache commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

weizhouapache commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

weizhouapache commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

weizhouapache commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

weizhouapache commented Oct 2, 2023

blueorangutan commented Oct 2, 2023

blueorangutan commented Oct 3, 2023

weizhouapache commented Oct 23, 2023

blueorangutan commented Oct 23, 2023

blueorangutan commented Oct 23, 2023

blueorangutan commented Oct 23, 2023

blueorangutan commented Oct 24, 2023

alexandremattioli commented Oct 25, 2023

shwstppr left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 26, 2023

rajujith left a comment

Choose a reason for hiding this comment

weizhouapache commented Oct 26, 2023

blueorangutan commented Oct 26, 2023

blueorangutan commented Oct 26, 2023

weizhouapache commented Oct 26, 2023

blueorangutan commented Oct 26, 2023

weizhouapache commented Oct 26, 2023

blueorangutan commented Oct 26, 2023

blueorangutan commented Oct 27, 2023

blueorangutan commented Oct 27, 2023

blueorangutan commented Oct 27, 2023

blueorangutan commented Oct 27, 2023

weizhouapache commented Sep 29, 2023 •

edited

codecov bot commented Sep 29, 2023 •

edited