Add sent and received bytes to listNetworks and listVirtualMachines. #4776

ravening · 2021-03-09T12:58:41Z

Description

Display the traffic data in networks and vm api response so that we can quickly get the traffic statistics about all the vm's in that network. User can traffic data for a particular network through api as well.
Also users can monitor the traffic on individual Vm as well which was not possible earlier.

(local) mgt01 > list virtualmachines filter=receivedbytes,sentbytes,
{
  "count": 2,
  "virtualmachine": [
    {
      "receivedbytes": 187686,
      "sentbytes": 99809
    },
    {
      "receivedbytes": 189086,
      "sentbytes": 90855
    }
  ]
}

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

DaanHoogland

clgtm needs testing, can you add info about the use-case to the description or refer an issue (even though it is trivial, it might hekp tracking if someone asks about this)

ravening · 2021-03-24T08:39:23Z

clgtm needs testing, can you add info about the use-case to the description or refer an issue (even though it is trivial, it might hekp tracking if someone asks about this)

@DaanHoogland added more details in description

blueorangutan · 2021-03-24T18:43:15Z

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 242

GabrielBrascher

Thanks for the PR @ravening. Overall it looks good (code-wise at least, I did not test it).

I raised a couple of points. Can you please take a look at them @ravening?

server/src/main/java/com/cloud/api/ApiResponseHelper.java

server/src/main/java/com/cloud/api/query/dao/UserVmJoinDaoImpl.java

blueorangutan · 2021-03-24T20:13:53Z

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 251

davidjumani · 2021-03-25T04:09:56Z

@ravening Can you fix the conflicts? Thanks

nvazquez · 2021-06-23T14:29:35Z

Hi @ravening there is still a conflict with this PR can you please resolve it?

blueorangutan · 2021-07-29T11:12:34Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 682

DaanHoogland · 2021-07-29T12:18:37Z

@blueorangutan test

blueorangutan · 2021-07-29T12:19:02Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-07-30T07:27:58Z

Trillian test result (tid-1418)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 59398 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4776-t1418-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 87 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3607.33	test_kubernetes_clusters.py
test_02_deploy_and_upgrade_kubernetes_cluster	`Failure`	3604.35	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.06	test_kubernetes_clusters.py
test_04_basic_lifecycle_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	39.65	test_kubernetes_clusters.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	401.94	test_vpc_redundant.py

nvazquez

LGTM

Tested on VM on shared network:

(localcloud) SBCM5> > list networks filter=id,receivedbytes,sentbytes,
{
  "count": 1,
  "network": [
    {
      "id": "8d4184b3-6e76-4fc6-bbdb-16e387a6efbe",
      "receivedbytes": 13519,
      "sentbytes": 9444
    }
  ]
}
(localcloud) SBCM5> > list virtualmachines filter=id,receivedbytes,sentbytes,
{
  "count": 1,
  "virtualmachine": [
    {
      "id": "1b954319-8a65-448b-bd1d-78f51ee2cd72",
      "receivedbytes": 13519,
      "sentbytes": 9444
    }
  ]
}

api/src/main/java/org/apache/cloudstack/api/response/UserVmResponse.java

RodrigoDLopez

Code LGTM.
But to me seems like you are using the key 'final' where we don't need it.

Did I understood something wrong?

Do we need more manual test on this PR? since @nvazquez has already did some tests.

api/src/main/java/org/apache/cloudstack/api/response/UserVmResponse.java

server/src/main/java/com/cloud/api/query/dao/UserVmJoinDaoImpl.java

api/src/main/java/org/apache/cloudstack/api/response/NetworkResponse.java

DaanHoogland · 2021-07-30T19:17:24Z

Code LGTM.
But to me seems like you are using the key 'final' where we don't need it.

Did I understood something wrong?

Do we need more manual test on this PR? since @nvazquez has already did some tests.

It is, in some circles, good practice to make sure a parameter is only changed consciously. A bit overdone indeed, i would agree.

nvazquez · 2021-08-02T19:40:53Z

Hi @ravening can you please address the open comments?

Display the traffic data in networks and vm api response

ravening · 2021-08-09T14:04:30Z

Hi @ravening can you please address the open comments?

@nvazquez done

nvazquez · 2021-08-09T15:40:25Z

@blueorangutan package

blueorangutan · 2021-08-09T15:41:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-08-09T16:09:49Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 812

nvazquez · 2021-08-09T16:17:30Z

@blueorangutan test

blueorangutan · 2021-08-09T16:18:05Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-08-11T01:40:05Z

Trillian test result (tid-1560)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 117605 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4776-t1560-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_affinity_groups_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_async_job.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vms_with_varied_deploymentplanners.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_with_userdata.py
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_domain_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_nested_virtualization.py
Intermittent failure detected: /marvin/tests/smoke/test_network_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_nic_adapter_type.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 59 look OK, 30 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestAccounts>:setup	`Error`	0.00	test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup	`Error`	0.00	test_accounts.py
test_DeleteDomain	`Error`	3.98	test_accounts.py
test_forceDeleteDomain	`Failure`	3.97	test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup	`Error`	5.33	test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup	`Error`	0.00	test_affinity_groups_projects.py
ContextSuite context=TestInternalLb>:setup	`Error`	0.00	test_internal_lb.py
ContextSuite context=TestAsyncJob>:setup	`Error`	0.00	test_async_job.py
test_01_create_iso_with_checksum_sha1	`Error`	67.42	test_iso.py
test_02_1_create_iso_with_checksum_sha256_negative	`Error`	66.80	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	67.09	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	67.09	test_iso.py
test_03_1_create_iso_with_checksum_md5_negative	`Error`	70.16	test_iso.py
test_03_create_iso_with_checksum_md5	`Error`	67.91	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	67.96	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	67.96	test_iso.py
test_01_create_iso	`Failure`	1588.17	test_iso.py
ContextSuite context=TestISO>:setup	`Error`	3104.09	test_iso.py
ContextSuite context=TestDomainsServiceOfferings>:setup	`Error`	1568.62	test_domain_service_offerings.py
ContextSuite context=TestLoadBalance>:setup	`Error`	0.00	test_loadbalance.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
ContextSuite context=TestDeployVMFromISO>:setup	`Error`	0.00	test_deploy_vm_iso.py
test_list_clusters_metrics	`Error`	1534.75	test_metrics_api.py
test_list_vms_metrics	`Error`	0.17	test_metrics_api.py
ContextSuite context=TestDeployVmWithVariedPlanners>:setup	`Error`	0.00	test_deploy_vms_with_varied_deploymentplanners.py
ContextSuite context=TestDeployVmWithUserData>:setup	`Error`	0.00	test_deploy_vm_with_userdata.py
ContextSuite context=TestRemoteDiagnostics>:setup	`Error`	0.00	test_diagnostics.py
ContextSuite context=TestAdapterTypeForNic>:setup	`Error`	0.00	test_nic_adapter_type.py
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_02_deploy_and_upgrade_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_04_basic_lifecycle_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_06_deploy_invalid_kubernetes_ha_cluster	`Failure`	0.01	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.00	test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_version	`Error`	60.56	test_kubernetes_supported_versions.py
ContextSuite context=TestListIdsParams>:setup	`Error`	0.00	test_list_ids_parameter.py
test_nic_secondaryip_add_remove	`Error`	1553.75	test_multipleips_per_nic.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup	`Error`	0.00	test_password_server.py
ContextSuite context=TestPrivateGwACL>:setup	`Error`	0.00	test_privategw_acl.py
ContextSuite context=TestProjectSuspendActivate>:setup	`Error`	1570.04	test_projects.py
ContextSuite context=TestVMWareStoragePolicies>:setup	`Error`	0.00	test_storage_policy.py
test_02_create_template_with_checksum_sha1	`Error`	67.41	test_templates.py
test_04_extract_template	`Failure`	128.33	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_10_attachAndDetach_iso	`Failure`	1514.42	test_vm_life_cycle.py
test_06_download_detached_volume	`Error`	318.07	test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup	`Error`	0.00	test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible	`Error`	806.21	test_hostha_kvm.py
test_hostha_kvm_host_degraded	`Error`	8.21	test_hostha_kvm.py

nvazquez · 2021-08-11T04:15:09Z

@blueorangutan test

blueorangutan · 2021-08-11T04:16:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

nvazquez · 2021-08-16T02:59:17Z

@blueorangutan package

blueorangutan · 2021-08-16T03:00:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-08-16T03:53:37Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 875

nvazquez · 2021-08-16T03:57:40Z

@blueorangutan test

blueorangutan · 2021-08-16T03:58:02Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-08-16T16:13:06Z

Trillian test result (tid-1649)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 42125 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4776-t1649-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 86 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_snapshots_with_removed_data_store	`Error`	50.28	test_snapshots.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	593.61	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	570.58	test_vpc_redundant.py
test_01_vpc_site2site_vpn_multiple_options	`Failure`	677.02	test_vpc_vpn.py

DaanHoogland approved these changes Mar 9, 2021

View reviewed changes

DaanHoogland added this to the 4.16.0.0 milestone Mar 24, 2021

DaanHoogland added component:api Severity:Trivial type:enhancement labels Mar 24, 2021

GabrielBrascher reviewed Mar 24, 2021

View reviewed changes

server/src/main/java/com/cloud/api/ApiResponseHelper.java Outdated Show resolved Hide resolved

server/src/main/java/com/cloud/api/query/dao/UserVmJoinDaoImpl.java Outdated Show resolved Hide resolved

ravening force-pushed the vm-traffic-data branch from 73a6816 to 5883f79 Compare July 29, 2021 07:39

apache deleted a comment from blueorangutan Jul 29, 2021

nvazquez approved these changes Jul 30, 2021

View reviewed changes

RodrigoDLopez reviewed Jul 30, 2021

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/response/UserVmResponse.java Outdated Show resolved Hide resolved

RodrigoDLopez approved these changes Jul 30, 2021

View reviewed changes

ravening and others added 3 commits August 9, 2021 16:01

Add sent and receivedbytes to listNetworks and listVirtualMachines.

4376d21

Display the traffic data in networks and vm api response

follow code convention

f67c28b

remove final keyword

8ae5ca9

ravening force-pushed the vm-traffic-data branch from 359b0d7 to 8ae5ca9 Compare August 9, 2021 14:03

nvazquez merged commit 00e014c into apache:main Aug 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add sent and received bytes to listNetworks and listVirtualMachines. #4776

Add sent and received bytes to listNetworks and listVirtualMachines. #4776

ravening commented Mar 9, 2021 •

edited

DaanHoogland left a comment

ravening commented Mar 24, 2021

blueorangutan commented Mar 24, 2021

GabrielBrascher left a comment

blueorangutan commented Mar 24, 2021

davidjumani commented Mar 25, 2021

nvazquez commented Jun 23, 2021

blueorangutan commented Jul 29, 2021

DaanHoogland commented Jul 29, 2021

blueorangutan commented Jul 29, 2021

blueorangutan commented Jul 30, 2021

nvazquez left a comment

RodrigoDLopez left a comment •

edited

DaanHoogland commented Jul 30, 2021

nvazquez commented Aug 2, 2021

ravening commented Aug 9, 2021

nvazquez commented Aug 9, 2021

blueorangutan commented Aug 9, 2021

blueorangutan commented Aug 9, 2021

nvazquez commented Aug 9, 2021

blueorangutan commented Aug 9, 2021

blueorangutan commented Aug 11, 2021

nvazquez commented Aug 11, 2021

blueorangutan commented Aug 11, 2021

nvazquez commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

nvazquez commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

Add sent and received bytes to listNetworks and listVirtualMachines. #4776

Add sent and received bytes to listNetworks and listVirtualMachines. #4776

Conversation

ravening commented Mar 9, 2021 • edited

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

DaanHoogland left a comment

Choose a reason for hiding this comment

ravening commented Mar 24, 2021

blueorangutan commented Mar 24, 2021

GabrielBrascher left a comment

Choose a reason for hiding this comment

blueorangutan commented Mar 24, 2021

davidjumani commented Mar 25, 2021

nvazquez commented Jun 23, 2021

blueorangutan commented Jul 29, 2021

DaanHoogland commented Jul 29, 2021

blueorangutan commented Jul 29, 2021

blueorangutan commented Jul 30, 2021

nvazquez left a comment

Choose a reason for hiding this comment

RodrigoDLopez left a comment • edited

Choose a reason for hiding this comment

DaanHoogland commented Jul 30, 2021

nvazquez commented Aug 2, 2021

ravening commented Aug 9, 2021

nvazquez commented Aug 9, 2021

blueorangutan commented Aug 9, 2021

blueorangutan commented Aug 9, 2021

nvazquez commented Aug 9, 2021

blueorangutan commented Aug 9, 2021

blueorangutan commented Aug 11, 2021

nvazquez commented Aug 11, 2021

blueorangutan commented Aug 11, 2021

nvazquez commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

nvazquez commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

blueorangutan commented Aug 16, 2021

ravening commented Mar 9, 2021 •

edited

RodrigoDLopez left a comment •

edited