Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added recursive fetch of child domains for listUsageRecords API call #4717

Merged
merged 10 commits into from Apr 10, 2021

Conversation

Spaceman1984
Copy link
Contributor

@Spaceman1984 Spaceman1984 commented Feb 22, 2021

Description

When calling the listUasageRecords API records per domain are fetched recursively. This is not the case if you specify a domain id.

This PR adds a new parameter to enable fetching records recursively (isRecursive) when passing the domain id.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Screenshots (if appropriate):

How Has This Been Tested?

This has been tested by calling the API with different parameters. With/without a domain id, with/without isRecursive and with/without an account id.
I have tested by calling the API logged in as a root admin, domain admin, and a user.

@Spaceman1984
Copy link
Contributor Author

@blueorangutan package

@Spaceman1984 Spaceman1984 marked this pull request as draft February 22, 2021 07:00
@Spaceman1984 Spaceman1984 changed the title Added recursive fetch of domains for listUsageRecords API call Added recursive fetch of child domains for listUsageRecords API call Feb 22, 2021
@Spaceman1984 Spaceman1984 marked this pull request as ready for review February 24, 2021 06:56
Copy link
Contributor

@shwstppr shwstppr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some suggestions @Spaceman1984 , please check

Copy link
Contributor

@shwstppr shwstppr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Spaceman1984 I see a few issues. Please see the comments.
I see probable failures or wrong results:

  • Domain admin calling API for an account from its subdomain
  • Domain admin calling API for an account in its domain passed with accountid param while using recursive=true

@shwstppr
Copy link
Contributor

shwstppr commented Mar 1, 2021

@blueorangutan package

@blueorangutan
Copy link

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@Spaceman1984
Copy link
Contributor Author

@blueorangutan package

@Spaceman1984
Copy link
Contributor Author

@blueorangutan test

@rohityadavcloud
Copy link
Member

@Spaceman1984 can you test if it works as before if the domain admin can list usage records for an account within the (a) same domain, (b) a sub-domain, and (c) a different domain (the domain admin is not admin of that domain).

}

// Check if a domain admin is allowed to access the requested account info.
Account account = _accountService.getAccount(accountId);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Spaceman1984 should we do a checkAccess() here as well with the provided caller and account?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rhtyd Using checkAccess() here will result in querying the database for every child domain. I'm querying the database once by asking for all the child domains of the current domain and checking if the requested account belongs to anyone of them.

for (DomainVO domain : domains)
domainIds.add(domain.getId());
sc.addAnd("domainId", SearchCriteria.Op.IN, domainIds.toArray());
if (accountId != -1 && accountId != Account.ACCOUNT_ID_SYSTEM && !isAdmin) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please simplify this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand, how do you mean?

if (cmd.isRecursive()) {
SearchCriteria<DomainVO> sdc = _domainDao.createSearchCriteria();
sdc.addOr("path", SearchCriteria.Op.LIKE, _domainDao.findById(domainId).getPath() + "%");
List<DomainVO> domains = _domainDao.search(sdc, null);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplify, reuse existing DomainDao::findAllChildren

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rhtyd Using findAllChildren here would not pass the top domain.

@Spaceman1984
Copy link
Contributor Author

Spaceman1984 commented Mar 3, 2021

@Spaceman1984 can you test if it works as before if the domain admin can list usage records for an account within the (a) same domain, (b) a sub-domain, and (c) a different domain (the domain admin is not admin of that domain).

@rhtyd I have already tested these scenarios. The (c) case should fail.

@DaanHoogland
Copy link
Contributor

@rhtyd I don't care if it goes in 4.15 or not, as it is a minor enhancement. I wouldn't compare listUsageRecords to other list APIs though. Most testing done and ok. I just want to see another test run and am busy with that.
clgtm btw


// By default users do not have access to this API.
// Adding checks here in case someone changes the default access.
checkUserAccess(cmd, accountId, caller, isNormalUser);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Spaceman1984 is this method call required for domain admin ?

@apache apache deleted a comment from blueorangutan Mar 19, 2021
@apache apache deleted a comment from blueorangutan Mar 23, 2021
@apache apache deleted a comment from blueorangutan Mar 23, 2021
@blueorangutan
Copy link

Trillian test result (tid-232)
Environment: vmware-60u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34736 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4717-t232-vmware-60u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_affinity_groups_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_async_job.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vgpu_enabled_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vms_with_varied_deploymentplanners.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_with_userdata.py
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_domain_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_nested_virtualization.py
Intermittent failure detected: /marvin/tests/smoke/test_network_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_portforwardingrules.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 51 look OK, 35 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestAddVmToSubDomain>:setup Error 90.46 test_accounts.py
test_DeleteDomain Error 85.69 test_accounts.py
test_forceDeleteDomain Failure 94.10 test_accounts.py
test_forceDeleteDomain Error 119.74 test_accounts.py
test_01_user_remove_VM_running Error 85.67 test_accounts.py
test_DeployVmAntiAffinityGroup_in_project Error 88.26 test_affinity_groups_projects.py
test_query_async_job_result Error 141.68 test_async_job.py
ContextSuite context=TestLoadBalance>:setup Error 0.00 test_loadbalance.py
test_3d_gpu_support Error 126.90 test_deploy_vgpu_enabled_vm.py
test_list_vms_metrics Error 90.12 test_metrics_api.py
test_deployvm_firstfit Error 111.93 test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated Error 15.79 test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdispersing Error 22.67 test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata Error 80.06 test_deploy_vm_with_userdata.py
test_deployvm_userdata_post Error 16.62 test_deploy_vm_with_userdata.py
test_network_acl Error 97.39 test_network_acl.py
ContextSuite context=TestRemoteDiagnostics>:setup Error 0.00 test_diagnostics.py
test_delete_account Error 98.64 test_network.py
test_delete_network_while_vm_on_it Error 11.49 test_network.py
test_delete_network_while_vm_on_it Error 12.56 test_network.py
test_deploy_vm_l2network Error 10.44 test_network.py
test_deploy_vm_l2network Error 11.50 test_network.py
test_l2network_restart Error 10.54 test_network.py
test_l2network_restart Error 11.61 test_network.py
ContextSuite context=TestL2Networks>:teardown Error 12.69 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 88.89 test_network.py
ContextSuite context=TestPublicIP>:setup Error 83.78 test_network.py
test_reboot_router Error 91.48 test_network.py
test_releaseIP Error 77.14 test_network.py
ContextSuite context=TestRouterRules>:setup Error 152.24 test_network.py
test_03_deploy_vm_domain_service_offering Error 119.23 test_domain_service_offerings.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 Error 124.28 test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 Error 126.42 test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Error 228.99 test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Error 231.44 test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces Error 137.10 test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces Error 138.51 test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces Error 181.52 test_internal_lb.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces Error 182.89 test_internal_lb.py
test_01_deploy_kubernetes_cluster Failure 94.53 test_kubernetes_clusters.py
test_02_invalid_upgrade_kubernetes_cluster Failure 98.19 test_kubernetes_clusters.py
test_03_deploy_and_upgrade_kubernetes_cluster Failure 97.05 test_kubernetes_clusters.py
test_04_deploy_and_scale_kubernetes_cluster Failure 87.81 test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster Failure 85.75 test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster Failure 77.69 test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster Failure 79.80 test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster Failure 83.83 test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown Error 321.50 test_kubernetes_clusters.py
ContextSuite context=TestListIdsParams>:setup Error 0.00 test_list_ids_parameter.py
test_nic_secondaryip_add_remove Error 87.34 test_multipleips_per_nic.py
test_nested_virtualization_vmware Error 81.99 test_nested_virtualization.py
test_01_create_delete_portforwarding_fornonvpc Error 83.59 test_portforwardingrules.py
test_02_vpc_privategw_static_routes Failure 172.36 test_privategw_acl.py
test_02_vpc_privategw_static_routes Error 176.18 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Failure 166.00 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Error 168.16 test_privategw_acl.py
test_04_rvpc_privategw_static_routes Failure 278.25 test_privategw_acl.py
test_04_rvpc_privategw_static_routes Error 279.42 test_privategw_acl.py
test_09_project_suspend Error 79.97 test_projects.py
test_10_project_activation Error 17.57 test_projects.py
ContextSuite context=TestResetVmOnReboot>:setup Error 0.00 test_reset_vm_on_reboot.py
test_01_so_removal_resource_update Error 79.75 test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup Error 0.00 test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup Error 0.00 test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup Error 0.00 test_router_dnsservice.py
test_02_routervm_iptables_policies Error 79.70 test_routers_iptables_default_policy.py
test_01_single_VPC_iptables_policies Error 80.06 test_routers_iptables_default_policy.py
test_01_single_VPC_iptables_policies Error 99.52 test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true Error 78.78 test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false Error 88.95 test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Error 160.47 test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Error 157.36 test_routers_network_ops.py
test_03_RVR_Network_check_router_state Error 153.58 test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup Error 0.00 test_routers.py
ContextSuite context=TestServiceOfferings>:setup Error 82.25 test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup Error 0.00 test_snapshots.py
ContextSuite context=TestVAppsVM>:setup Error 44.49 test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 156.72 test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Error 172.21 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Failure 157.83 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Error 176.38 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Failure 167.94 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Error 183.44 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Failure 154.81 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Error 172.34 test_vpc_redundant.py
test_05_rvpc_multi_tiers Failure 154.89 test_vpc_redundant.py
test_05_rvpc_multi_tiers Error 170.40 test_vpc_redundant.py
test_01_VPC_nics_after_destroy Failure 87.50 test_vpc_router_nics.py
test_02_VPC_default_routes Failure 87.53 test_vpc_router_nics.py
test_01_redundant_vpc_site2site_vpn Failure 272.62 test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn Error 274.76 test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options Failure 143.23 test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options Error 145.37 test_vpc_vpn.py
test_01_vpc_remote_access_vpn Failure 76.82 test_vpc_vpn.py
test_01_vpc_remote_access_vpn Error 77.89 test_vpc_vpn.py
test_01_vpc_site2site_vpn Failure 132.07 test_vpc_vpn.py
test_01_vpc_site2site_vpn Error 134.22 test_vpc_vpn.py

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 224

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 228

@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 232

@rohityadavcloud
Copy link
Member

@blueorangutan test

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan
Copy link

Trillian test result (tid-247)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36028 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4717-t247-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestKubernetesCluster>:teardown Error 72.98 test_kubernetes_clusters.py
test_01_migrate_VM_and_root_volume Error 67.12 test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks Error 51.10 test_vm_life_cycle.py

@apache apache deleted a comment from blueorangutan Mar 25, 2021
@blueorangutan
Copy link

Trillian test result (tid-262)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43620 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4717-t262-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 83 look OK, 3 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_10_traceroute_in_vr Failure 61.23 test_diagnostics.py
test_03_deploy_and_upgrade_kubernetes_cluster Failure 800.14 test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown Error 83.14 test_kubernetes_clusters.py
ContextSuite context=TestVAppsVM>:setup Error 43.52 test_vm_life_cycle.py

Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm and tested

@rohityadavcloud
Copy link
Member

Is this ready for merging @DaanHoogland @Spaceman1984 ?

@andrijapanicsb
Copy link
Contributor

Not until IR is done.

@blueorangutan
Copy link

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 309

@blueorangutan
Copy link

Trillian test result (tid-331)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38841 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4717-t331-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 84 look OK, 2 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_07_deploy_kubernetes_ha_cluster Failure 3615.24 test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster Failure 0.05 test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster Failure 0.05 test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown Error 42.91 test_kubernetes_clusters.py
test_01_migrate_VM_and_root_volume Error 67.22 test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks Error 50.09 test_vm_life_cycle.py

@rohityadavcloud
Copy link
Member

@andrijapanicsb are you lgtm on this?

@andrijapanicsb
Copy link
Contributor

LGTM

@rohityadavcloud rohityadavcloud merged commit 99a9063 into apache:4.15 Apr 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants