Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iptables speedup #2083

Closed
wants to merge 8 commits into from
Closed

Iptables speedup #2083

wants to merge 8 commits into from

Conversation

DaanHoogland
Copy link
Contributor

No description provided.

@DaanHoogland DaanHoogland changed the base branch from master to 4.9 May 6, 2017 19:18
@DaanHoogland DaanHoogland reopened this May 6, 2017
@karuturi karuturi added this to the 4.10.0.0 milestone May 8, 2017
@DaanHoogland
Copy link
Contributor Author

DaanHoogland commented May 8, 2017

@karuturi I don't think it needs CI (yet), same as with #2084

Copy link
Contributor Author

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the code commited must be replaced with code under compatible license!

@@ -2,6 +2,22 @@
#
# -*- coding: utf-8 -*-
#
# Licensed to the Apache Software Foundation (ASF) under one
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a big crime of mine. merge must not happen

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mailed the author of iptables_convert , Johannes Hubertz johannes@hubertz.de, @sl0, and he agreed to publish his code under apache license version2. so I am forgiven.
He has a focus on security and python iptables utilities so I will work with him on the best integration into CloudStack of his code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DaanHoogland any update on this?

@DaanHoogland DaanHoogland modified the milestones: 4.9.3, 4.10.0.0 May 8, 2017
@DaanHoogland
Copy link
Contributor Author

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-714

@DaanHoogland
Copy link
Contributor Author

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@sl0
Copy link

sl0 commented May 12, 2017

Thank you very much for considering to use my iptables_converter.py in Apache CloudStack. Originally it was licensed GPLv3, but I agree to have a copy here under Apache Foundation License, as that is a Free Software License as well.
Have fun!
sl0

@blueorangutan
Copy link

Trillian test result (tid-1084)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 66915 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2083-t1084-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermitten failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermitten failure detected: /marvin/tests/smoke/test_network.py
Intermitten failure detected: /marvin/tests/smoke/test_password_server.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermitten failure detected: /marvin/tests/smoke/test_router_dns.py
Intermitten failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermitten failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermitten failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Test completed. 35 look ok, 14 have error(s)

Test Result Time (s) Test File
test_02_VPC_default_routes Failure 753.65 test_vpc_router_nics.py
test_01_VPC_nics_after_destroy Failure 743.51 test_vpc_router_nics.py
test_05_rvpc_multi_tiers Failure 335.62 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Failure 273.83 test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers Failure 304.73 test_vpc_redundant.py
test_02_redundant_VPC_default_routes Failure 821.43 test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 361.23 test_vpc_redundant.py
test_02_attach_volume Failure 815.75 test_volumes.py
test_01_create_volume Failure 831.15 test_volumes.py
test_10_attachAndDetach_iso Failure 831.11 test_vm_life_cycle.py
test_04_change_offering_small Failure 946.79 test_service_offerings.py
test_router_dns_guestipquery Failure 340.65 test_router_dns.py
test_router_dhcphosts Failure 230.75 test_router_dhcphosts.py
test_04_rvpc_privategw_static_routes Failure 1021.78 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Failure 850.76 test_privategw_acl.py
test_02_vpc_privategw_static_routes Failure 815.11 test_privategw_acl.py
test_isolate_network_password_server Failure 235.92 test_password_server.py
test_reboot_router Failure 503.34 test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule Failure 831.84 test_network.py
test_network_rules_acquired_public_ip_2_nat_rule Failure 826.80 test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule Failure 828.07 test_network.py
test_02_port_fwd_on_non_src_nat Failure 826.27 test_network.py
test_01_port_fwd_on_src_nat Failure 821.14 test_network.py
test_assign_and_removal_lb Failure 220.50 test_loadbalance.py
test_02_create_lb_rule_non_nat Failure 220.59 test_loadbalance.py
test_01_create_lb_rule_src_nat Failure 220.63 test_loadbalance.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80 Failure 293.73 test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80 Failure 213.10 test_internal_lb.py
test_01_vpc_site2site_vpn Error 262.74 test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn Error 378.52 test_vpc_vpn.py
test_05_rvpc_multi_tiers Error 617.90 test_vpc_redundant.py
test_02_list_snapshots_with_removed_data_store Error 0.04 test_snapshots.py
ContextSuite context=TestRouterDHCPHosts>:teardown Error 276.16 test_router_dhcphosts.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces Error 232.42 test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces Error 172.05 test_internal_lb.py
test_01_vpc_remote_access_vpn Success 71.10 test_vpc_vpn.py
test_09_delete_detached_volume Success 156.61 test_volumes.py
test_08_resize_volume Success 156.44 test_volumes.py
test_07_resize_fail Success 161.46 test_volumes.py
test_06_download_detached_volume Success 156.54 test_volumes.py
test_05_detach_volume Success 150.71 test_volumes.py
test_04_delete_attached_volume Success 151.31 test_volumes.py
test_03_download_attached_volume Success 156.27 test_volumes.py
test_deploy_vm_multiple Success 292.96 test_vm_life_cycle.py
test_deploy_vm Success 0.03 test_vm_life_cycle.py
test_advZoneVirtualRouter Success 0.02 test_vm_life_cycle.py
test_09_expunge_vm Success 125.18 test_vm_life_cycle.py
test_08_migrate_vm Success 40.96 test_vm_life_cycle.py
test_07_restore_vm Success 0.09 test_vm_life_cycle.py
test_06_destroy_vm Success 130.91 test_vm_life_cycle.py
test_03_reboot_vm Success 125.89 test_vm_life_cycle.py
test_02_start_vm Success 10.22 test_vm_life_cycle.py
test_01_stop_vm Success 35.31 test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName Success 95.90 test_templates.py
test_08_list_system_templates Success 0.04 test_templates.py
test_07_list_public_templates Success 0.05 test_templates.py
test_05_template_permissions Success 0.10 test_templates.py
test_04_extract_template Success 5.17 test_templates.py
test_03_delete_template Success 5.12 test_templates.py
test_02_edit_template Success 90.14 test_templates.py
test_01_create_template Success 85.80 test_templates.py
test_10_destroy_cpvm Success 161.61 test_ssvm.py
test_09_destroy_ssvm Success 163.77 test_ssvm.py
test_08_reboot_cpvm Success 131.63 test_ssvm.py
test_07_reboot_ssvm Success 133.68 test_ssvm.py
test_06_stop_cpvm Success 131.78 test_ssvm.py
test_05_stop_ssvm Success 163.99 test_ssvm.py
test_04_cpvm_internals Success 1.24 test_ssvm.py
test_03_ssvm_internals Success 3.40 test_ssvm.py
test_02_list_cpvm_vm Success 0.14 test_ssvm.py
test_01_list_sec_storage_vm Success 0.14 test_ssvm.py
test_01_snapshot_root_disk Success 11.15 test_snapshots.py
test_03_delete_service_offering Success 0.04 test_service_offerings.py
test_02_edit_service_offering Success 0.05 test_service_offerings.py
test_01_create_service_offering Success 0.08 test_service_offerings.py
test_02_sys_template_ready Success 0.14 test_secondary_storage.py
test_01_sys_vm_start Success 0.21 test_secondary_storage.py
test_09_reboot_router Success 40.38 test_routers.py
test_08_start_router Success 35.32 test_routers.py
test_07_stop_router Success 10.17 test_routers.py
test_06_router_advanced Success 0.08 test_routers.py
test_05_router_basic Success 0.04 test_routers.py
test_04_restart_network_wo_cleanup Success 5.71 test_routers.py
test_03_restart_network_cleanup Success 60.57 test_routers.py
test_02_router_internal_adv Success 1.25 test_routers.py
test_01_router_internal_basic Success 0.65 test_routers.py
test_router_dns_externalipquery Success 0.11 test_router_dns.py
test_router_dhcp_opts Success 21.87 test_router_dhcphosts.py
test_01_updatevolumedetail Success 0.12 test_resource_detail.py
test_01_reset_vm_on_reboot Success 130.94 test_reset_vm_on_reboot.py
test_createRegion Success 0.04 test_regions.py
test_create_pvlan_network Success 5.24 test_pvlan.py
test_dedicatePublicIpRange Success 0.44 test_public_ip_range.py
test_01_vpc_privategw_acl Success 92.28 test_privategw_acl.py
test_01_primary_storage_nfs Success 35.87 test_primary_storage.py
test_createPortablePublicIPRange Success 15.27 test_portable_publicip.py
test_createPortablePublicIPAcquire Success 15.64 test_portable_publicip.py
test_UpdateStorageOverProvisioningFactor Success 0.14 test_over_provisioning.py
test_oobm_zchange_password Success 30.77 test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership Success 16.46 test_outofbandmanagement.py
test_oobm_issue_power_status Success 5.25 test_outofbandmanagement.py
test_oobm_issue_power_soft Success 15.35 test_outofbandmanagement.py
test_oobm_issue_power_reset Success 15.39 test_outofbandmanagement.py
test_oobm_issue_power_on Success 15.38 test_outofbandmanagement.py
test_oobm_issue_power_off Success 15.36 test_outofbandmanagement.py
test_oobm_issue_power_cycle Success 15.39 test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones Success 77.69 test_outofbandmanagement.py
test_oobm_enable_feature_valid Success 5.21 test_outofbandmanagement.py
test_oobm_enable_feature_invalid Success 0.13 test_outofbandmanagement.py
test_oobm_disable_feature_valid Success 0.20 test_outofbandmanagement.py
test_oobm_disable_feature_invalid Success 0.16 test_outofbandmanagement.py
test_oobm_configure_invalid_driver Success 0.10 test_outofbandmanagement.py
test_oobm_configure_default_driver Success 0.12 test_outofbandmanagement.py
test_oobm_background_powerstate_sync Success 18.46 test_outofbandmanagement.py
test_extendPhysicalNetworkVlan Success 15.37 test_non_contigiousvlan.py
test_01_nic Success 439.75 test_nic.py
test_releaseIP Success 258.19 test_network.py
test_public_ip_user_account Success 10.27 test_network.py
test_public_ip_admin_account Success 40.28 test_network.py
test_delete_account Success 282.88 test_network.py
test_nic_secondaryip_add_remove Success 283.38 test_multipleips_per_nic.py
test_list_zones_metrics Success 0.28 test_metrics_api.py
test_list_volumes_metrics Success 5.51 test_metrics_api.py
test_list_vms_metrics Success 222.19 test_metrics_api.py
test_list_pstorage_metrics Success 0.40 test_metrics_api.py
test_list_infrastructure_metrics Success 0.45 test_metrics_api.py
test_list_hosts_metrics Success 0.41 test_metrics_api.py
test_list_clusters_metrics Success 0.35 test_metrics_api.py
login_test_saml_user Success 19.37 test_login.py
test_03_list_snapshots Success 0.09 test_list_ids_parameter.py
test_02_list_templates Success 0.04 test_list_ids_parameter.py
test_01_list_volumes Success 0.03 test_list_ids_parameter.py
test_07_list_default_iso Success 0.08 test_iso.py
test_05_iso_permissions Success 0.12 test_iso.py
test_04_extract_Iso Success 5.24 test_iso.py
test_03_delete_iso Success 95.20 test_iso.py
test_02_edit_iso Success 0.08 test_iso.py
test_01_create_iso Success 21.12 test_iso.py
test_dedicateGuestVlanRange Success 10.32 test_guest_vlan_range.py
test_UpdateConfigParamWithScope Success 0.15 test_global_settings.py
test_rolepermission_lifecycle_update Success 6.33 test_dynamicroles.py
test_rolepermission_lifecycle_list Success 6.05 test_dynamicroles.py
test_rolepermission_lifecycle_delete Success 5.95 test_dynamicroles.py
test_rolepermission_lifecycle_create Success 6.01 test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates Success 6.17 test_dynamicroles.py
test_role_lifecycle_update_role_inuse Success 5.97 test_dynamicroles.py
test_role_lifecycle_update Success 6.06 test_dynamicroles.py
test_role_lifecycle_list Success 6.05 test_dynamicroles.py
test_role_lifecycle_delete Success 5.98 test_dynamicroles.py
test_role_lifecycle_create Success 6.07 test_dynamicroles.py
test_role_inuse_deletion Success 6.14 test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers Success 8.58 test_dynamicroles.py
test_role_account_acls Success 8.59 test_dynamicroles.py
test_default_role_deletion Success 6.08 test_dynamicroles.py
test_04_create_fat_type_disk_offering Success 0.23 test_disk_offerings.py
test_03_delete_disk_offering Success 0.05 test_disk_offerings.py
test_02_edit_disk_offering Success 0.08 test_disk_offerings.py
test_02_create_sparse_type_disk_offering Success 0.07 test_disk_offerings.py
test_01_create_disk_offering Success 0.11 test_disk_offerings.py
test_deployvm_userdispersing Success 76.16 test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated Success 20.67 test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit Success 60.76 test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post Success 10.58 test_deploy_vm_with_userdata.py
test_deployvm_userdata Success 55.84 test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize Success 5.99 test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize Success 6.00 test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize Success 343.62 test_deploy_vm_root_resize.py
test_deploy_vm_from_iso Success 222.64 test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroup Success 106.44 test_affinity_groups.py
test_03_delete_vm_snapshots Skipped 0.00 test_vm_snapshots.py
test_02_revert_vm_snapshots Skipped 0.00 test_vm_snapshots.py
test_01_create_vm_snapshots Skipped 0.00 test_vm_snapshots.py
test_06_copy_template Skipped 0.00 test_templates.py
test_static_role_account_acls Skipped 0.02 test_staticroles.py
test_01_scale_vm Skipped 0.00 test_scale_vm.py
test_01_primary_storage_iscsi Skipped 0.04 test_primary_storage.py
test_06_copy_iso Skipped 0.00 test_iso.py
test_deploy_vgpu_enabled_vm Skipped 0.01 test_deploy_vgpu_enabled_vm.py

@DaanHoogland
Copy link
Contributor Author

a lot of errors. need to investigate. see if i screwed up something during rebasing

@rohityadavcloud
Copy link
Member

needs review on failing tests, I'll review once the issues are fixed. thanks.

@rohityadavcloud
Copy link
Member

rohityadavcloud commented Jul 22, 2017

@DaanHoogland is this still valid, should we consider? The branch has conflicts and need to be rebased.

@rohityadavcloud
Copy link
Member

@DaanHoogland ping

@DaanHoogland
Copy link
Contributor Author

I have no idea nor time to look at those errors. we'll have to rebase and target this for master

@rohityadavcloud
Copy link
Member

Thanks @sl0, can you re-publish the source file under Apache License 2.0 in your repo that we can use, or send to us as a pull request?

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@sl0
Copy link

sl0 commented Aug 20, 2017 via email

@rohityadavcloud
Copy link
Member

Hello Johannes ( @sl0 ),
Thank you for replying. Sorry for the confusion, I'm not requesting you to move towards a new license but dual-license your work, i.e. both your GPL and Apache license. It would be best if you do that in your repository so we can point to your work in case of any issues. One way to do that is to include the Apache License in your repo's root folder and include the Apache license on all the file headers like it's been done in this PR. You may, for example, look at https://github.com/facebook/rocksdb/tree/master and others who have adopted a dual-license approach. Thanks a lot for the conversation and your swift reply.

Regards, Rohit (@rhtyd).

@DaanHoogland
Copy link
Contributor Author

@kiwiflyer this has gone completely from my radar, sorry. I would like to see it in, but have no time to look at it for a while. It is not my code so I'd have to study on it. @sl0's module as a pypi thingy makes sense but to us it will require a refactor of the code, removing the code copy and adding the install in the systemvm build.

@agx
Copy link

agx commented Oct 12, 2017

@DaanHoogland from what I can see to use iptables-conv verbatim we'd mostly need to

  • not sys.exit(1) from the module
  • move to using logger instead of print (default output can remain unchanged by configuring the logger appropriately)
  • be able to write a file instead of stdout (stdout can stay the default when run as a script)

Did I miss s.th.? @sl0 does this sound reasonable?

@sl0
Copy link

sl0 commented Nov 17, 2017

Hellllo @DaanHoogland and @rhtyd ,
a new version 0.910 of iptables-converter is available. Yesterday it was released at pypi: https://pypi.python.org/pypi/iptables-converter/0.9.10 Now it ships as a python-module with entry-points. Some documentation is prepared: https://conv.readthedocs.io/en/latest/
Some more tests cover 97% now and flake8 silence are added as well. python 2.7, 3.5 and 3.6 work with tox as testrunner usind py.test . Anything missing? Thank you very much for your answer in advance
Kind regards from Cologne
Johannes
(sl0)

Boris Schrijver and others added 8 commits November 23, 2017 14:02
…tables numerous times

 Conflicts:
	systemvm/patches/debian/config/opt/cloud/bin/cs/CsAddress.py
	systemvm/patches/debian/config/opt/cloud/bin/cs/CsNetfilter.py
 Conflicts:
	systemvm/patches/debian/config/opt/cloud/bin/cs/CsDhcp.py
 Conflicts:
	systemvm/patches/debian/config/opt/cloud/bin/configure.py
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>
@rohityadavcloud
Copy link
Member

Thanks @sl0 I've pulled the updated release, we'll work on refactoring it if needed.
@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1288

@rohityadavcloud
Copy link
Member

Related for reference: https://github.com/apache/cloudstack/pull/1400/files

@rohityadavcloud rohityadavcloud modified the milestones: 4.9.3, 4.11 Dec 10, 2017
@rohityadavcloud rohityadavcloud removed this from the 4.11 milestone Jan 2, 2018
@rohityadavcloud
Copy link
Member

Due to lack of activity, I've removed the 4.11 milestone. We can re-engage and work on this for a future minor 4.11.x release.

@borisstoyanov
Copy link
Contributor

@DaanHoogland do you think this would make it in 4.11.1? if not we can drop it from the list perhaps?

@DaanHoogland DaanHoogland removed this from the 4.11.1.0 milestone Apr 20, 2018
@rohityadavcloud
Copy link
Member

@borisstoyanov I think let's keep all the tickets marked with 4.11.1.0 as it is. I initially planned to do the iptables speeup changes as part of my vr downtime PR, depending on time/scope. We may bump this to 4.11.2.0 milestone if this does not make into 4.11.1.0

@rohityadavcloud rohityadavcloud added this to the 4.11.1.0 milestone Apr 24, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants