Use TrafficType to determine nic id #14

borisroman · 2016-04-14T17:52:31Z

Fix for KVM, where the public ip would be added to the guest network. Reason is because it used 'broadcastUrl' as an index, which turned out not to be unique and was overwritten which resulted in the public ip address being added to the wrong interface (see screenshots).

remibergsma · 2016-04-14T17:54:28Z

Public ip plugged OK:

Private gw also works fine:

remibergsma · 2016-04-14T17:55:54Z

This was the problem observed:

remibergsma · 2016-04-16T09:06:09Z

This works fine for redundant VPC and isolated networks, but it seems to break creation of single VPCs (on KVM/VLAN) somehow:

Creating Admin Account for Domain ID ==> 45175887-03b0-11e6-8f33-5254001daa61
Starting test for single VPC
Enabling the VPC offering created
Creating VPC with offering ID d3fe5bab-dd0a-4081-a598-28e2853dbbe8
Creating a VPC in the account: test-TestRouterRules-test_01_vpc_password_service_single_vpc-ZFJN99
Traceback (most recent call last):
  File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
    testMethod()
  File "/data/git/cs1/cosmic/cosmic-core/test/integration/smoke/test_password_server.py", line 246, in test_01_vpc_password_service_single_vpc
    self.perform_password_service_tests(vpc_off)
  File "/data/git/cs1/cosmic/cosmic-core/test/integration/smoke/test_password_server.py", line 273, in perform_password_service_tests
    vpc_1 = self.createVPC(vpc_off, cidr = '10.0.0.0/16')
  File "/data/git/cs1/cosmic/cosmic-core/test/integration/smoke/test_password_server.py", line 419, in createVPC
    self.fail('Unable to create VPC due to %s ' % e)
  File "/usr/lib64/python2.7/unittest/case.py", line 450, in fail
    raise self.failureException(msg)
AssertionError: Unable to create VPC due to Job failed: {jobprocstatus : 0, created : u'2016-04-16T10:59:41+0200', jobresult : {errorcode : 530, errortext : u'Failed to create VPC'}, cmd : u'org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin', userid : u'45177167-03b0-11e6-8f33-5254001daa61', jobstatus : 2, jobid : u'a8f8379b-6d6e-4021-b6c6-6ecc42d4f626', jobresultcode : 530, jobresulttype : u'object', jobinstancetype : u'None', accountid : u'451764aa-03b0-11e6-8f33-5254001daa61'} 
=== TestName: test_01_vpc_password_service_single_vpc | Status : FAILED ===

Caused by:

":{"result":true,"details":"success","wait":0}},{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: null","null - failed: [INFO] update_config.py :: Processing incoming file => ip_associations.json.36ca9375-f133-48b5-9e5d-008c9e3a2684[INFO] Processing JSON file ip_associations.json.36ca9375-f133-48b5-9e5d-008c9e3a2684Traceback (most recent call last):  File \"/opt/cloud/bin/update_config.py\", line 138, in <module>    process_file()  File \"/opt/cloud/bin/update_config.py\", line 55, in process_file    qf.load(None)  File \"/opt/cloud/bin/merge.py\", line 268, in load    proc = updateDataBag(self)  File \"/opt/cloud/bin/merge.py\", line 94, in __init__    self.process()  File \"/opt/cloud/bin/merge.py\", line 106, in process    dbag = self.processIP(self.db.getDataBag())  File \"/opt/cloud/bin/merge.py\", line 196, in processIP    dbag = cs_ip.merge(dbag, ip)  File \"/opt/cloud/bin/cs_ip.py\", line 32, in merge    ip['device'] = 'eth' + str(ip['nic_dev_id'])KeyError: 'nic_dev_id'"],"result":false,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }

It seems nic_dev_id is missing. I cannot access the router to confirm, it gets killed too fast.

I do see that it is present in a redundant VPC (which works fine).

redundant:
root@r-4-VM:/var/cache/cloud/processed# zcat ip_associations.json.8a460784-ecd7-4ef1-8ecf-ab66d0cadb50.gz | python -m json.tool
{
    "ip_address": [
        {
            "add": true, 
            "first_i_p": true, 
            "gateway": "192.168.23.1", 
            "netmask": "255.255.255.0", 
            "new_nic": false, 
            "nic_dev_id": 1, 
            "one_to_one_nat": false, 
            "public_ip": "192.168.23.4", 
            "source_nat": true, 
            "vif_mac_address": "06:46:8e:00:00:18"
        }
    ], 
    "type": "ips"
}

remibergsma · 2016-04-19T08:22:42Z

Test results from Jenkins:
https://beta-jenkins.mcc.schubergphilis.com/job/cosmic/job/0002-tracking-repo-branch-build/14/

One test failed, rerun that and it passed:

[root@cs1 integration]# nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
> smoke/test_vpc_redundant.py \
>

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /tmp//MarvinLogs//Apr_19_2016_07_58_24_OIMCAI. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
Creating a VPC offering..
Enabling the VPC offering created
Creating a VPC network in the account: test-TestVPCRedundancy-test_05_rvpc_multi_tiers-M7KHFT
Starting test_05_rvpc_multi_tiers
Create NetworkOffering
Created and Enabled NetworkOffering
Adding Network={'netmask': '255.255.255.0', 'displaytext': 'Test Network', 'name': 'NETWORK-10.1.1.1'}
Created network with ID: ccac24db-8f9c-497e-906d-3214d1054875
Creating VM in network=NETWORK-10.1.1.1
Created VM=f1443b34-d503-4cd7-9ec6-2bf4d961188c in network=NETWORK-10.1.1.1
Create NetworkOffering
Created and Enabled NetworkOffering
Adding Network={'netmask': '255.255.255.0', 'displaytext': 'Test Network', 'name': 'NETWORK-10.1.2.1'}
Created network with ID: a47392ee-ac7e-4d10-9072-510c383b56a9
Creating VM in network=NETWORK-10.1.2.1
Created VM=8042dcd2-3780-4918-a082-fec389dd00d1 in network=NETWORK-10.1.2.1
Create NetworkOffering
Created and Enabled NetworkOffering
Adding Network={'netmask': '255.255.255.0', 'displaytext': 'Test Network', 'name': 'NETWORK-10.1.3.1'}
Created network with ID: e6fffd97-1657-46b9-a82c-0208de026d1f
Creating VM in network=NETWORK-10.1.3.1
Created VM=a1eabb84-5822-4401-ac9d-7ef5ec452377 in network=NETWORK-10.1.3.1
[SSH] Trying SSH Connection to host 192.168.22.22 on port 22 as user root. RetryCount: 60
[SSH] Connection to host 192.168.22.22 on port 22 is SUCCESSFUL
[SSH] Executing command via host 192.168.22.22: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.1.203 sh /opt/cloud/bin/checkrouter.sh  Output: [u'Status: BACKUP']
[SSH] Trying SSH Connection to host 192.168.22.22 on port 22 as user root. RetryCount: 60
[SSH] Connection to host 192.168.22.22 on port 22 is SUCCESSFUL
[SSH] Executing command via host 192.168.22.22: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.3.143 sh /opt/cloud/bin/checkrouter.sh  Output: [u'Status: MASTER']
Associating public IP for network: NETWORK-10.1.1.1
Associated 192.168.23.9 with network ccac24db-8f9c-497e-906d-3214d1054875
Creating NAT rule in network for vm with public IP
Adding NetworkACL rules to make NAT rule accessible
nwacl_nat={'startport': u'22', 'endport': u'22', 'aclid': u'dcc06645-bf00-46d0-8ec0-d67b09a36b6e', 'fordisplay': True, 'cidrlist': u'0.0.0.0/0', 'tags': [], 'number': 1, 'traffictype': u'Ingress', 'state': u'Active', 'action': u'Allow', 'protocol': u'tcp', 'id': u'cbbe2ea8-3317-40dd-83fa-fe0287c139e4'}
Associating public IP for network: NETWORK-10.1.2.1
Associated 192.168.23.10 with network a47392ee-ac7e-4d10-9072-510c383b56a9
Creating NAT rule in network for vm with public IP
Adding NetworkACL rules to make NAT rule accessible
nwacl_nat={'startport': u'22', 'endport': u'22', 'aclid': u'd11acdc3-aa51-488a-b0aa-61e7330e9f37', 'fordisplay': True, 'cidrlist': u'0.0.0.0/0', 'tags': [], 'number': 1, 'traffictype': u'Ingress', 'state': u'Active', 'action': u'Allow', 'protocol': u'tcp', 'id': u'6543840a-ffdd-4f54-9f2d-2f7103eff339'}
Associating public IP for network: NETWORK-10.1.3.1
Associated 192.168.23.11 with network e6fffd97-1657-46b9-a82c-0208de026d1f
Creating NAT rule in network for vm with public IP
Adding NetworkACL rules to make NAT rule accessible
nwacl_nat={'startport': u'22', 'endport': u'22', 'aclid': u'73c9b90b-72db-404a-8891-3c2ec1c2cebb', 'fordisplay': True, 'cidrlist': u'0.0.0.0/0', 'tags': [], 'number': 1, 'traffictype': u'Ingress', 'state': u'Active', 'action': u'Allow', 'protocol': u'tcp', 'id': u'd5c5981c-0707-418c-a80e-13290f80e7a6'}
Checking if we can SSH into VM=VM-f1443b34-d503-4cd7-9ec6-2bf4d961188c on public_ip=192.168.23.9 (False)
[SSH] Trying SSH Connection to host 192.168.23.9 on port 22 as user root. RetryCount: 5
[SSH] Connection to host 192.168.23.9 on port 22 is SUCCESSFUL
SSH into VM=VM-f1443b34-d503-4cd7-9ec6-2bf4d961188c on public_ip=192.168.23.9 is successful
Checking if we can SSH into VM=VM-8042dcd2-3780-4918-a082-fec389dd00d1 on public_ip=192.168.23.10 (False)
[SSH] Trying SSH Connection to host 192.168.23.10 on port 22 as user root. RetryCount: 5
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/marvin/sshClient.py", line 120, in createConnection
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 311, in connect
    raise NoValidConnectionsError(errors)
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on  or 192.168.23.10
[SSH] Trying SSH Connection to host 192.168.23.10 on port 22 as user root. RetryCount: 4
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/marvin/sshClient.py", line 120, in createConnection
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 311, in connect
    raise NoValidConnectionsError(errors)
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on  or 192.168.23.10
[SSH] Trying SSH Connection to host 192.168.23.10 on port 22 as user root. RetryCount: 3
[SSH] Connection to host 192.168.23.10 on port 22 is SUCCESSFUL
SSH into VM=VM-8042dcd2-3780-4918-a082-fec389dd00d1 on public_ip=192.168.23.10 is successful
Checking if we can SSH into VM=VM-a1eabb84-5822-4401-ac9d-7ef5ec452377 on public_ip=192.168.23.11 (False)
[SSH] Trying SSH Connection to host 192.168.23.11 on port 22 as user root. RetryCount: 5
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/marvin/sshClient.py", line 120, in createConnection
    timeout=self.timeout)
  File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 311, in connect
    raise NoValidConnectionsError(errors)
NoValidConnectionsError: [Errno None] Unable to connect to port 22 on  or 192.168.23.11
[SSH] Trying SSH Connection to host 192.168.23.11 on port 22 as user root. RetryCount: 4
[SSH] Connection to host 192.168.23.11 on port 22 is SUCCESSFUL
SSH into VM=VM-a1eabb84-5822-4401-ac9d-7ef5ec452377 on public_ip=192.168.23.11 is successful
router.redundant.vrrp.interval is ==> [{category : u'Advanced', name : u'router.redundant.vrrp.interval', value : u'1', description : u'seconds between VRRP broadcast. It would 3 times broadcast fail to trigger fail-over mechanism of redundant router'}]
[SSH] Trying SSH Connection to host 192.168.22.22 on port 22 as user root. RetryCount: 60
[SSH] Connection to host 192.168.22.22 on port 22 is SUCCESSFUL
[SSH] Executing command via host 192.168.22.22: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.1.203 sh /opt/cloud/bin/checkrouter.sh  Output: [u'Status: BACKUP']
[SSH] Trying SSH Connection to host 192.168.22.22 on port 22 as user root. RetryCount: 60
[SSH] Connection to host 192.168.22.22 on port 22 is SUCCESSFUL
[SSH] Executing command via host 192.168.22.22: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.3.143 sh /opt/cloud/bin/checkrouter.sh  Output: [u'Status: MASTER']
Checking if we can SSH into VM=VM-8042dcd2-3780-4918-a082-fec389dd00d1 on public_ip=192.168.23.10 (False)
[SSH] Trying SSH Connection to host 192.168.23.10 on port 22 as user root. RetryCount: 5
[SSH] Connection to host 192.168.23.10 on port 22 is SUCCESSFUL
SSH into VM=VM-8042dcd2-3780-4918-a082-fec389dd00d1 on public_ip=192.168.23.10 is successful
Checking if we can SSH into VM=VM-a1eabb84-5822-4401-ac9d-7ef5ec452377 on public_ip=192.168.23.11 (False)
[SSH] Trying SSH Connection to host 192.168.23.11 on port 22 as user root. RetryCount: 5
[SSH] Connection to host 192.168.23.11 on port 22 is SUCCESSFUL
SSH into VM=VM-a1eabb84-5822-4401-ac9d-7ef5ec452377 on public_ip=192.168.23.11 is successful
[SSH] Trying SSH Connection to host 192.168.22.22 on port 22 as user root. RetryCount: 60
[SSH] Connection to host 192.168.22.22 on port 22 is SUCCESSFUL
[SSH] Executing command via host 192.168.22.22: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.1.203 sh /opt/cloud/bin/checkrouter.sh  Output: [u'Status: BACKUP']
[SSH] Trying SSH Connection to host 192.168.22.22 on port 22 as user root. RetryCount: 60
[SSH] Connection to host 192.168.22.22 on port 22 is SUCCESSFUL
[SSH] Executing command via host 192.168.22.22: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.3.143 sh /opt/cloud/bin/checkrouter.sh  Output: [u'Status: MASTER']
Checking if we can SSH into VM=VM-8042dcd2-3780-4918-a082-fec389dd00d1 on public_ip=192.168.23.10 (False)
[SSH] Trying SSH Connection to host 192.168.23.10 on port 22 as user root. RetryCount: 5
[SSH] Connection to host 192.168.23.10 on port 22 is SUCCESSFUL
SSH into VM=VM-8042dcd2-3780-4918-a082-fec389dd00d1 on public_ip=192.168.23.10 is successful
Destroying routers
Stopping router 2fbd0566-c762-4aa0-b04e-b1e5924075bc
Stopping router d65a77cb-e1ec-4adc-8ac3-79bb45356f5c
=== TestName: test_05_rvpc_multi_tiers | Status : SUCCESS ===

===final results are now copied to: /tmp//MarvinLogs/test_vpc_redundant_Q8DM7M===

Ping @miguelaferreira @wilderrodrigues @borisroman

Please merge together with MissionCriticalCloudOldRepos/cosmic-core#112

borisroman force-pushed the kvm_nic_id branch from a949e30 to 2a425c7 Compare April 18, 2016 11:35

Use TrafficType to determine nic id

bcd319b

borisroman force-pushed the kvm_nic_id branch from 9b884a4 to bcd319b Compare April 18, 2016 15:02

remibergsma changed the title ~~Use bridge name to determine nic id~~ Use TrafficType to determine nic id Apr 18, 2016

wilderrodrigues merged commit 6810a65 into master Apr 19, 2016

wilderrodrigues deleted the kvm_nic_id branch April 19, 2016 09:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use TrafficType to determine nic id #14

Use TrafficType to determine nic id #14

borisroman commented Apr 14, 2016 •

edited by remibergsma

remibergsma commented Apr 14, 2016

remibergsma commented Apr 14, 2016

remibergsma commented Apr 16, 2016 •

edited

remibergsma commented Apr 19, 2016

Use TrafficType to determine nic id #14

Use TrafficType to determine nic id #14

Conversation

borisroman commented Apr 14, 2016 • edited by remibergsma

remibergsma commented Apr 14, 2016

remibergsma commented Apr 14, 2016

remibergsma commented Apr 16, 2016 • edited

remibergsma commented Apr 19, 2016

borisroman commented Apr 14, 2016 •

edited by remibergsma

remibergsma commented Apr 16, 2016 •

edited