test_vpc_redundant sometimes fails on XCP-ng 7.6-8/8.1 #4224
Comments
|
i.e. listRouters returns 2 VRs, but we expect 1 VR to be returned. |
|
Same kind of failures, from another PR: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4068-t2187-xcpng80.zip |
DaanHoogland
added
component:marvin-tests
component:marvin
and removed
component:marvin-tests
labels
|
@weizhouapache did you find the root-cause fix for this? |
andrijapanicsb
changed the title
|
@DaanHoogland can you advise after your work on marvin tests? |
|
@nvazquez @andrijapanicsb It does seem more a timing issue than a test instability issue. I will put this on my name to have a look see. |
|
Ping @DaanHoogland are you working on this, in recent runs I don't see this failure against xenserver. Though in general the vpc smoketests intermittently fail across all three hypervisors. |
|
@rhtyd I'm not actively researching this but it is a concern. I'll remove my name for now but will keep an eye on. |
|
@nvazquez @sureshanaparti @davidjumani @shwstppr @Pearl1594 @DaanHoogland @weizhouapache - any update on this, who's assigned/working on this? Thanks. |
|
|
I run the test on xcp82 environment 10 times, all are successful. I think this can be closed. |
|
Ok, let's close it until next failures. |
Happens on XCP-ng 7.6-8/8.1 all the time when blueorangutan is invoked (when repeated manually a few times, sometimes it will pass completely, but usually it will fail).
Does NOT happen (as it seems, haven't seen it) with XS 7.1 (and KVM/VMware)
Pasting below the runinfo.txt related parts - these are making it hard to see if the tests are not robust enough, or if there is some kind of failure on the ACS side.
Could someone please look into the root cause of failing tests with redundant VPCs?
2020-07-27 11:00:51,863 - DEBUG - ========Sending GET Cmd : listRouters======= 2020-07-27 11:00:51,900 - DEBUG - Response : [{domain : u'ROOT', domainid : u'3f9835f1-cff3-11ea-a290-1e00800139e7', guestmacaddress : u'02:00:42:e6:00:03', templatename : u'SystemVM Template (XenServer)', scrip tsversion : u'199a9ebcfb29d7f92dc5209461655a32\n', healthchecksfailed : True, linklocalip : u'169.254.168.24', zoneid : u'3b87962d-43e3-43e6-aec8-44e752adb4b2', linklocalmacaddress : u'0e:00:a9:fe:a8:18', linklo calnetworkid : u'8ee75f93-69c3-4972-b87d-196a5c2e84db', linklocalnetmask : u'255.255.0.0', publicmacaddress : u'1e:00:51:00:00:03', id : u'86667242-1455-4c34-9168-b98b0bcbf3e0', networkdomain : u'cs5cloud.intern al', guestnetworkid : u'fa1d1a22-231c-4dd9-b2fb-9560c6f7bed7', hostname : u'pr4068-t2204-xcpng81-xs2', gateway : u'10.1.63.254', publicip : u'10.1.37.223', state : u'Running', version : u'4.15.0', role : u'VIRTU AL_ROUTER', podid : u'bd593c3a-f865-42a8-a4fe-6bcf4e8a8533', serviceofferingid : u'17eb7c2a-3219-4453-ab71-44f5ac1b8dfa', zonename : u'pr4068-t2204-xcpng81', podname : u'Pod1', name : u'r-4-VM', guestnetworkname : u'NETWORK-10.1.2.1', nic : [{networkid : u'8ee75f93-69c3-4972-b87d-196a5c2e84db', macaddress : u'0e:00:a9:fe:a8:18', id : u'6bedde43-5ab2-46d5-9a18-ac640d976991', traffictype : u'Control', netmask : u'255.255 .0.0', ipaddress : u'169.254.168.24', gateway : u'169.254.0.1', isdefault : False}, {networkid : u'55e27b1c-8205-4b3b-be9c-ae113e2a4342', macaddress : u'1e:00:51:00:00:03', isolationuri : u'vlan://7', broadcastu ri : u'vlan://7', traffictype : u'Public', netmask : u'255.255.224.0', id : u'13496314-fa8d-43a2-a30b-d51478d86363', ipaddress : u'10.1.37.223', gateway : u'10.1.63.254', isdefault : True}, {networkid : u'7cbed1 13-ed53-4e45-845f-9e6d5d533175', macaddress : u'02:00:76:6c:00:03', isolationuri : u'vlan://1986', type : u'Isolated', broadcasturi : u'vlan://1986', traffictype : u'Guest', netmask : u'255.255.255.0', ipaddress : u'10.1.1.109', id : u'879e4c7b-d0b3-4dc6-9c14-516cc44a7d8c', networkname : u'NETWORK-10.1.1.1', gateway : u'10.1.1.1', isdefault : False}, {networkid : u'fa1d1a22-231c-4dd9-b2fb-9560c6f7bed7', macaddress : u' 02:00:42:e6:00:03', isolationuri : u'vlan://1992', type : u'Isolated', broadcasturi : u'vlan://1992', traffictype : u'Guest', netmask : u'255.255.255.0', ipaddress : u'10.1.2.113', id : u'a324e4a7-127e-4530-a438 -f1f00bd2169d', networkname : u'NETWORK-10.1.2.1', gateway : u'10.1.2.1', isdefault : False}], publicnetworkid : u'55e27b1c-8205-4b3b-be9c-ae113e2a4342', redundantstate : u'BACKUP', hostid : u'83d38508-d3a1-49f3 -acc1-4a191c1eb0c7', templateid : u'22bea20c-649e-4745-b73b-d44b72f46a16', requiresupgrade : False, publicnetmask : u'255.255.224.0', account : u'test-TestVPCRedundancy-test_01_create_redundant_VPC_2tiers_4VMs_4 IPs_4PF_ACL-UM7Q58', vpcid : u'f6a98c6a-7ca4-4f08-a8bd-b44c55ca2ee7', isredundantrouter : True, created : u'2020-07-27T10:50:57+0000', hypervisor : u'XenServer', dns1 : u'10.2.0.50', vpcname : u'TestVPC-1QKJV0', dns2 : u'8.8.8.8', guestnetmask : u'255.255.255.0', guestipaddress : u'10.1.2.113', serviceofferingname : u'System Offering For Software Router'}, {domain : u'ROOT', domainid : u'3f9835f1-cff3-11ea-a290-1e00800 139e7', guestmacaddress : u'02:00:1e:af:00:02', templatename : u'SystemVM Template (XenServer)', scriptsversion : u'199a9ebcfb29d7f92dc5209461655a32\n', healthchecksfailed : False, linklocalip : u'169.254.73.84' , zoneid : u'3b87962d-43e3-43e6-aec8-44e752adb4b2', linklocalmacaddress : u'0e:00:a9:fe:49:54', linklocalnetworkid : u'8ee75f93-69c3-4972-b87d-196a5c2e84db', linklocalnetmask : u'255.255.0.0', publicmacaddress : u'1e:00:51:00:00:03', id : u'4142944f-de32-4628-94c6-b062ff8ea5a7', networkdomain : u'cs5cloud.internal', guestnetworkid : u'fa1d1a22-231c-4dd9-b2fb-9560c6f7bed7', hostname : u'pr4068-t2204-xcpng81-xs1', gatewa y : u'10.1.63.254', publicip : u'10.1.37.223', state : u'Running', version : u'4.15.0', role : u'VIRTUAL_ROUTER', podid : u'bd593c3a-f865-42a8-a4fe-6bcf4e8a8533', serviceofferingid : u'17eb7c2a-3219-4453-ab71-44 f5ac1b8dfa', zonename : u'pr4068-t2204-xcpng81', podname : u'Pod1', name : u'r-3-VM', guestnetworkname : u'NETWORK-10.1.2.1', nic : [{networkid : u'8ee75f93-69c3-4972-b87d-196a5c2e84db', macaddress : u'0e:00:a9: fe:49:54', id : u'87429061-035f-402d-9398-a73b31218b68', traffictype : u'Control', netmask : u'255.255.0.0', ipaddress : u'169.254.73.84', gateway : u'169.254.0.1', isdefault : False}, {networkid : u'55e27b1c-82 05-4b3b-be9c-ae113e2a4342', macaddress : u'1e:00:51:00:00:03', isolationuri : u'vlan://7', broadcasturi : u'vlan://7', traffictype : u'Public', netmask : u'255.255.224.0', id : u'70abd2d1-a7c4-4d99-8489-6c2a3587 bba7', ipaddress : u'10.1.37.223', gateway : u'10.1.63.254', isdefault : True}, {networkid : u'7cbed113-ed53-4e45-845f-9e6d5d533175', macaddress : u'02:00:12:7b:00:02', isolationuri : u'vlan://1986', type : u'Is olated', broadcasturi : u'vlan://1986', traffictype : u'Guest', netmask : u'255.255.255.0', ipaddress : u'10.1.1.34', id : u'add5dbaa-ba0b-4f8c-aad0-9b5f58bef876', networkname : u'NETWORK-10.1.1.1', gateway : u' 10.1.1.1', isdefault : False}, {networkid : u'fa1d1a22-231c-4dd9-b2fb-9560c6f7bed7', macaddress : u'02:00:1e:af:00:02', isolationuri : u'vlan://1992', type : u'Isolated', broadcasturi : u'vlan://1992', trafficty pe : u'Guest', netmask : u'255.255.255.0', ipaddress : u'10.1.2.7', id : u'a8b8603b-81a6-4dd0-b587-365ea9d8c6da', networkname : u'NETWORK-10.1.2.1', gateway : u'10.1.2.1', isdefault : False}], publicnetworkid : u'55e27b1c-8205-4b3b-be9c-ae113e2a4342', redundantstate : u'BACKUP', hostid : u'05b1f989-51ae-4774-a7e5-22ef597f913f', templateid : u'22bea20c-649e-4745-b73b-d44b72f46a16', requiresupgrade : False, publicnetmask : u'255.255.224.0', account : u'test-TestVPCRedundancy-test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL-UM7Q58', vpcid : u'f6a98c6a-7ca4-4f08-a8bd-b44c55ca2ee7', isredundantrouter : True, created : u'2020- 07-27T10:50:57+0000', hypervisor : u'XenServer', dns1 : u'10.2.0.50', vpcname : u'TestVPC-1QKJV0', dns2 : u'8.8.8.8', guestnetmask : u'255.255.255.0', guestipaddress : u'10.1.2.7', serviceofferingname : u'System Offering For Software Router'}] 2020-07-27 11:00:51,904 - CRITICAL - FAILED: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL: ['Traceback (most recent call last):\n', ' File "/usr/lib64/python2.7/unittest/case.py", line 369, in run\n testMethod()\n', ' File "/marvin/tests/smoke/test_vpc_redundant.py", line 547, in test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL\n self.check_routers_state(1)\n', ' File "/marvin/tests/smoke/test_vp c_redundant.py", line 313, in check_routers_state\n self.query_routers(count, showall)\n', ' File "/marvin/tests/smoke/test_vpc_redundant.py", line 300, in query_routers\n "Check that %s routers were inde ed created" % count)\n', ' File "/usr/lib64/python2.7/unittest/case.py", line 553, in assertEqual\n assertion_func(first, second, msg=msg)\n', ' File "/usr/lib64/python2.7/unittest/case.py", line 546, in _b aseAssertEqual\n raise self.failureException(msg)\n', 'AssertionError: Check that 1 routers were indeed created\n'] 2020-07-27 11:00:51,950 - DEBUG - Payload: {'forced': True, 'apiKey': u'LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q', 'response': 'json', 'command': 'stopRouter', 'sign ature': '3GalIiKFGN9BsnNE81LHSCzcuzY=', 'id': u'86667242-1455-4c34-9168-b98b0bcbf3e0'} 2020-07-27 11:00:51,950 - DEBUG - ========Sending GET Cmd : stopRouter=======The text was updated successfully, but these errors were encountered: