Autotests hanging on 'Address already in use'

[pploegaert]

Quite frequently observed on several virtual test envs:
Occurs during creation of test vm:

• ci_scenario_hypervisor_ha_test
• ci_scenario_mds_regression

Scenario_helpers.log shows:

2017-08-05 00:17:32 91800 +0200 - ovs-node01-1604 - 14336/140137224808192 - scenario_helpers/vm_handler - 2566 - INFO - Prepped everything for VM HA-test-000.
2017-08-05 00:17:33 04100 +0200 - ovs-node01-1604 - 14336/140137224808192 - scenario_helpers/vm_handler - 2568 - INFO - Creating VM `HA-test-000`
2017-08-05 00:17:34 88700 +0200 - ovs-node01-1604 - 14336/140135991604992 - scenario_helpers/vm_handler - 2571 - ERROR - Error while listening for VM messages.. Got [Errno 98] Address already in use: None

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/tmp/tmp.RndTRfekd7/rpyc/core/protocol.py", line 305, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/tmp/tmp.RndTRfekd7/rpyc/core/protocol.py", line 535, in _handle_call
    return self._local_objects[oid](*args, **dict(kwargs))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 98] Address already in use

2017-08-05 00:17:36 91000 +0200 - ovs-node01-1604 - 14336/140137224808192 - scenario_helpers/vm_handler - 2576 - INFO - Created VM `HA-test-000`!

[JeffreyDevloo]

The hanging issue will be due to a thread not being killed properly which is expecting the message of the vms.
The address in use can be resolved by binding the port at the moment when the free port was fetched, thus leaving that port not available for other processes.

[sjovdnbos]

Fixed in #609
Package information: openvstorage-test_3.4.1-dev.1516637295.4b6bfc8-1_amd64.deb

[JeffreyDevloo]

Released in https://github.com/openvstorage/integrationtests/releases/tag/3.4.1
Packaged in openvstorage-test_3.4.1-1_amd64.deb