Log collection after test failure can cause the whole suite to collapse

[JeffreyDevloo]

Problem description

Collecting logs is not completely safe. When the connection is interrupted the whole log collecting will raise causing a complete restart.
We should capture these errors, log them and then proceed testing

Error found:

"2018-02-07 03:11:15.733260", "failed": true, "rc": 1, "start": "2018-02-06 23:40:10.012215", "stderr": "libvirt: QEMU Driver error : Domain not found: no domain with matching name 'HA-test-000'
libvirt: QEMU Driver error : Domain not found: no domain with matching name 'mds-regression-000'
Traceback (most recent call last):
  File \"<string>\", line 1, in <module>
  File \"/opt/OpenvStorage/ci/main.py\", line 328, in main
    w.run()
  File \"/opt/OpenvStorage/ci/main.py\", line 82, in run
    Workflow._run_with_retry(self.scenario, self.config['ci']['scenario_retries'])
  File \"/opt/OpenvStorage/ci/main.py\", line 69, in _run_with_retry
    target(*args, **kwargs)
  File \"/opt/OpenvStorage/ci/main.py\", line 270, in scenario
    exclude_scenarios=exclude_scenarios)
  File \"/opt/OpenvStorage/ci/autotests.py\", line 73, in run
    module_result = mod.run(blocked)
  File \"/opt/OpenvStorage/ci/scenarios/arakoon/ovs_3671_validate_archiving_of_existing_arakoon_data_on_create_and_extend_test/main.py\", line 155, in run
    return ArakoonArchiving().main(blocked)
  File \"/opt/OpenvStorage/ci/autotests.py\", line 459, in wrapped
    result = [str(ex), '', 'Logs collected between {0} and {1}'.format(start, end), '', LogCollector.get_logs(components=log_components, since=start, until=end)]
  File \"/opt/OpenvStorage/ci/autotests.py\", line 407, in get_logs
    return LogFileTimeParser.execute_search_on_remote(since=since, until=until, search_locations=units)
  File \"/usr/lib/python2.7/dist-packages/ovs_extensions/log/log_reader.py\", line 139, in execute_search_on_remote
    with remote(hosts, [LogFileTimeParser], username=username, password=password) as remotes:
  File \"/usr/lib/python2.7/dist-packages/ovs_extensions/generic/remote.py\", line 66, in __init__
    self.machines = [SshMachine(ip, user=self.username, password=password, ssh_opts=tuple(ssh_opts)) for ip in self.ips]
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/ssh_machine.py\", line 106, in __init__
    new_session = new_session)
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/remote.py\", line 129, in __init__
    self._session = self.session(new_session = new_session)
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/ssh_machine.py\", line 150, in session
    self.encoding, isatty, self.connect_timeout)
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/session.py\", line 141, in __init__
    self.run(\"\")
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/session.py\", line 224, in run
    return run_proc(self.popen(cmd), retcode)
  File \"/usr/lib/python2.7/dist-packages/plumbum/commands/processes.py\", line 186, in run_proc
    stdout, stderr = proc.communicate()
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/session.py\", line 92, in communicate
    line = pipe.readline()
  File \"/usr/lib/python2.7/dist-packages/plumbum/machines/session.py\", line 42, in readline
    raise EOFError()
EOFError"

[JeffreyDevloo]

Will be fixed by #613

[JeffreyDevloo]

Released in https://github.com/openvstorage/integrationtests/releases/tag/3.4.1
Packaged in openvstorage-test_3.4.1-1_amd64.deb