LoggingThread should recover from temporary outage on hub

[rohanpm]

When a task is running in a kobo worker, it's the responsibility of the LoggingThread to send log messages back to the hub.

This is done by the following loop:

    def run(self):
        """Send queue content to hub."""
        while self._running or not self._queue.empty() or self._send_data:
            if self._queue.empty():
                self._event.wait(5)

            self._event.clear()
            while True:
                try:
                    self._send_data += self._queue.get_nowait()
                except six.moves.queue.Empty:
                    break

            if not self._send_data:
                continue

            now = int(time.time())
            if self._running and len(self._send_data) < 1200 and now - self._send_time < 5:
                continue

            if isinstance(self._send_data, six.text_type):
                self._send_data = self._send_data.encode('utf-8')

            try:
                self._hub.upload_task_log(StringIO(self._send_data), self._task_id, "stdout.log", append=True)
                self._send_time = now
                self._send_data = ""
            except Fault:
                continue

Problem: if self._hub.upload_task_log raises an error (other than an XML-RPC Fault), then the logging thread simply stops. However, the main thread of the task being executed doesn't necessarily stop.

The end result of this is that if the kobo hub has a temporary outage while a task is in progress, that task might continue executing but all logs after the outage would be lost.

It would be better if the LoggingThread kept retrying the task log uploads, for as long as the task's main thread is alive.

Steps to reproduce

• Start a long-running task
• Take the kobo hub down temporarily during task execution

Actual behavior

• Task may continue running but with no further log messages uploaded

Expected behavior

• If task continues running, then it resumes uploading logs after kobo hub is restored.