Download Attempts-Data Returned

[timronan]

Verifies that bytes are returned in each download attempt. Creates logical frame work to handle different types of download attempts. Changes output messages for each download attempt.

[chad-earthscope]

I do not believe this code is doing what is expected:

        p = Popen([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
        output, errors = p.communicate()
        # Print statement is used to add bytes to the output pipe. Semi-kluge but works until we find a better solution.
        # Potentially associated with the asynchronous behavior of the workers downloads.
        print("\033[F")
        # Determines if bytes are returned by the subprocess created by workers.execute.
        if len(output) is not 0:
            ErrorStatistics.bytecount += 1 #Increments the byte count in manager.ErrorStatistics
        return p

The code effective says "if the subprocess produces any data on it's stdout stream, increment a number". This is not the same thing as knowing that a download worker actually received data.

Also, I believe this general _popen call is used for all workers (download, ingestion, indexing) and it would be doing the same logic and accumulation for non-download workers.

[chad-earthscope]

This is not the actual byte count, it's an incremented value, so confusingly named.

[chad-earthscope]

This is a console sequence for "Cursor up one line". It should not be needed at all.

[chad-earthscope]

This is testing if the stdout stream of the called process returned anything, combined with that "Cursor up one line", who knows what it's getting. It's very likely unrelated to anything about bytes downloaded. I printed the output, it is some non-sensical repeat of my debugging print statement.

[chad-earthscope]

Needs to be confirmed if we go with this approach, but this may be high-jacking all stdout and stderr output, preventing those from going into the rover log files.

[chad-earthscope]

Even more problematic: the p.communicate() call blocks/waits for the process to terminate. This effectively turns rover into single-process program, with no parallelism.

In the rover model this _popen call just spins off the worker processes and they work in the background. The check for termination is in Workers.check() method, using process.poll() to determine when worker processes have terminated.