Make it safe for workload executors to output large amounts of text

[prashantmital]

Currently, when running the workload executor, we PIPE the output from its stderr and stdout streams. To avoid blocking the child process when the pipe fills up, we need to continuously read the worker_subprocess output (via worker_subprocess.stdout.read(..)).

Thanks to @ShaneHarvey for the context:

See the note in subprocess.wait:

This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that.

https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait

Just to be clear: there's no risk of deadlock in the current approach. The risk is that the child process can fill up the pipe with output and then it will block until we call worker_subprocess.communicate() down below.

A more elegant solution will be to redirect the worker_subprocess output to a file. Then we can open and read the file after the process exits.

[prashantmital]

Since we no longer require workload executors to send all output to STDOUT, we can simply not pipe the standard streams of the workload executor subprocess back to the parent astrolabe process. This means we will get a unified log on evergreen containing astrolabe logs, workload executor stdout, and workload executor stderr. That should suffice for the time being and also resolve this issue of the pipe getting saturated and blocking the execution of workload executor.