batch refactoring / multi-threading

[tylerflex]

Goals:

1. Simplifies how Batch and Job work internally.
2. Improves the visual output from Batch.
3. Implements multi-threading for batch operations (upload, download, etc).

Refactor

Previously, Job objects were defined like this

class Job:
    ...
    task_id = None

    @pd.root_validator()
    def _upload(cls, values) -> None:
          # upload task
          # set task id internally

This meant that when you created a Job, the task is immediately uploaded.
Also when you save a Job.to_file() the task info is contained.

This design made it a bit annoying to do normal multi-threading over job upload.

In the refactored version, I set it up to have upload in its own method.

class Job:
    ...
    task_id_cached = None

    @cached_property
    def task_id(self):
        # return self.task_id_cached or call `self.upload()` 

    def upload(self):
         ...

    def json(self):
          # extend json to add `cached_properties["task_id"] to json file for backwards compatibility when loading from file

Now a similar refactoring was done for Batch to move the upload() bits to their own method.

Before, the Batch created all Job objects and uploaded them at initialization time in a validator.

class Batch:
    ...
    jobs : Dict[str, Job] = None

    @pd.root_validator
    def _upload():
        # for each simulation in the batch, make a job (which automatically uploads it).

instead, now that we have the Job with its own upload method, I simplified this to.

class Batch:
    ...
    jobs : Dict[str, Job] = None

    @cached_property
    def jobs():
        # construct all of the Jobs for this Batch

    def upload(self):
        # upload each Job in a for loop

Multithreading

Implemented basic multi-threading for Batch.upload() and Batch.download(). Works as expected on the Design plugin notebook.

Also made it so that Batch.run() will download by default.

Output / Logging

• The batch upload, download, and BatchData loading all just log a single output instead of one line for every file.

[daquinteroflex]

Nice! Looks great, excited to test this!

[tylerflex]

Thanks @lei-flex @daquinteroflex @lucas-flexcompute for the comments.
I think this is ready to go. Did a bit of testing with WebAPI.ipynb, Design.ipynb and the adjoint batching tutorial. All looks good. Seems faster than before and also the output is much more reasonable.

Let me know if you think this needs more review or work on any particular aspects, thanks!

[lei-flex]

ship it.