Issue downloading result pickle file after execution

[AlejandroEsquivel]

After dispatching a basic (iris dataset) classical SVM example I seem only to get a successful dispatch sometimes while the remaining attempts fail due to the ecs plugin not being able to download the result object.

[2022-08-18 16:34:53,772] [ERROR] execution.py: Line 371 in _run_task: Exception occurred when running task 0: An error occurred (400) when calling the HeadObject operation: Bad Request
[2022-08-18 16:34:53,773] [ERROR] execution.py: Line 379 in _run_task: Run task exception
Traceback (most recent call last):
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/covalent_dispatcher/_core/execution.py", line 355, in _run_task
    output, stdout, stderr = await loop.run_in_executor(tasks_pool, execute_callable)
  File "/Users/alejandro/.pyenv/versions/3.8.13/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/covalent_ecs_plugin/ecs.py", line 265, in execute
    return self._query_result(result_filename, task_results_dir, task_arn, image_tag)
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/covalent_ecs_plugin/ecs.py", line 497, in _query_result
    s3.download_file(self.s3_bucket_name, result_filename, local_result_filename)
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/boto3/s3/inject.py", line 190, in download_file
    return transfer.download_file(
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/boto3/s3/transfer.py", line 320, in download_file
    future.result()
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/s3transfer/download.py", line 354, in _submit
    response = client.head_object(
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/alejandro/.pyenv/versions/qa/lib/python3.8/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

However when manually downloading the produced result.pkl file in the S3 bucket, it seems that the result was successfully written and the contents contained the expected output of the task.

So seems to be limited to when the executor is attempting to download the result file.