Implement parallelization to speed up S3 artifacts upload and download

[nitays]

Summary

Implement a client that supports parallelism such as s5cmd to enable much faster S3 artifacts I/O speeds, which is crucial when working with many and\or large files.

Use Cases

When working with many and\or large artifacts via S3 artifact repository on a Workflow, the part of uploading and downloading artifacts in the beginning of every step can take more than the processing time of the step itself.

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[tooptoop4]

one caution is that a lot of these libs don't handle files larger than 5gb ie peak/s5cmd#29

[pranavarnav2]

one caution is that a lot of these libs don't handle files larger than 5gb ie peak/s5cmd#29

That is only when copying files from bucket to bucket, which I don't think is a use case with Argo Workflows artifacts. I've just copied a 10GB file from my local computer to an S3 bucket with s5cmd with no problem.