New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using GCS multipart upload is not supported. #217
Comments
Encountering this issue as well, specifically when uploading to a GCS bucket. The following fails with the misleading InvalidArgument error:
while the following succeeds:
|
Thanks for the report.
If you try to upload a file to an object store, s5cmd will split the file and upload the parts in parallel to achieve maximum throughput, using the S3 Multipart Upload API contract. The problem is, GCS doesn't have multipart upload support in their S3-compatible API. The misleading error you're encounting is the result of the lack of this support. An excerpt from the official GCS docs: So, multipart upload is not supported out of the box. If you use |
Native s5cmd support for GCP could be massively beneficial for that cloud as GCP's native gsutil tooling offers very poor download performance compared to s5cmd. gsutil upload performance is not great either, however this can't be improved by using s5cmd until multi-part upload support is added. See this benchmarking deep-dive I wrote on the subject: |
Is this still an existing issue? |
It seems that GCS now supports both ListObjectsV2 and S3 multipart upload protocols according to their changelog. I can't test it now but if someone could test and report, that'd be very helpful. I'm closing the issue. Please feel free to re-open if you see any problem with GCS multipart uploads. |
Uploading files larger then the partition size will cause s5cmd to use multipart uploads, which is not supported.
Current work around is to set:
for copying to prevent sharding.
Error is also misleading:
The text was updated successfully, but these errors were encountered: