Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GCP's S3 API for Google Cloud Buckets requires content-length header for create-multi-part-upload #491

Open
allada opened this issue Dec 15, 2023 · 0 comments
Open

GCP's S3 API for Google Cloud Buckets requires content-length header for create-multi-part-upload #491

allada opened this issue Dec 15, 2023 · 0 comments
Assignees
@adam-singer
Labels
bug Something isn't working

Comments

@allada
Copy link
Member

allada commented Dec 15, 2023

In the event a user is using Google Cloud Buckets and S3 store, uploads >5MB are required to be uploaded using mutlti-part; however, it is not possible because the aws-sdk-rust library does not publish the content-length header for that payload resulting in the request failing and being unable to do that upload in GCP.

The only current work around is to use dedup store and make sure it splits the objects into smaller chunks.
@allada allada mentioned this issue Dec 15, 2023
Closed
@allada allada added the bug Something isn't working label Dec 15, 2023
allada added a commit that referenced this issue Dec 19, 2023
@allada
[Breaking] Mark S3 store experimental
05a6dd7 
S3 store is too unstable to consider it production worthy.

In the mean time we will mark it experimental until we get the bugs
fixed.

related: #491
related: #460
related: #428
@adam-singer adam-singer self-assigned this Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adam-singer adam-singer

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adam-singer @allada