Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uploading large objects using multipart #391

Closed
sk4mi opened this issue Oct 19, 2023 · 2 comments
Closed

Uploading large objects using multipart #391

sk4mi opened this issue Oct 19, 2023 · 2 comments
Assignees
@oxyno-zeta
Labels
enhancement New feature or request no-issue-activity

Comments

@sk4mi
Copy link

sk4mi commented Oct 19, 2023 •
edited by oxyno-zeta

Is your feature request related to a problem? Please describe.
I'm trying to upload large files through s3-proxy, but I get EntityTooLarge: Your proposed upload exceeds the maximum allowed size error.

Describe the solution you'd like
After some research, I found this aws s3 doc which describes how to perform multipart upload.
Maybe it works without change, but i don't manage to upload file using curl:

curl -XPUT -H "Content-Type: multipart/form-data" --form file='@gentoo_root.img'  -L https
://s3-proxy.local/ -v
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://REDACTED
* [HTTP/2] [1] [:method: PUT]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
* [HTTP/2] [1] [content-length: 6442451163]
* [HTTP/2] [1] [content-type: multipart/form-data; boundary=------------------------FL8SuDEZRyPHrR90fgYD9T]
> PUT / HTTP/2
> Host: s3-proxy.local
> User-Agent: curl/8.4.0
> Accept: */*
> Content-Length: 6442451163
> Content-Type: multipart/form-data; boundary=------------------------FL8SuDEZRyPHrR90fgYD9T
< HTTP/2 500
< date: Thu, 19 Oct 2023 13:39:06 GMT
< content-type: text/html; charset=utf-8
< content-length: 300
< cache-control: no-cache, no-store, no-transform, must-revalidate, private, max-age=0
< expires: Thu, 01 Jan 1970 00:00:00 UTC
< pragma: no-cache
< strict-transport-security: max-age=15724800; includeSubDomains

<!DOCTYPE html>
<html>
  <body>
    <h1>Internal Server Error</h1>
    <p>EntityTooLarge: Your proposed upload exceeds the maximum allowed size
	status code: 400, **********
  </body>

I don't know if it's my utilization who is wrong, or if there some missing feature.

Thanks in advance
@sk4mi sk4mi added the enhancement New feature or request label Oct 19, 2023
@sk4mi sk4mi changed the title Uploading large objects using multipart upload Uploading large objects using multipart Oct 19, 2023
@oxyno-zeta
Copy link
Owner

Hello @sk4mi ,

Thanks for your issue.

With you, I learned that PutObject is limited to 5Gb file size (cf https://www.learnaws.org/2022/08/22/boto3-s3-upload_file-vs-put-object/#:~:text=No%20support%20for%20multipart%20uploads,files%20larger%20than%205%20GB. )
The current implementation must be changed to a s3manager which handles the multipart upload...
I will try to work on this soon.

FYI: I've redacted your message because you leaked your instance URL

Regards,

Oxyno-zeta

@oxyno-zeta oxyno-zeta self-assigned this Oct 21, 2023
oxyno-zeta added a commit that referenced this issue Oct 23, 2023
@oxyno-zeta
feat: Change upload to S3Manager instead of classic PUT to increase m…
25bb790 
…aximum file size upload

Related to #391
@oxyno-zeta oxyno-zeta mentioned this issue Oct 25, 2023
Merged
4 tasks
oxyno-zeta added a commit that referenced this issue Oct 27, 2023
@oxyno-zeta
feat: Change upload to S3Manager instead of classic PUT to increase m…
133dfce 
…aximum file size upload

Related to #391
@github-actions GitHub Actions
Copy link

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oxyno-zeta oxyno-zeta

Labels
enhancement New feature or request no-issue-activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@oxyno-zeta @sk4mi