Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

larger than 2 GB single-part uploads #1

Closed
gaul opened this issue Jul 27, 2014 · 3 comments
Closed

larger than 2 GB single-part uploads #1

gaul opened this issue Jul 27, 2014 · 3 comments

Comments

@gaul
Copy link
Owner

gaul commented Jul 27, 2014

jclouds dependency on Java 6 limits S3Proxy to 2 GB single-part uploads. When jclouds merges jclouds/jclouds#426 S3Proxy can support larger uploads, although many providers have a 5 GB single-part limit.

@gaul
Copy link
Owner Author

gaul commented Aug 27, 2014

Waiting for jclouds 2.0 release.

gaul added a commit that referenced this issue Jan 12, 2015
This release allows larger than 2 GB single-part blobs and supports
object metadata with the filesystem provider on Linux and Windows.  It
also includes support for object size in container lists.  Fixes #1.
Fixes #6.
gaul added a commit that referenced this issue Jan 14, 2015
This release allows larger than 2 GB single-part blobs and supports
object metadata with the filesystem provider on Linux and Windows.  It
also includes support for object size in container lists.  Fixes #1.
Fixes #6.
gaul added a commit that referenced this issue Jan 16, 2015
This release allows larger than 2 GB single-part blobs and supports
object metadata with the filesystem provider on Linux and Windows.  It
also includes support for object size in container lists.  Fixes #1.
Fixes #6.
@gaul gaul closed this as completed in 23f8e77 Jan 22, 2015
@pstrzelczak
Copy link

I tried to upload > 2GB object into s3proxy 1.4.0 launched within a docker container obtained from https://hub.docker.com/r/stevemayne/s3proxy/ but the upload failed with: "Reason: content length must be non-negative, was: -2093729376"

I checked s3proxy code and it seems that https://issues.apache.org/jira/browse/JCLOUDS-264 is not the only problem.

Another problem is that in S3ProxyHandler.handlePutBlob() Content-Length provided in a request was parsed but not used to build PayloadBlobBuilder until cea4a42.
Instead HttpServletRequest.getContentLength() was used that returns and int and leads to overflow.
I checked that there is HttpServletRequest.getContentLengthLong() that can possibly be used instead of parsing Content-Length header by hand.

Anyways, 1.4.0 is unusable for for >2GB single-part uploads.

@gaul
Copy link
Owner Author

gaul commented May 17, 2016

@pstrzelczak Unfortunately S3Proxy 1.4.0 still uses jclouds 1.9.x which does not include a fix for this issue. I am still waiting on jclouds 2.0.0 release before release S3Proxy 1.5.0. Can you try the Dockerfile from #123 instead which uses the master branch? cea4a42 is not relevant to your issue which only applies to v4 signatures which 1.4.0 does not support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants