s3 EntityTooLarge

[aws623]

95356 [Thread-2] INFO  org.freeeed.ec2.S3Agent  - Putting file [xxx.zip]
120528 [Thread-2] ERROR org.freeeed.ec2.S3Agent  - Error putting file into bucket
org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode: 400, ResponseStatus: Bad Request, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>7771682432</ProposedSize><MaxSizeAllowed>5368709120</MaxSizeAllowed><RequestId>xxx</RequestId><HostId>xxx</HostId></Error>
        at org.jets3t.service.S3Service.putObject(S3Service.java:2123)
        at org.freeeed.ec2.S3Agent.putFileInS3(S3Agent.java:274)
        at org.freeeed.ec2.ProcessAgent.runUpload(ProcessAgent.java:203)
        at org.freeeed.ec2.ProcessAgent.runCompleteProcess(ProcessAgent.java:139)
        at org.freeeed.ec2.ProcessAgent.run(ProcessAgent.java:74)
        at java.lang.Thread.run(Unknown Source)

[markkerzner]

Hi, aws623, can you tell me more?

Mark Kerzner, Managing Partner, Elephant Scale http://elephantscale.com/
Book a call with me here http://www.meetme.so/markkerzner
Mobile: 713-724-2534, Skype: mark.kerzner1

On Fri, Sep 30, 2016 at 11:50 AM, aws623 notifications@github.com wrote:

95356 [Thread-2] INFO org.freeeed.ec2.S3Agent - Putting file [xxx.zip]
120528 [Thread-2] ERROR org.freeeed.ec2.S3Agent - Error putting file into
bucket
org.jets3t.service.S3ServiceException: S3 Error Message. -- ResponseCode:
400, ResponseStatus: Bad Request, XML Error Message: EntityTooLargeYour proposed upload exceeds the maximum
allowed size77716824325368709120xxxxxx
at org.jets3t.service.S3Service.putObject(S3Service.java:2123)
at org.freeeed.ec2.S3Agent.putFileInS3(S3Agent.java:274)
at org.freeeed.ec2.ProcessAgent.runUpload(ProcessAgent.java:203)
at org.freeeed.ec2.ProcessAgent.runCompleteProcess(ProcessAgent.java:139)
at org.freeeed.ec2.ProcessAgent.run(ProcessAgent.java:74)
at java.lang.Thread.run(Unknown Source)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#158, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAk2ebOQBvTDWPc5oTgT54KiezyddA7Vks5qvT3dgaJpZM4KLSpN
.

[aws623]

Sure. Basically, it appears that there is a 5gb cap on objects that can be uploaded to s3 via the API free-eed currently uses. I am running the latest version on Windows 10. I tried to play with the settings to see if the zips could be broken up, but that does not help. It took me awhile to figure out some other issues which were related to me using the wrong security group on the cluster --but I have manually uploaded the larger zips (about 7 gigs) to the correct bucket and hoping this works now. But in other words, it looks like using the "PutObject" method is limited to 5gb https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html

[aws623]

Mark, the next problem I am running into is "scp: FreeEed/3.project.s3: No
such file or directory"

On Fri, Sep 30, 2016 at 12:57 PM, Alexander Simon asimon623@gmail.com
wrote:

Sure. Basically, it appears that there is a 5gb cap on objects that can
be uploaded to s3 via the API free-eed currently uses. I am running the
latest version on Windows 10. I tried to play with the settings to see if
the zips could be broken up, but that does not help.

It took me awhile to figure out some other issues which were related to me
using the wrong security group on the cluster --but I have manually
uploaded the larger zips (about 7 gigs) to the correct bucket and hoping
this works now.

But in other words, it looks like using the "PutObject" method is limited
to 5gb

https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/
s3-multipart-upload.html

[markkerzner]

So decided, waiting till the new cloud version, right? I am working on the
next maintenance release right now, then the cloud

Mark Kerzner, Managing Partner, Elephant Scale http://elephantscale.com/
Book a call with me here http://www.meetme.so/markkerzner
Mobile: 713-724-2534, Skype: mark.kerzner1

On Fri, Sep 30, 2016 at 12:25 PM, aws623 notifications@github.com wrote:

Mark, the next problem I am running into is "scp: FreeEed/3.project.s3: No
such file or directory"

On Fri, Sep 30, 2016 at 12:57 PM, Alexander Simon asimon623@gmail.com
wrote:

Sure. Basically, it appears that there is a 5gb cap on objects that can
be uploaded to s3 via the API free-eed currently uses. I am running the
latest version on Windows 10. I tried to play with the settings to see if
the zips could be broken up, but that does not help.

It took me awhile to figure out some other issues which were related to
me
using the wrong security group on the cluster --but I have manually
uploaded the larger zips (about 7 gigs) to the correct bucket and hoping
this works now.

But in other words, it looks like using the "PutObject" method is limited
to 5gb

https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/
s3-multipart-upload.html

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#158 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAk2ebGpPW6HS1LHewIgNC40erXAgiY2ks5qvUX_gaJpZM4KLSpN
.

[aws623]

Yea waiting. Thank you!
On Sun, Oct 2, 2016 at 1:14 PM Mark Kerzner notifications@github.com
wrote:

So decided, waiting till the new cloud version, right? I am working on the
next maintenance release right now, then the cloud

Mark Kerzner, Managing Partner, Elephant Scale http://elephantscale.com/
Book a call with me here http://www.meetme.so/markkerzner
Mobile: 713-724-2534, Skype: mark.kerzner1

On Fri, Sep 30, 2016 at 12:25 PM, aws623 notifications@github.com wrote:

Mark, the next problem I am running into is "scp: FreeEed/3.project.s3:
No
such file or directory"

On Fri, Sep 30, 2016 at 12:57 PM, Alexander Simon asimon623@gmail.com
wrote:

Sure. Basically, it appears that there is a 5gb cap on objects that can
be uploaded to s3 via the API free-eed currently uses. I am running the
latest version on Windows 10. I tried to play with the settings to see
if
the zips could be broken up, but that does not help.

It took me awhile to figure out some other issues which were related to
me
using the wrong security group on the cluster --but I have manually
uploaded the larger zips (about 7 gigs) to the correct bucket and
hoping
this works now.

But in other words, it looks like using the "PutObject" method is
limited
to 5gb

https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/
s3-multipart-upload.html

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<
https://github.com/markkerzner/FreeEed/issues/158#issuecomment-250803238>,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAk2ebGpPW6HS1LHewIgNC40erXAgiY2ks5qvUX_gaJpZM4KLSpN

.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#158 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF5SV3ICgJsi7f1s3SLp8f9zrEmIGTvDks5qv-ZZgaJpZM4KLSpN
.