Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload never gets above 40% #1111

Closed
bknill opened this issue Aug 25, 2016 · 5 comments
Closed

Upload never gets above 40% #1111

bknill opened this issue Aug 25, 2016 · 5 comments
Labels
guidance Question that needs advice or information.

Comments

@bknill
Copy link

bknill commented Aug 25, 2016 •
edited
Loading

Hi,

I've got a strange issue where I'm uploading images to S3 and the progress keeps dropping until it timeouts. This is in a Cordova based app, I'm downloading images from a camera and then attempting to upload to s3.

The upload code
` fileEntry.file(function(file) {

var reader = new FileReader();

reader.onloadend = function (evt) {

  var theBody = this.result;
  var bucket = new AWS.S3({params: {Bucket: 'bucket-name'}});
  // var opts = {queueSize: 2, partSize: 1024 * 1024 * 10};
  var params = { Key: file.name,
    ContentEncoding: 'base64',
    ContentType: file.type,
    Body: theBody};

  bucket.upload(params, function (err, data) {
    console.log(err);
    console.log(data);
    if(data)callback(data);
  }).on('httpUploadProgress', function(evt) {
    console.log('Progress:',evt,formatBytes(evt.loaded),formatBytes(evt.total), parseInt(evt.loaded/evt.total * 100) + "%");
  });
};

reader.readAsDataURL(file);

})`

I've tried this with a small text file and it works well, however on larger (4.5mb) image files it never completes, a trace of the progress.

It's only a 4.5mb file it shouldn't be such an issue. I'm using the nearest location for my bucket.

Any clues!?

Progress: Object {loaded: 688128, total: 6171107, part: 1} 688.128 KB 6.171 MB 11%
awsController.js:47 Progress: Object {loaded: 1376256, total: 6171107, part: 2} 1.376 MB 6.171 MB 22%
awsController.js:47 Progress: Object {loaded: 1605632, total: 6171107, part: 1} 1.606 MB 6.171 MB 26%
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 2} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 1845731, total: 6171107, part: 2} 1.846 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 2075107, total: 6171107, part: 1} 2.075 MB 6.171 MB 33%
aws-sdk.min.js:15 PUT https://bucket-name.s3-ap-southeast-2.amazonaws.com/1_1.jpg?partNumber…4hGgSX0zNgzmuwJC.ihQAwkhdJvamVvOM5Y1AnG6djwnjBzbFG2_y_Xy5lSENsSh0tGgUjeQ-- 400 (Bad Request)a.XHRClient.a.util.inherit.handleRequest @ aws-sdk.min.js:15i @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14t @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.Credentials.a.util.inherit.get @ aws-sdk.min.js:14o @ aws-sdk.min.js:14a.Config.a.util.inherit.getCredentials @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14t @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.Credentials.a.util.inherit.get @ aws-sdk.min.js:14o @ aws-sdk.min.js:14a.Config.a.util.inherit.getCredentials @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17a.Request.n.runTo @ aws-sdk.min.js:15a.Request.n.send @ aws-sdk.min.js:15a.S3.ManagedUpload.a.util.inherit.uploadPart @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14a.emit @ aws-sdk.min.js:18a.XHRClient.a.util.inherit.finishRequest @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 2} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 1} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 2064384, total: 6171107, part: 2} 2.064 MB 6.171 MB 33%
awsController.js:47 Progress: Object {loaded: 1605632, total: 6171107, part: 1} 1.606 MB 6.171 MB 26%
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 1} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 1845731, total: 6171107, part: 2} 1.846 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 2075107, total: 6171107, part: 1} 2.075 MB 6.171 MB 33%
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 2} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 1} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 1376256, total: 6171107, part: 1} 1.376 MB 6.171 MB 22%
aws-sdk.min.js:15 PUT https://bucket-name.s3-ap-southeast-2.amazonaws.com/1_1.jpg?partNumber…4hGgSX0zNgzmuwJC.ihQAwkhdJvamVvOM5Y1AnG6djwnjBzbFG2_y_Xy5lSENsSh0tGgUjeQ-- net::ERR_CONNECTION_RESETa.XHRClient.a.util.inherit.handleRequest @ aws-sdk.min.js:15i @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14t @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.Credentials.a.util.inherit.get @ aws-sdk.min.js:14o @ aws-sdk.min.js:14a.Config.a.util.inherit.getCredentials @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16
awsController.js:47 Progress: Object {loaded: 1376256, total: 6171107, part: 2} 1.376 MB 6.171 MB 22%
awsController.js:47 Progress: Object {loaded: 1376256, total: 6171107, part: 2} 1.376 MB 6.171 MB 22%
aws-sdk.min.js:15 PUT https://bucket-name.s3-ap-southeast-2.amazonaws.com/1_1.jpg?partNumber…4hGgSX0zNgzmuwJC.ihQAwkhdJvamVvOM5Y1AnG6djwnjBzbFG2_y_Xy5lSENsSh0tGgUjeQ-- net::ERR_CONNECTION_RESETa.XHRClient.a.util.inherit.handleRequest @ aws-sdk.min.js:15i @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14t @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.Credentials.a.util.inherit.get @ aws-sdk.min.js:14o @ aws-sdk.min.js:14a.Config.a.util.inherit.getCredentials @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16
awsController.js:47 Progress: Object {loaded: 1376256, total: 6171107, part: 1} 1.376 MB 6.171 MB 22%
awsController.js:47 Progress: Object {loaded: 1376256, total: 6171107, part: 1} 1.376 MB 6.171 MB 22%
awsController.js:47 Progress: Object {loaded: 1605632, total: 6171107, part: 2} 1.606 MB 6.171 MB 26%
awsController.js:47 Progress: Object {loaded: 1835008, total: 6171107, part: 1} 1.835 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 1845731, total: 6171107, part: 2} 1.846 MB 6.171 MB 29%
awsController.js:47 Progress: Object {loaded: 2075107, total: 6171107, part: 1} 2.075 MB 6.171 MB 33%
aws-sdk.min.js:15 PUT https://bucket-name.s3-ap-southeast-2.amazonaws.com/1_1.jpg?partNumber…4hGgSX0zNgzmuwJC.ihQAwkhdJvamVvOM5Y1AnG6djwnjBzbFG2_y_Xy5lSENsSh0tGgUjeQ-- 400 (Bad Request)a.XHRClient.a.util.inherit.handleRequest @ aws-sdk.min.js:15i @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16(anonymous function) @ aws-sdk.min.js:14t @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.Credentials.a.util.inherit.get @ aws-sdk.min.js:14o @ aws-sdk.min.js:14a.Config.a.util.inherit.getCredentials @ aws-sdk.min.js:14(anonymous function) @ aws-sdk.min.js:14a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16a.SequentialExecutor.a.util.inherit.emit @ aws-sdk.min.js:16a.Request.n.emitEvent @ aws-sdk.min.js:15e @ aws-sdk.min.js:15a.runTo @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:17(anonymous function) @ aws-sdk.min.js:15(anonymous function) @ aws-sdk.min.js:15a.SequentialExecutor.a.util.inherit.callListeners @ aws-sdk.min.js:16n @ aws-sdk.min.js:16
awsController.js:47 Progress: Object {loaded: 2075107, total: 6171107, part: 1} 2.075 MB 6.171 MB 33%
awsController.js:43 Error: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.(…)
@chrisradek
Copy link
Contributor

@bknill
Is your connection slow, or taking over 2 minutes to upload a part?
Can you set the timeout for S3 to 0 like in this thread:
#949

It sounds like your connection is timing out before it has a chance to finish the upload, and the test in the linked issue should verify that.

@bknill
Copy link
Author

bknill commented Aug 25, 2016

Thanks I'm trying that, I do have a slow connection however I can upload these files via the browser or cloudberry relatively quickly!

It's currently still dropping back to 11%
awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 917504, totalSize: 6171107, lengthComputable: true…} 917.504 KB 6.171 MB 14% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1163264, totalSize: 6171107, lengthComputable: true…} 1.163 MB 6.171 MB 18% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1376256, totalSize: 6171107, lengthComputable: true…} 1.376 MB 6.171 MB 22% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1654784, totalSize: 6171107, lengthComputable: true…} 1.655 MB 6.171 MB 26% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1884160, totalSize: 6171107, lengthComputable: true…} 1.884 MB 6.171 MB 30% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 2113536, totalSize: 6171107, lengthComputable: true…} 2.114 MB 6.171 MB 34% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 2342912, totalSize: 6171107, lengthComputable: true…} 2.343 MB 6.171 MB 37% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 2342912, totalSize: 6171107, lengthComputable: true…} 2.343 MB 6.171 MB 37% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 688128, totalSize: 6171107, lengthComputable: true…} 688.128 KB 6.171 MB 11% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 917504, totalSize: 6171107, lengthComputable: true…} 917.504 KB 6.171 MB 14% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1146880, totalSize: 6171107, lengthComputable: true…} 1.147 MB 6.171 MB 18% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1359872, totalSize: 6171107, lengthComputable: true…} 1.36 MB 6.171 MB 22% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1589248, totalSize: 6171107, lengthComputable: true…} 1.589 MB 6.171 MB 25% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 1589248, totalSize: 6171107, lengthComputable: true…} 1.589 MB 6.171 MB 25% awsController.js:50 Progress: XMLHttpRequestProgressEvent {isTrusted: true, part: 1, position: 688128, totalSize: 6171107, lengthComputable: true…} 688.128 KB 6.171 MB 11%

@bknill
Copy link
Author

bknill commented Aug 25, 2016

Setting the Queue size to 1 fixed it
queueSize: 1,

@bknill bknill closed this as completed Aug 25, 2016
@usamamashkoor
Copy link

@bknill how setting it to 1 fixed it i have the same problem but still facing the same issue.
Thanks

@srchase srchase added the guidance Question that needs advice or information. label Dec 24, 2018
@lock
Copy link

lock bot commented Sep 29, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.

@lock lock bot locked as resolved and limited conversation to collaborators Sep 29, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
guidance Question that needs advice or information.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@srchase @bknill @usamamashkoor @chrisradek