S3: tracking overall upload progress for multiple files.

[andrey-bahrachev]

Hey there!
Currently it's possible to keep track of upload progress only on per file basis, but there's no way of calculating the overall upload progress when multiple files are uploaded in parallel. It would be really great to have such ability. Thanks.

[andrey-bahrachev]

Ah, found a way around using this in the upload progress callback.

s3.upload(params, function (err, data) {
    ...
}).on('httpUploadProgress', function(progress) {
    // Here you can use `this.body` to determine which file this particular
    // event is related to and use that info to calculate overall progress.
});

I hope this will help someone else.
This wasn't obvious though and it still would be nice to have an easier way of doing this.

[chrisradek]

@andrey-bahrachev
We do have a pull request, #679 that would expose the object's key as part of the progress. I had a comment on it, but can look into expediting merging it in.

[andrey-bahrachev]

@chrisradek Thanks for quick response.
That would be awesome!

[chrisradek]

I've merged in #788. This will be available via npm with the next release of the SDK, otherwise you can install the SDK directly from github.

[andrey-bahrachev]

Brilliant! Thanks a lot!

[rossthedevigner]

@andrey-bahrachev Do you have an example how you're monitoring overall progress based on this.body info?

[andrey-bahrachev]

@rossthedevigner Oh, it's been a while, so chances are that things might have changed since then, but here it is:

/**
 * Uploads file(s) directly to AWS S3.
 * 
 * @param form -  jQuery object representing the form.
 * @param {array} files -  Files to be uploaded (derived from HTML5 File API).
 * @param successCallback - Callback function to be called upon successful file(s) upload.
 */
uploadFiles: function(form, files, successCallback) {
    var filesData = [];
    var sizeTotal = 0;
    var loaded = [];
    for (var i = 0; i < files.length; i++) {
        filesData.push({name: files[i].name, size: files[i].size, type: files[i].type});
        sizeTotal += files[i].size;
        loaded[files[i].name] = 0;
    }
    app.checkStorage(form, filesData, function(resp) {
        // Initialize the Amazon Cognito credentials provider.
        AWS.config.region = resp.cognitoRegion;
        AWS.config.sslEnabled = true;
        AWS.config.credentials = new AWS.CognitoIdentityCredentials({IdentityPoolId: resp.cognitoPoolId});
        // Initialize S3 service handler.
        var s3 = new AWS.S3({params: {Bucket: resp.bucket}});
        var uploadBar = form.find('.upload-bar');
        var uploadBarMeter = uploadBar.find('div');
        uploadBar.removeClass('hidden');
        for (var i = 0, filesLen = files.length; i < filesLen; i++) {
            var params = {Key: resp.files[i].key, ContentType: files[i].type, Body: files[i], ACL: 'private'};
            s3.upload(params, function (err, data) {
                if (err) {
                    yii.app.activateForm(form);
                    swal('File upload error');
                    return false;
                }
            }).on('httpUploadProgress', function(e) {
                loaded[this.body.name] = e.loaded;
                var loadedTotal = 0;
                for (var j in loaded) {
                    loadedTotal += loaded[j];
                }
                var progress = Math.round(loadedTotal / sizeTotal * 100);
                uploadBarMeter.css({width: progress + '%'});
                if (loadedTotal === sizeTotal) {
                    successCallback(form, {files: resp.files, form: form.serialize()});
                }
            });
        }
    });
},

[smrgrg]

Thanks a lot!

[vedangkoolkarni]

God bless you for this!

[venu20]

@andrey-bahrachev with the above approach, If i make 20 to 30k calls it would be really slow. Is there any other approach to do bulk upload to S3 ?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.