Spliting into multiple chunks causing upload to get an 503 error

[pinghsien422]

@tanaikech

Hi,

Can you to see if there's a bug in your library, because in the response:

if (res.status == "Uploading") {
  msg =
    Math.round(
      (res.progressNumber.current / res.progressNumber.end) * 100
    ) + "%";
} else {
  msg = res.status;
}

The res.progressNumber.end is always just 1. Is there a lower limit as to we can set the chunkSize to be?
My logic in spliting the chunk size is as follows:

var chunk_size = Math.max(Math.min(Math.floor(e.target.files[0].size / 5), 5242880), 1048576);

It should be spliting the file uploads by 5 chunks, but it's not. Below shows my debug output:

file size: 9729426
chunkSize: 1945885
res.progressNumber.current: 0
res.progressNumber.end: 1

The upload only happened once, not 5 times.

[pinghsien422]

Digging into the src code, I see that chunkSize must be multiples of 1024 in order to take effect (Might be a good idea to document that requirement somewhere), so I updated my code to do this:

function calculateChunkSize(file_size) {
    var chunk_size = Math.floor(Math.floor(file_size / 5) / 1024) * 1024;
    chunk_size = Math.max(Math.min(chunk_size, 5242880), 1048576);
    return chunk_size;
}

And successfully to get the file to split into 6 chunks. However, after doing this, the code just hangs for a long time in the first chunk, and errors out on the second chunk:

?id=52:4164			file size: 9653164
?id=52.4165			chunkSize: 1930240
?id=52:3795			res.progressNumber.current: 0
?id=52.3796			res.progressNumber.end: 6
?id=52.3795			res.progressNumber.current: 1
?id=52.3796			res.progressNumber.end: 6
resumableupload2_js.js:196	PUT https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable&fields=id%2CwebContentLink%2CwebViewLink%2CiconLink%2CthumbnailLink%2Csize&upload_id=ADPycdv-tnnS1t7HK4Eb7TgJ2aWtpgwuG3jtKCjwSvm8q2pcvwji2ejKrbw6DxtClP73oetPAhjeFAWE0Y7OlvYihq2oIw 503
VM576:1				Uncaught (in promise) SyntaxError: Unexpected token 'I', "Invalid re"... is not valid JSON

This seem to have something to do with having too many requests... when I split into 2 chunks things are fine, but fails if split into 6 chunks.

[tanaikech]

Thank you for your comment. When I tested a sample file (the size is 12,081,664 bytes) using my library, the number of chunks is 10. And I confirmed that the file could be correctly uploaded. Unfortunately, I cannot replicate your situation. I apologize for this. But, I would like to try to correctly replicate your situation. When I could correctly replicate your situation, I would like to think of a solution. I deeply apologize that when I tested my library, no error occurs.

[pinghsien422]

Thank you for your response. Below is the callstack for your appraisal, I will do more testing too, and when I find more clues that may help you, I will let you know.

Uncaught (in promise) SyntaxError: Unexpected token 'I', "Invalid re"... is not valid JSON
Promise.then (async)
(anonymous) 						@ resumableupload2_js.js:208
Promise.then (async)
(anonymous) 						@ resumableupload2_js.js:201
doUpload 						@ resumableupload2_js.js:195
fr.onload 							@ resumableupload2_js.js:186
load (async)
(anonymous) 						@ resumableupload2_js.js:174
getFile @ resumableupload2_js.js:172
ResumableUploadToGoogleDrive2.Do 	@ resumableupload2_js.js:59
await in ResumableUploadToGoogleDrive2.Do (async)
(anonymous) 						@ ?id=52:4205
(anonymous) 						@ ?id=52:3884
i 								@ jquery-1.12.4.min.js:2
fireWith 							@ jquery-1.12.4.min.js:2
y 								@ jquery-1.12.4.min.js:4
c 								@ jquery-1.12.4.min.js:4
XMLHttpRequest.send (async)
send 							@ jquery-1.12.4.min.js:4
ajax 								@ jquery-1.12.4.min.js:4
n.<computed> 					@ jquery-1.12.4.min.js:4
onCreateGoogleDriveFile 			@ ?id=52:3877
(anonymous) 						@ ?id=52:4192
dispatch 							@ jquery-1.12.4.min.js:3
r.handle 							@ jquery-1.12.4.min.js:3

This error doesn't occur if I set the chunkSize statically to 5,242,880 with a file size 33,587,748 bytes, uploading in 7 chunks, it succeeds. When I switched to using the calculateChunkSize() above to dynamically calculate the chunkSize, it fails. I'm not sure if google has a limitation of what the chunkSize must be.

[pinghsien422]

I thnk I figured out the issue. Google API document on resumable upload states:

Add the chunk's data to the request body. Create chunks in multiples of 256 KB (256 x 1024 bytes) in size, except for the final chunk that completes the upload. Keep the chunk size as large as possible so that the upload is efficient.

So this means the chunkSize must be a multiple of 262,144 bytes instead of 1024. Updating my calculateChunkSize() to the following works:

function calculateChunkSize(file_size) {
    var chunk_size = Math.floor(Math.ceil(file_size / 5) / 262144) * 262144;
    chunk_size = Math.max(Math.min(chunk_size, 7864320), 2621440);
    return chunk_size;
}

You might want to update the logic below on the condition to apply resource.chunkSize to ensure it is a multiple of 262144:

if (
    "chunkSize" in resource &&
    resource.chunkSize >= 262144 &&
    resource.chunkSize % 1024 == 0 // Change this to resource.chunkSize % 262144 == 0
)

[tanaikech]

Thank you for replying. In my this answer, I clearly wrote about the specification of a chunk. https://stackoverflow.com/a/61133144 But, I noticed that I didn't add it in this library. So, I added it in README.md.

[tanaikech]

Now, I updated this library by reflecting on the discussion here.

[pinghsien422]

Thank you very much for your fix!

[tanaikech]

Welcome. And, I appreciate your contribution.