Extremely slow performance for small files

[sohamm17]

I have folders with small files. For example, 25 MB with 3350 files. It's taking forever to upload. The estimated time is showing 3 hours.

[odeke-em]

Hello @sohamm17, welcome to drive and thank you for reporting this!

As you've probably experienced with other progress estimators, time to upload is just an estimate ( which is mostly never accurate or even close to accurate ). How long does it actually to upload?
If the time to perform the actual upload is slow then, that is a real issue but the time estimate is beyond my control. As you'll see during uploads it fluctuates.

Here is some fun

[odeke-em]

Btw I think this is a duplicate of #77

[sohamm17]

Yeah. I know it's an estimate. But in reality it takes high time. I think the issue has been reported already. Sorry for duplication.

[odeke-em]

3 hours really for 25MB?? That's amazing then. I've uploaded 1.9GB in about 20 minutes (from school and home) plus one of my deeply nested open source directories in less than 40 minutes.

[sohamm17]

Yeah. I have more than 3500 files in there. I feel this is a parallelism issue as you mention in that other thread. I wish I knew GO so that I could help you!

[odeke-em]

Oh yes, in deed it is a parallelism thing but still 3 hours for 3500 files is something else. I'll work on getting the parallelism back in, probably next week.

[sohamm17]

Yeah! It might be a different issue. It might be that a single file-uploading has some house-keeping task, and that kind of time-consuming tasks are done for every file in that directory. So, it's taking lots of time. Parallelism would definitely speed-up things to some extent.

[odeke-em]

@sohamm17 thank you for the patience! Please take a look at the work in progress PR #210.

[odeke-em]

@sohamm17 please run

go get -u github.com/odeke-em/drive/drive-gen && drive-gen

the latest code has parallel uploads and should handle errors quite well with retries.
Cheers!