Deeply nested files/directories cause performance issues.

[jpambrun]

Pushing 605 files for a total of 6 MB took about 16 minutes from a modern SSD over a 100+ mbps university Internet connection. Is this expected?

I was attempting to migrate from Dropbox using this tool. I have hundreds of thousands of small files totaling about 8GB. Is this a feasible use case?

This looks like a great tool and don't want to appear rude. I just want to know if I'm doing it wrong.

[odeke-em]

No worries, you aren't being rude, this is a legitimate concern. To be honest I think upload speed is relative. From my university I am able to push about 2GB in about 19 minutes. Your specs seem even more impressive. To start ruling out cases, you can try uploading a big file and see that the upload speed should be good. I suspect it has to do with setting up the respective hierarchies, hence queueing /scheduling might be the bottle neck. Also might you be uploading *.wav files?
Thank you for reporting this. Resolving this surely will improve user experience.

[jpambrun]

I have pushed another folder containing about 70 files for 750 MB and that took 120 sec. This seems great for few big files, but very slow for numerous small files. The big files were PDFs, the small ones Matlab source files.

[odeke-em]

So there you go, it is with the scheduling. I actually cut out a whole lot of parallelism to ensure correctness first. This is was lacking before and causing crashes due to resource exhaustion. Offline, I have been working on getting parallelism in and I have a couple of prototypes. However, I just need to do some heavy testing.

[jpambrun]

Got it. Thanks. Looking forward to it.
On Mar 3, 2015 4:06 PM, "Emmanuel Odeke" notifications@github.com wrote:

So there you go, it is with the scheduling. I actually cut out a whole lot
of parallelism to ensure correctness first. This is was lacking before and
causing crashes due to resource exhaustion. Offline, I have been working on
getting parallelism in and I have a couple of prototypes. However, I just
need to do some heavy testing.

—
Reply to this email directly or view it on GitHub
#77 (comment).

[odeke-em]

Watch this space: I'll ask for your help in testing it out, if you don't mind.

[odeke-em]

Hello folks,
Thank you very much for your patience. I got time this weekend to take a look at how to resolve this and I have spun up a work in progress PR #210 to address it. Please take a look at it if interested.

[odeke-em]

Please get the latest from master. Thank you very much for reporting this issue and for the discussions everyone, plus the patience. Closing it. Please re-open if it persists.

[gauteh]

I am having this issue with a bunch of source trees. I have been uploading a lot of stuff lately, could google be throttleing me?

[odeke-em]

Hey @gauteh please take a look at #480