uploading slow with tons of files

[W-i-n-7]

the uploading starts to get really slow when you are uploading files in the thousands
im uploading games to my site and they have alot of files and noticed the upload waits on 100% for more and more time the more files there are

[W-i-n-7]

its very slow with over 5000 files
and it affects on LAN too

[W-i-n-7]

opening multiple tabs slows down the other uploads

[rejetto]

i tried to reproduce this problem, but maybe my computer is too fast (m1 pro).
I even tried artificially slowing down with Chrome

still i could get over 200MB/s on a single large file, even with over 5k files in queue.
Can you still replicate the problem?
The easiest way to try is to "pick folder" and select a big one. You can do it multiple times, doesn't matter the files are duplicate.

I suspect the problem is not with the list in itself, but with its update on the screen. So, my idea is to not show all the items, but cut at 500.

[W-i-n-7]

my pc is very fast aswell could be even faster than yours am not sure (i5-13400f)
the issue is with uploading multiple small files it takes an age to get to the next file that completes instantly due to it being so small
so 90% of the time uploading is spent waiting for the next file to start uploading while theres multiple in queue

[W-i-n-7]

i tried to reproduce this problem, but maybe my computer is too fast (m1 pro). I even tried artificially slowing down with Chrome

still i could get over 200MB/s on a single large file, even with over 5k files in queue. Can you still replicate the problem? The easiest way to try is to "pick folder" and select a big one. You can do it multiple times, doesn't matter the files are duplicate.

I suspect the problem is not with the list in itself, but with its update on the screen. So, my idea is to not show all the items, but cut at 500.

https://hdun.org/random/hfs_upload_issues.mp4

[rejetto]

ah yes, you got files that are exceptionally small, 10 bytes each.
There's no decent way at the moment to get good performances.
Of course you may zip them before uploading.
I think I can add an "unzip" command without much effort. It could be available if you have upload permission.
Anyway, I also devised an optimization that should improve things already.
Let's see how faster you can get
0.52-beta6.2

[rejetto]

ok, i made a test myself.
we are passing from a performance index of 7 to 44 with the version above, but then i've managed to improve my technique and reach 75.
To my surprise, the main slowing factor was repainting of the list.

[W-i-n-7]

it repaints the whole list when 1 file dissapears?

[rejetto]

something like that

[W-i-n-7]

could there be expand/hide button if more than 500 files if hiddenwould show progress bar for all files in 1

[rejetto]

that's roughly what i did