Sync performance slow for many files

[onny]

Expected behaviour

Sync should be able to process many small files in an appropriate time.

Actual behaviour

I tested syncing with 10MB of 1000 Files (bi-directional) and it took about 7 min. I guess this is an issue because the client uses WebDAV and starts an http-request for every single file.

Client configuration

Client version: 2.5.0 Beta 2

Operating system: ArchLinux x64

OS language: English

Qt version used by client package (Linux only, see also Settings dialog): 5.11.2

Client package (From ownCloud or distro) (Linux only): nextcloud-desktop-git (AUR)

Installation path of client: /

[DevPre24]

This is still an issue. I need to sync across multiple machines a cryfs container (3.7GB 240.000 files, 4000 subfolders) but the upload tops up at 20KB/s and it would take days to upload everything.

You could take a look at MegaSync approach to parallel uploads/downloads, their client is opensource too.

[cmorty]

I have the same issue and had a quick peek using Process Monitor. It seems like most of the time is spent accessing sync.db. To me it seems like it's re-read in 4K steps for each file. With a DB of >100MB and thus over 25K systems calls thing must become slow.

[WilliamHorner]

I have just experienced the same issue. 67,000 files, 24Gb.
NOTE: This post is not about the sync algorithm, it is about the serious deterioration in performance while files are transferring.

Important observations:

1. The first 1000 files download in about 30 seconds. The next 1000 files in about 2 mins. The following 1000 in about 4 mins and so on. After a while it takes 1hr for 1000 files.
2. If I stop the desktop client and restart it I immediate go back to 1000 files downloaded in about 30 seconds.
3. The above is reproducible and consistent.
4. If I look at the activity list it is jumping all over the place. The sync messages randomly change, reporting files that were downloaded previously. The files being reported are not the files actually downloading at that moment.
5. The CPU is consistently running flat out at 100% as the performance deteriorates further.

So my guess is that there is some sort of array building up. The ability to get the thing going fast by simply restarting the service rules out the database in my opinion. However, restarting the service regularly takes a big hit with initialising the sync algorithm and so is not worthwhile until after about 8000 files have downloaded.

Taking a look at the code, I think the "ActivityListModel" class is a strong candidate for the above issues. In particular "combineActivityLists()" and the "resetModel" followed by "beginInsertRows". I am familiar with QAbstractListModel and the code in this method is very likely to exhibit the behaviour of all 5 of the above observations.

I need to understand a but more about how the ActivityList is built up but my guess is that a little attention to this section of code would reap significant benefits.

[otakugenx]

I am running 2.6.1git on Linux Mint Cinnamon 19.3 and it is so slow. I just set this machine up and I am syncing 12GB and it is going apx 500k. It currently says 9 Hours left, 333MB of 12GB file 34 of 14492. I have 300MB internet and this has been running for apx 2 hours now. This is not the first machine to do this. Another machine I setup yesterday had the same results.

[cieska]

So slow, that I have to leave PC on 247 to complete sync of my documents. Sometimes is just 10GB takes a few days.
OpenSuse TW. KDE, server on local GB LAN. CPU one core 70% of the time peaking on 100%. Server CPU idle.

[stendarr]

I don't know how relevant this is to most of you but I'm gonna leave this here just in case:
I had a similar issue with really slow file transfers and high CPU usage, even though my client and server were in the same network and were communicating over gigabit Ethernet. As it turned out, the lousy router provided by my ISP was the bottleneck. I'm not sure how exactly it crippled my file transfers, but I'm pretty sure it was DNS because after I added the local IP of my Nextcloud server to the hosts file on my client machine, everything worked flawlessly and speeds were up by several orders of magnitude.

Still not sure what caused the high CPU usage exactly, but I haven't seen the Nextcloud client use more than a few % ever since.

Edit for clarification: When I add an entry to the hosts file so my server's domain resolves to its local IP I have really fast file transfers. As soon as I remove that entry, it goes back to being slow. This works on every client in my network, be it Linux, Windows, or Android - so I'm fairly certain it has something to do with my ISP provided router. Using a non-ISP router from ASUS I had fast file transfers either way.

[mburnicki]

Just a guess: Is it possible that IPv6 is tried first, but it doesn't work, and after IPv6 has timed out, IPv4 is used and succeeds? Or vice versa?

[WilliamHorner]

Thanks for those observations. I think we can use your observations too to support what I have seen inside the NextCloud code, but I would suggest that neither DNS nor ISP are likely to be related to the cause. Just the actions that took place when you made those modifications and I would suggest, some happy coincidences.

Basically, what I can see is that the NextCloud activity list builds up over time with many entries and becomes VERY slow and CPU intensive to update. When you restart the computer or bounce the nextcloud sync service on the PC, the activity list empties and everything goes fast again. And as you say, once the big sync of files has completed and you reboot/restart you will go back to a few % ever after (until you set out to resync the whole folder again from scratch).

I am quite confident that the root cause is a piece of the NextCloud client display update code that is also sitting inside the same file sync thread, and the display update has to complete before the sync continues for every single file, even if the client is not open on the screen! Not a particularly good code design.

I am hoping to find some time to test a fix to that section of code for NextCloud shortly. I can see exactly where it is happening in their code.

[KopfKrieg]

Just a guess: Is it possible that IPv6 is tried first, but it doesn't work, and after IPv6 has timed out, IPv4 is used and succeeds? Or vice versa?

Theoretically yes, but I've deactivated IPv6 here (exactly because of this issue) and uploads are still as slow as always.

---

By the way, do we know where the bottleneck is? I mean, the problem of uploading many small files also occurs on Android, so is this a client or a server problem?

[error10]

Just a guess: Is it possible that IPv6 is tried first, but it doesn't work, and after IPv6 has timed out, IPv4 is used and succeeds? Or vice versa?

Not here. My Nextcloud server is dual stack and works perfectly on IPv6. It's just uploading one small file at a time...

[dhrunia]

I have a Nextcloud external storage configured on my self hosted Nextcloud instance. When I am syncing a folder (which has many subdirectories) from remote nextcloud storage the sync client spends more time checking for changes than downloading the contents themselves. Moreover, the sync client sometimes even crashes when I try to expand a folder(with many subdirectories) on external storage #1959.