Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync performance slow for many files #691

Open
onny opened this issue Sep 23, 2018 · 27 comments
Open

Sync performance slow for many files #691

onny opened this issue Sep 23, 2018 · 27 comments
Assignees
Labels

Comments

@onny
Copy link

@onny onny commented Sep 23, 2018

Expected behaviour

Sync should be able to process many small files in an appropriate time.

Actual behaviour

I tested syncing with 10MB of 1000 Files (bi-directional) and it took about 7 min. I guess this is an issue because the client uses WebDAV and starts an http-request for every single file.

Client configuration

Client version: 2.5.0 Beta 2

Operating system: ArchLinux x64

OS language: English

Qt version used by client package (Linux only, see also Settings dialog): 5.11.2

Client package (From ownCloud or distro) (Linux only): nextcloud-desktop-git (AUR)

Installation path of client: /

@camilasan camilasan self-assigned this Oct 1, 2018
@camilasan camilasan added this to To do in Desktop Client 2.5 Series via automation Oct 1, 2018
@rullzer rullzer removed this from To do in Desktop Client 2.5 Series Oct 19, 2018
@DevPre24
Copy link

@DevPre24 DevPre24 commented Feb 24, 2019

This is still an issue. I need to sync across multiple machines a cryfs container (3.7GB 240.000 files, 4000 subfolders) but the upload tops up at 20KB/s and it would take days to upload everything.

You could take a look at MegaSync approach to parallel uploads/downloads, their client is opensource too.

@cmorty
Copy link

@cmorty cmorty commented Jun 12, 2019

I have the same issue and had a quick peek using Process Monitor. It seems like most of the time is spent accessing sync.db. To me it seems like it's re-read in 4K steps for each file. With a DB of >100MB and thus over 25K systems calls thing must become slow.

@WilliamHorner
Copy link

@WilliamHorner WilliamHorner commented Dec 18, 2019

I have just experienced the same issue. 67,000 files, 24Gb.
NOTE: This post is not about the sync algorithm, it is about the serious deterioration in performance while files are transferring.

Important observations:

  1. The first 1000 files download in about 30 seconds. The next 1000 files in about 2 mins. The following 1000 in about 4 mins and so on. After a while it takes 1hr for 1000 files.
  2. If I stop the desktop client and restart it I immediate go back to 1000 files downloaded in about 30 seconds.
  3. The above is reproducible and consistent.
  4. If I look at the activity list it is jumping all over the place. The sync messages randomly change, reporting files that were downloaded previously. The files being reported are not the files actually downloading at that moment.
  5. The CPU is consistently running flat out at 100% as the performance deteriorates further.

So my guess is that there is some sort of array building up. The ability to get the thing going fast by simply restarting the service rules out the database in my opinion. However, restarting the service regularly takes a big hit with initialising the sync algorithm and so is not worthwhile until after about 8000 files have downloaded.

Taking a look at the code, I think the "ActivityListModel" class is a strong candidate for the above issues. In particular "combineActivityLists()" and the "resetModel" followed by "beginInsertRows". I am familiar with QAbstractListModel and the code in this method is very likely to exhibit the behaviour of all 5 of the above observations.

I need to understand a but more about how the ActivityList is built up but my guess is that a little attention to this section of code would reap significant benefits.

@otakugenx
Copy link

@otakugenx otakugenx commented Dec 22, 2019

I am running 2.6.1git on Linux Mint Cinnamon 19.3 and it is so slow. I just set this machine up and I am syncing 12GB and it is going apx 500k. It currently says 9 Hours left, 333MB of 12GB file 34 of 14492. I have 300MB internet and this has been running for apx 2 hours now. This is not the first machine to do this. Another machine I setup yesterday had the same results.

@cieska
Copy link

@cieska cieska commented Jan 8, 2020

So slow, that I have to leave PC on 247 to complete sync of my documents. Sometimes is just 10GB takes a few days.
OpenSuse TW. KDE, server on local GB LAN. CPU one core 70% of the time peaking on 100%. Server CPU idle.

@stendarr
Copy link

@stendarr stendarr commented Jan 8, 2020

I don't know how relevant this is to most of you but I'm gonna leave this here just in case:
I had a similar issue with really slow file transfers and high CPU usage, even though my client and server were in the same network and were communicating over gigabit Ethernet. As it turned out, the lousy router provided by my ISP was the bottleneck. I'm not sure how exactly it crippled my file transfers, but I'm pretty sure it was DNS because after I added the local IP of my Nextcloud server to the hosts file on my client machine, everything worked flawlessly and speeds were up by several orders of magnitude.

Still not sure what caused the high CPU usage exactly, but I haven't seen the Nextcloud client use more than a few % ever since.

Edit for clarification: When I add an entry to the hosts file so my server's domain resolves to its local IP I have really fast file transfers. As soon as I remove that entry, it goes back to being slow. This works on every client in my network, be it Linux, Windows, or Android - so I'm fairly certain it has something to do with my ISP provided router. Using a non-ISP router from ASUS I had fast file transfers either way.

@mburnicki
Copy link

@mburnicki mburnicki commented Jan 8, 2020

Just a guess: Is it possible that IPv6 is tried first, but it doesn't work, and after IPv6 has timed out, IPv4 is used and succeeds? Or vice versa?

@WilliamHorner
Copy link

@WilliamHorner WilliamHorner commented Jan 8, 2020

Thanks for those observations. I think we can use your observations too to support what I have seen inside the NextCloud code, but I would suggest that neither DNS nor ISP are likely to be related to the cause. Just the actions that took place when you made those modifications and I would suggest, some happy coincidences.

Basically, what I can see is that the NextCloud activity list builds up over time with many entries and becomes VERY slow and CPU intensive to update. When you restart the computer or bounce the nextcloud sync service on the PC, the activity list empties and everything goes fast again. And as you say, once the big sync of files has completed and you reboot/restart you will go back to a few % ever after (until you set out to resync the whole folder again from scratch).

I am quite confident that the root cause is a piece of the NextCloud client display update code that is also sitting inside the same file sync thread, and the display update has to complete before the sync continues for every single file, even if the client is not open on the screen! Not a particularly good code design.

I am hoping to find some time to test a fix to that section of code for NextCloud shortly. I can see exactly where it is happening in their code.

@KopfKrieg
Copy link

@KopfKrieg KopfKrieg commented Jan 8, 2020

Just a guess: Is it possible that IPv6 is tried first, but it doesn't work, and after IPv6 has timed out, IPv4 is used and succeeds? Or vice versa?

Theoretically yes, but I've deactivated IPv6 here (exactly because of this issue) and uploads are still as slow as always.


By the way, do we know where the bottleneck is? I mean, the problem of uploading many small files also occurs on Android, so is this a client or a server problem?

@error10
Copy link

@error10 error10 commented Jan 28, 2020

Just a guess: Is it possible that IPv6 is tried first, but it doesn't work, and after IPv6 has timed out, IPv4 is used and succeeds? Or vice versa?

Not here. My Nextcloud server is dual stack and works perfectly on IPv6. It's just uploading one small file at a time...

@dhrunia
Copy link

@dhrunia dhrunia commented May 16, 2020

I have a Nextcloud external storage configured on my self hosted Nextcloud instance. When I am syncing a folder (which has many subdirectories) from remote nextcloud storage the sync client spends more time checking for changes than downloading the contents themselves. Moreover, the sync client sometimes even crashes when I try to expand a folder(with many subdirectories) on external storage #1959.

@NicolasGoeddel
Copy link

@NicolasGoeddel NicolasGoeddel commented May 28, 2020

I am running the Nextcloud Client in version 2.6.4git on an Ubuntu 18.04 to sync 50,000+ directories. Starting the client lasts at least 15 minutes until it has discovered all local files and created the 50,000 inotify watchers. Then it seems to ask the server for every file if it has changed or not. It only handles a few files per second which in total leads to over an hour until the frontend is finally visible.
This is unacceptable. I don't know what the owncloud client makes better here but it starts within a few minutes with double the amount of directories and files. And most importantly the GUI is responsive all the time.

What's about the 2.7.0 beta version? Does it help in this case? Has anyone tested it already? It's really annoying that the client checks every single file against the server. It's soooo slow!

@Myridium
Copy link

@Myridium Myridium commented Jun 10, 2020

Yeah I have been trying with 2.6.4-1. Sorry to say this app is useless if this issue cannot be resolved. What a terrible oversight in what otherwise appears to be a great application.

@MinhHaDuong
Copy link

@MinhHaDuong MinhHaDuong commented Jun 26, 2020

@WilliamHorner blames the client display code. Does anybody know if the command line client works better?
https://docs.nextcloud.com/desktop/2.6/advancedusage.html#nextcloud-command-line-client

@jgonsior
Copy link

@jgonsior jgonsior commented Jul 3, 2020

I've switched back to Syncthing for syncing of the files because of this issue, and use Nextcloud just as a fancy web GUI and because of more easy to use iOS and Android clients.

@snowflakeas
Copy link

@snowflakeas snowflakeas commented Jul 6, 2020

Same problem with only a few hundred small files across a VPN (30Mbps up) the speed is in bits per second and frequent stalls.
2.6.4 client and NC 18 server.
Large files are fine.

@2bein
Copy link

@2bein 2bein commented Jul 31, 2020

I'm still on Nextcloud 17, will update soon. I cant confirm that first 1000 files go quick... I just synced 500 files (pdfs and gpx-tracks) of about 200 MB an it took half an hour over my GB LAN. I'm not a Linux freak, but managed to install nextcloud with MariaDB on my Synology DS412+ and using Client 2.6.5 on Windows 10 (Laptop core i7 8thGen). Will get DS420+ soon and see & hope if that helps... If not or no solution here, I think I will have to use Dropbox for small files, its faster even over my 10MB/s upload via ISP!!!

@Jonatino
Copy link

@Jonatino Jonatino commented Sep 2, 2020

Do we have any ideas on how we can address this issue? Would it make sense instead of http to use multiple sockets for constant stream of data? Or continue to use the current http/webdav way but allow for parallel processing of http requests?

@WilliamHorner
Copy link

@WilliamHorner WilliamHorner commented Sep 2, 2020

@NicolasGoeddel
Copy link

@NicolasGoeddel NicolasGoeddel commented Sep 2, 2020

So the model is completely dependent from the view? That's a bummer.
It would be so nice if the synchronisation process itself would be independent from all the QT things. So one could use it also on a headless server.

@Jonatino
Copy link

@Jonatino Jonatino commented Sep 2, 2020

@WilliamHorner so this is a purely client sided issue? The server doesn't have any issues serving/retrieving large amounts of files?

@Jonatino
Copy link

@Jonatino Jonatino commented Sep 2, 2020

Just adding more use case to this issue: https://dl.dropboxusercontent.com/s/0xci50aox5myxek/SywW6g1ErD.png

It's been over 2 days to sync 68gb of files on a 10gigabit network with nextcloud 100% on SSDs.

@cmorty
Copy link

@cmorty cmorty commented Sep 2, 2020

@Jonatino I'm pretty sure it's client side, as things get slower and slower. Restarting nexclound every now and then brings things up to speed for a short while....

@moro1111
Copy link

@moro1111 moro1111 commented Nov 23, 2020

Same problem with the desktop sync client, while uploading many small files:

  • 21k files
  • 15GB
  • 4 days to upload (according to the forecast of the client)

The client is uploading only 1-2 files per second which results in a speed of sometimes < 1 kb/s (!), which is insane. While for bigger file, the speed reaches about 3 mb/s.
Due to that, and since the CPU runs at stable 10% with about 8gb RAM while 16gb are available, i don't think, that the network-connection, the CPU speed, RAM, etc. are bottlenecking the process (~3 mb/s are possible).

Loading 15gb in 4 days is ridiculous and makes nextcloud for my case just unusable.

(I assume the desktop client is loading each file one-by-one, instead of packing a bunch of small files together in one .zip and load them all together to max out on the network speed. This results in a massive overhead for many small files, since they are all handled and loaded separately. But, as i said, thats just what i assume.)

@agross
Copy link

@agross agross commented Nov 23, 2020

@moro1111 Your assessment is correct. HTTP is a very chatty protocol and you have to announce/negotiate each file, which results in big overhead. The Nextcloud clients are in desperate need of a pipelining/streaming protocol. I believe that with http/2 this could be achieved.

@moro1111
Copy link

@moro1111 moro1111 commented Nov 23, 2020

@agross Good to know, thank you! But then i assume there is currently no real way to fix that, or find a workaround by messing with the config of Apache, PHP, Nextcloud, etc?
Do you know, if they are currently working on changing the communication protocol, so one could hope that this will be fixed in some time?
... i fear the wont, since this problem is already 2 years old.

@Jonatino
Copy link

@Jonatino Jonatino commented Nov 23, 2020

@moro1111 I've switched to seafile until this issue is fixed. With the same library seafile synced 70gigs with ~600,000 files in less than 2 minutes over my 10gbit LAN network. The web UI on nextcloud is far better though imo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.