New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local file discovery should be done at most once per run, use file watcher events to seed Discovery #2297

Closed
ckamm opened this Issue Oct 10, 2014 · 65 comments

Comments

@ckamm
Member

ckamm commented Oct 10, 2014

Currently the client runs a full local discovery every five minutes due to limitations in our file watching code (see #2268 for previous discussion). These limitations should be removed to allow the client to be reliable with only a single run of the local file discovery or without it entirely.

On Mac: We don't get notifications for changes our own process caused, so a single run should be enough. We may be able to hook into the filesystem change journal to even skip that.

On Linux: We get full path information about changes and that could be used to filter out the changes on files we cause ourselves.

On Windows: Similar to Linux (have to use ReadDirectoryChanges instead of FindFirstChangeNotification) and could even possibly skip the single discovery by using "Change Journals".

@moscicki

This comment has been minimized.

Contributor

moscicki commented Oct 10, 2014

These kind of stuff becomes important as we see people having 50K+ files in their sync folder. If you manage to sort this out to work at scale of a few 100K files in a sync directory then you will have extreme competitive advantage. But that's some way to go, still....

@guruz

This comment has been minimized.

Collaborator

guruz commented Oct 22, 2014

Linking #880 since it's related.
I still have some ideas for Windows directory traversal..

@ckamm

This comment has been minimized.

Member

ckamm commented Nov 6, 2014

@moscicki points out in #2451:

Please also consider that in some cases an (advanced) user may require to keep the ability of periodic full scans of the local sync folder (on per folder-pair basis). This is if someone have their local folders on mounted filesystems which do not always reliably provide the notifications (e.g. in some cases the AFS). In that case a setting in the folders config file could be sufficient.

ckamm added a commit that referenced this issue Nov 20, 2014

Windows filewatcher: switch to ReadDirectoryChangesW.
Based on danimo's #2454 fix for #2455 and related to #2297.

ckamm added a commit to ckamm/owncloud-client that referenced this issue Nov 20, 2014

OSX: Trust the file watcher. owncloud#2297
* Use a bigger default force sync interval (2h)
* Allow the file watcher to schedule a sync while a sync for
  the same file is running.

ckamm added a commit to ckamm/owncloud-client that referenced this issue Nov 20, 2014

ckamm added a commit to ckamm/owncloud-client that referenced this issue Nov 20, 2014

@ckamm

This comment has been minimized.

Member

ckamm commented Nov 20, 2014

The current state in master is:

  • Force sync interval changed from 5 minutes to 2 hours
  • The file watchers report changes with full paths
  • The propagator keeps track of files the sync touches and then folders ignore change notifications on files that were touched by the sync for 15s

This should make full discovery runs much more rare!

@reylon

This comment has been minimized.

reylon commented Dec 5, 2014

Seems you have almost perfect solved the problem. Exist a simple solution for a beginner that installed all standard? Owcloud server 7.0.3 + debian7 + nginx + php5 + mysql
Thanks

@guruz

This comment has been minimized.

Collaborator

guruz commented Apr 19, 2018

People interested in testing this should have a look at the recent daily builds of 2.5: https://download.owncloud.com/desktop/daily/

@guruz

This comment has been minimized.

Collaborator

guruz commented Jun 4, 2018

This is part of 2.5.0 alpha1, did anyone give it a try? :) https://github.com/owncloud/client/releases

@coreyman

This comment has been minimized.

coreyman commented Jul 8, 2018

Tried to install it via the MSI and got a ton of .dll errors and registry issues.

@guruz

This comment has been minimized.

Collaborator

guruz commented Jul 8, 2018

@coreyman And how is the MSI for a daily build? (There were fixes in meanwhile)
https://download.owncloud.com/desktop/daily/ownCloud-2.6.0.315-daily20180708.343.msi

@coreyman

This comment has been minimized.

coreyman commented Jul 9, 2018

@guruz the daily build you linked seems to be working good. The local detection is a lot faster. I have probably a little more than 80k files and can notice the difference. The only issue now is the remote detection being slow :)

@guruz

This comment has been minimized.

Collaborator

guruz commented Jul 23, 2018

2.5.0 beta1 is out :-)
https://central.owncloud.org/t/desktop-sync-client-2-5-0-beta1-released/14667
Everyone, please comment here if we can close this issue. Thank you.

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 3, 2018

Test report from @coreyman indicates we are good here.
With small test sets under 2000 files there is not much difference between 2.4.1 and 2.5.0beta1 in speed seen here.
For what it is worth: For a long time I have not seen lengthy discovery phases at all.

This speedup should be announced with 2.5.0

OK: 'Force Sync now' with 1700 files previously synced is instantly, was 1..2sec with 242
SKIP: I cannot think of a good local test setup for this. Please reopen if there is an easy one.

@jnweiger jnweiger closed this Aug 3, 2018

@guruz

This comment has been minimized.

Collaborator

guruz commented Aug 10, 2018

This speedup should be announced with 2.5.0

Yep it should. I hope @michaelstingl @lefherz @JKawohl can somehow pack this in a blogpost with graphs?

@moscicki

This comment has been minimized.

Contributor

moscicki commented Aug 11, 2018

@michaelstingl

This comment has been minimized.

Collaborator

michaelstingl commented Aug 11, 2018

@jnweiger please share your test setup and procedure.

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 12, 2018

As mentioned above, I have not yet found a good way to test.

Maybe generate

  • 33 folders with 3300 files
  • 330 folders with 330 files
  • 3300 folders with 33 files
  • 10/10/10/10 foldertree with 10 files

then do wall clock time of the discovery phase for

  • initial sync up
    *1000 added files scattered
  • 1000 added files in one folder
  • 1000 files changed in gew folders
  • 1000 files changed in 1000 folders.

all this with 2.4.1 and 2.4.2 and 2.5.0

@guruz should that reveal the speedup?

@ogoffart

This comment has been minimized.

Collaborator

ogoffart commented Aug 13, 2018

The speedup is only visible for consecutive sync (i.e: not the first sync after creating the client.)

Test case: in a very large sync folder (by large i mean huge amount of directories and files), do a small modification to a resonably small file. Time the time it takes for that file to be on the server. It should be much faster with 2.5
(Note, should be especially true if the HD is slow, and the fs cache is empty)

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 14, 2018

@ogoffart I have difficulties measuring the speedup. Here are my results:

Testing on a slow USB-2 SSD, server running on the same machine on a SATA SSD.
10000 file of 6 bytes each, scattered in 100 folders.

for i in $(seq 1 100); do echo $i; mkdir -p "t$i"; for j in $(seq 1 100); do echo hello > t$i/f$j; done; done

client 2.5.0daily20181014

  • initial sync up: 7:04.4 sec
  • first force resync: 8.9 sec
  • 2nd, 3rd force resync: 0.5 sec
  • client stop; start: 1.5 sec
  • small file change: 0.5 sec
  • client stop; rm db; client start: 20.8 sec

client 2.4.3

  • initial sync up: 8:01.0 sec
  • first force resync: 1.3 sec
  • 2nd, 3rd force resync: 0.5 sec
  • client stop; start: 1.5 sec
  • small file change: 0.5 sec
  • client stop; rm db; client start: 22.8 sec

client 2.4.1

  • initial sync up: 8:06.0 sec
  • first force resync: 12.2 sec
  • 2nd, 3rd force resync: 0.5 sec
  • client stop; start: 1.5 sec
  • small file change: 0.5 sec
  • client stop; rm db; client start: 22.0 sec
@Trefex

This comment has been minimized.

Trefex commented Aug 14, 2018

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 14, 2018

@Trefex I'll need to avoid the initial sync for that, otherwise we get results next month. Will try.

@Trefex

This comment has been minimized.

Trefex commented Aug 14, 2018

@dragotin

This comment has been minimized.

Contributor

dragotin commented Aug 14, 2018

Why is 2.4.3 so much faster on 'first force resync:' compared to the other two? Typo?

@moscicki

This comment has been minimized.

Contributor

moscicki commented Aug 15, 2018

What about inotify watch limits? Also with 1M files it will take around 1GB of kernel memory according to: https://unix.stackexchange.com/questions/13751/kernel-inotify-watch-limit-reached

Will the sync client actually warn the user that there are more files to watch than the current limit?

Also, I would be good to check/document what would be the practical limits for oc sync client per platform (OSX, Win, Linux)?

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 15, 2018

@dragotin Not exactly sure, but I noticed that with 2.4.3 when the initial sync finished, the ichon was green only for one or two seconds, then it became blue again and was busy for another 15 seconds. My guess is, it then already did, what the other two would do only on the first resync. Tried twice with same behaviour.

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 15, 2018

@moscicki yes, we warn the user when we run out of inotify listeners. See #6119 for details.
I am curious what exact limit is needed for 1M files, and how much kmem that adds.
My 10k files test did not hit the limit 8000.

@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 15, 2018

200k files - a demonstrator that shows good speedup in 2.5.0

Summary: 200.000 files scattered in 2.000 folders on the server are in sync with a 2.4.1 and a 2.5.0 desktop client. When editing, adding or deleting a single file, the 2.4.1 client needs less than 10 seconds.

for ii in $(seq 1 100); do for i in $(seq 1 100); do echo "$i/$ii"; mkdir -p "t$i/$ii"; for j in $(seq 1 100); do cp /etc/samba/smb.conf t$i/$ii/f$j; done; done; done

have enough inotify watches for 1M files.
echo 200000 | sudo tee -a /proc/sys/fs/inotify/max_user_watches

client 2.5.0daily20181014

  • vmstat-m.20180815-13:03:56.txt
  • selective sync with t1 t2 t3 t4 t5 t6 t7 t8 t9 10 (100k file).
  • initial 'sync' with sqlite removed: 2:05.1 sec
  • enabling folders t11 t12 t13 t14 t15 t16 t17 t18 t19 t20 (new total 200k files)
    (false warning printed, that unchecked folders will be removed. they are not removed.)
  • second 'sync' with more folders added:
    • Thousands of false messages scroll through the activity log: 'Datei ist seit der Entdeckung geändert worden'
    • Files are being downloaded from the server although identical files exist locally.
    • The client process grows to 1.8GB memory footprint.
    • Duration: 38:34.0 sec (all 4 CPUs of a quadcore busy with client and server activity)
    • 5 sec idle, then another sync: 10:28.3 sec (one CPU busy, no server activity seen)
    • The client process drops to 1.1GB memory footprint.
  • vmstat-m.20180815-14:28:33.txt
  • client stop; start: 45.5 sec
  • small local file change: 6.6 sec, 5.1 sec (including 3 sec before the sync starts)
  • small remote file change: 3.4 sec, 4.4 sec

client quit: vmstat-m.20180815-14:29:55.txt

client 2.4.1

  • selective sync with t1 t2 t3 t4 t5 t6 t7 t8 t9 10 (100k file).
  • initial 'sync' with sqlite removed: 3:20.2 sec
  • enabling folders t11 t12 t13 t14 t15 t16 t17 t18 t19 t20 (new total 200k files)
  • second 'sync' with more folders added:
    • The client process grows to 2.4GB memory footprint.
    • 80 messages correclty name the remaining unsynced folder. (Nothing scrolls wildly.)
    • Duration: 10:15.6 sec
    • The client process remains at to 2.4GB memory footprint.
  • first force resync: 48.2 sec
  • 2nd, 3rd force resync: 46.5 sec, 35.2 sec
  • client stop; start: 28.5 sec
  • small local file change: 33.7 sec, 45.1 sec
  • small remote file change: 25.1 sec, 31.3 sec
@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 15, 2018

1M files - a demonstrator that shows moderate speedup in 2.5.0

Summary: 1.000.000 files scattered in 10.000 folders on the server are in sync with a 2.4.1 and a 2.5.0 desktop client. When editing, adding or deleting a single file, the 2.4.1 client needs between 1 and 2 minutes, but 2.5.0 only needs 20 to 30sec to sync to the server.

client 2.5.0daily20181014

  • client stop; rm db; client start: ca 15 min (1M files are now included)
  • small local file change: 30.1 sec, 22.0 sec, 20.3 sec
  • small remote file change: 1:05.5 sec, 27.4 sec, 51.9 sec

client 2.4.1

  • client stop; rm db; client start: ca 15 min (1M files are now included)
  • small local file change: 2:48.8 sec, 1:04.7 sec, 1:02.6 sec
  • small remote file change: 1:02.1 sec, 1:45.3 sec, 1:03.2 sec
@jnweiger

This comment has been minimized.

Contributor

jnweiger commented Aug 16, 2018

@moscicki Good news about inotify watch limits:
With a test tree of exactly 1010010 files in 10103 folders, the official number of watches appears to be 10390.
This number was found by bisecting /proc/sys/fs/inotify/max_user_watches settings.
Compared with earlier tests, this confirms the impression that we have one watch per folder plus ca. 200 -- the number of files seems irrelevant.

Linux slabinfo (via vmstat -s and slabtop, using echo 3 > /proc/sys/vm/drop_caches) did not seem to be accurate enough to actually measure additional kernel memory consumption for the watches. In some cases kernel memory footprint even appeared to be smaller while the client was watching the tree.

@gnanet

This comment has been minimized.

gnanet commented Sep 2, 2018

@jnweiger i consider switching from 2.4.3 to a 2.5 client on Ubuntu 14.04.5,
while checking /proc/sys/fs/inotify/max_user_watches the value is: 65536

So i looked into /etc/sysctl.d and found these settings with grep inotify.max_user_watches *.conf

100-owncloud-inotify.conf:fs.inotify.max_user_watches = 524288
100-sync-inotify.conf:fs.inotify.max_user_watches = 524288
30-tracker.conf:fs.inotify.max_user_watches = 65536

AGAIN the current value is 65536, this confirms my assumption, that the usual load-order specification has two digits for a reason:
According the docs

All configuration files are sorted by their filename in lexicographic order, regardless of which of the directories they reside in. If multiple files specify the same option, the entry in the file with the lexicographically latest name will take precedence. It is recommended to prefix all filenames with a two-digit number and a dash, to simplify the ordering of the files.

The #4107 (comment) suggesting to set the owncloud-inotify.conf load-order to 100, and the resulting commit owncloud/administration@8218eb3 created a bug.

@guruz

This comment has been minimized.

Collaborator

guruz commented Sep 3, 2018

@gnanet Thanks for debugging. Your suggestion would be 99-*-inotify.conf?

@ogoffart

This comment has been minimized.

Collaborator

ogoffart commented Sep 3, 2018

@ogoffart ogoffart closed this Sep 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment