Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up delete propagation when it's not a rename #4953

Closed
codl opened this Issue May 14, 2018 · 7 comments

Comments

Projects
None yet
6 participants
@codl
Copy link

codl commented May 14, 2018

As I understand it, the file watcher batches edits by 10s periods and deletes by 60s periods

However it seems that if a file is deleted and then recreated shortly after, then that whole edit will be delayed by 60 seconds. Is this intentional? This is essentially the same thing as clobbering a file so I would expect the delay to be the same as clobbering.

Version Information

Syncthing Version: v0.14.47
OS Version: Arch Linux

@imsodin

This comment has been minimized.

Copy link
Member

imsodin commented May 14, 2018

Yes it is intentional.

I don't know what clobbering means, maybe it's relevant to know that the FS watcher/aggregation/delay part does not know about file contents.

I am currently struggling with making up a concrete example where the policy actually helps, but the general idea is: Existing files can be delayed up to 6x the wait time if modified multiple times. New files can be reconstructed from existing files on the remote, so delay anything that was deleted (i.e. also delete&create) by 6x the wait time too.
I mean I can come up with a scenario, but it is not very convincing: Rename file A to B, create a different file A and keep modifying the new file B, without changing its content much/at all. Then if A wasn't delayed, B wouldn't be reconstructed from A on the remote (as A was already replaced).

However I think it's fine to err on the save side, so unless someone has some convincing arguments against it, I'd keep the current behaviour.

@Ferroin

This comment has been minimized.

Copy link

Ferroin commented May 14, 2018

IMO, the current behavior (10 second batching for most stuff, 6x that for deletion or multiple modifications) is sensible for most use cases, but it would be kind of nice to be able to tweak both values on a per-folder basis. As a couple of examples:

  • I have a folder where I have a bunch of playlists, and when I edit those, I do so pretty slowly, and would much rather batch all the changes I make into one update, so for that folder, it would be nice to be able to bump up both values significantly.

  • On the other side of things, I do have a small handful of things where changes are very infrequent and consist of single writes, but I want them propagated as fast as possible, even if it means that data may not be reused as much as it otherwise would. In this case, it would be wonderful to just drop the batching down to 1 second, and remove the multiplier.

@imsodin

This comment has been minimized.

Copy link
Member

imsodin commented May 14, 2018

Lets continue this discussion on the forum: https://forum.syncthing.net/t/discussing-fs-watcher-delays-4953/11593

I am not dismissing the issue, it's a legitimate question and there might be need for change, but there is still clearly need to explain and discuss before a clear path forward (if any) turns up. For that the forum is the better venue (reducing mail noise). Will be reopened if necessary.

@imsodin imsodin closed this May 14, 2018

imsodin added a commit to imsodin/syncthing that referenced this issue May 15, 2018

@imsodin

This comment has been minimized.

Copy link
Member

imsodin commented May 15, 2018

I am reopening this as I want to close it by my proposal on the accompanying forum discussion:

I am now implementing a mechanism, that will speed things up in most cases:
If there are no “creations” being delayed, “removals” don’t get an additional delay either (i.e. only the normal 10s instead of 1min).

@imsodin imsodin reopened this May 15, 2018

calmh added a commit to calmh/syncthing that referenced this issue Jun 2, 2018

Merge branch 'master' into localflags
* master:
  gui: Disable rescan button while scanning (fixes syncthing#4977) (syncthing#4979)
  test: Add another variant of API timeout to skip when benchmarking
  gui, man: Update docs & translations
  build: Use commit date as assets change date
  script: Use source data from environment when generating assets
  lib/scanner: Copy execute bits from previous version on Windows (fixes syncthing#4969) (syncthing#4970)
  lib/watchaggregator: Speedup propagation of removals (fixes syncthing#4953)  (syncthing#4955)
  gui: Update to Font Awesome v5 (syncthing#4889)
  lib/watchaggregator: Prevent race on config update (syncthing#4938)
  lib/db: Update global count when removing the previous global version (syncthing#4968)
  gui: Check if folder exists in folderLabel (fixes syncthing#4965) (syncthing#4966)
  lib/model: Move String method to folder (syncthing#4964)
  gui, man: Update docs & translations
  lib/model: Refactor override implementation into sendOnlyFolder
  lib/model: Refactor folderScanner into folder

@calmh calmh added this to the v0.14.49 milestone Jun 5, 2018

@calmh calmh changed the title Watcher delays edit with a delete by 60 seconds Speed up delete propagation when it's not a rename Jun 11, 2018

@calmh calmh added the enhancement label Jun 11, 2018

@dburkhardt

This comment has been minimized.

Copy link

dburkhardt commented Jun 12, 2018

Since the discussion on the forum is as old as this, and someone is looking at this, I'll comment here. I really dislike the delay in deleting files. I use syncthing to help me do data analysis on a remote server. A common workflow is as follows:

  1. Run a script on remote that creates a figure
  2. Wait for syncthing to sync that figure to local
  3. Inspect figure on local
  4. Realize I've made a mistake, edit the script, repeat from 1.

Now when I rerun, usually the new file has the same name as the old. So because of this delay in fs_watcher, the latency is seeing changes is 60 seconds. This delay is very annoying and I can't figure out a way to change it.

Inotify was almost instantaneous for this process, and I would really like to see this with fs_watcher.

@AudriusButkevicius

This comment has been minimized.

Copy link
Member

AudriusButkevicius commented Jun 12, 2018

Well that won't happen, as we'll have a different set of users who will be moaning that we don't handle renames.

@calmh

This comment has been minimized.

Copy link
Member

calmh commented Jun 12, 2018

Why are you complaining on a fixed issue? If the fix, which is in 0.14.49 (release candidate out today), does not fix your use case please open a corresponding new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.