Atomic saves in text editors cause havoc, unable to watch individual files

[nathany]

Reported by @robfig at howeyc/fsnotify#91. See also nathany/looper#14.

When editors use atomic saves they Create a new file and then do a Rename rather than just a straight Write (like when touching a file).

Related issues:

• @mstum wanted to watch a single configuration file (Is it possible to watch a single file? howeyc/fsnotify#94), and @IYF also had trouble watching a single file (why not don't support watch file? howeyc/fsnotify#37).
• If you Add a directory and then Remove a single file within the directory, the watcher will still report a Create due to atomic saves. https://code.google.com/p/go/issues/detail?id=7788

Dealing with text editors may be out of scope of fsnotify itself, but it could at least provide recommendations in the Wiki.

@robfig also reported some other oddities, though those may be specific to kqueue on OS X.

[mstum]

It seems that there are two major use cases to use a Watcher:

1. Watch very few individual files, e.g. config files for hot reload
2. Watch a folder, e.g., a "Queue" folder in which the user copies images/logs to be processed by the app and moved to an output folder

Number 1 suffers from said problem. I don't know specifics of OS implementations wrt efficiency, but it seems that setting up a "Delete" watcher which checks if the file is recreated and automatically sets up a new OS watcher for the new file would do the trick. To my knowledge, Move Operations are atomic in all decent file systems (I know they are in NTFS, I don't know or care if they are in FAT32/exFAT), but I don't know if a move can overwrite or if it's required to delete first and then move/rename. In that case, the watcher could be setup on the directory, equipped with a timer that checks every few ms and is automatically deleted if the file isn't coming back after let's say 1s. That works for small files, but let's say the file is a gigabyte in size, I don't know if that would work (then again, I don't know how often a hot-reload-watcher would be used on a large file)

I feel that a "persistent write timer" is more a high level utility construct: Allow me to specify how long the timer stays alive to re-check for file creation. In fact, I wonder if it's best to have fsnotify be a small layer of abstraction on top of the OS watcher - warts/gotchas and all - and then have a "Utility" for high level stuff. That way, the (what I consider) common use case of hot-reloading a tiny config file that's created within a second of a delete is trivial, but someone who wants something "weird" (like hot reloading gigabyte-sized files that are using copy-on-save) still has low level access.

[clipperhouse]

We might consider an abstraction over fsnotify proper to specify “desired” behavior.

One case might be to compare before & after on a given file, not unlike source control. Then emit a changed event, defined as the bytes having actually changed. This is more inline with what we expect with change events in JS in a browser, eg.

The CREATE/DELETE cycles that editors like Sublime generate might be mitigated by a throttle/debounce. Though at that point, it’s (logically) close to polling. And there is the possibility of races & inconsistency.

[nathany]

TextMate2 (OS X) use the exchangedata system call to perform the saves (not sure what that is). But Hugo users are having problems detecting saves: gohugoio/hugo#1053 (comment)

[arp242]

Closing this as a duplicate of #372, since it's essentially the same problem. That issue is much newer, but also more comprehensive.