Proposed changes to event translation

[dfaust]

I have collected a list of all events I found relevant for rsnotify on Linux, Windows and OS X. The list contains the native events and the rsnotify events they get translated to. As you can see some differ between platforms. So I propose some changes:

• On OS X ITEM_CHANGE_OWNER is fired when changing permissions, rsnotify should watch for it and report it as CHMOD (like on Linux)
• When a file is moved within a monitored directory, the inotify back-end currently reports a RENAME and a CREATE event. Windows and OS X report two RENAME events. Although that could be changed on Windows, it would be hard to do that on OS X. So I propose to change the CREATE event on linux to a RENAME event.
• It is currently not possible to properly detect the old and the new file name of a rename. Inotify provides a cookie and I think that cookie should be sent to the user. On Windows a rsnotify could create a cookie if two rename events are successive and in the same event group. There should be no confusion with only getting parts of rename events like on Linux because Windows reports FILE_ACTION_REMOVED when files are moved out of the monitored directory and FILE_ACTION_ADDED when they are moved in. So moving a file out and another in can't be mistaken as one rename. With FSEvent this is not as simple, more below.
• If a file is moved out of the monitored directory on Linux, currently a RENAME event is reported. This should be changed to a REMOVE event, like on Windows.
• Currently the Inotify back-end reports IGNORE events. These don't seem to fit the design goals of rsnotify and there are no comparable events for Windows and OS X. So I propose to remove them.
• Inotify and FSEvent can report problems via IN_Q_OVERFLOW and MUST_SCAN_SUBDIRS. rsnotify should forward them so the API user knows they have to rescan the watched directories. Edit: Windows can report a ERROR_NOTIFY_ENUM_DIR error when it's buffer overflows.

Remaining problems:

• FSEvent only emits ITEM_RENAMED whenever a file gets renamed without further information. It also lumps events together so using some smart logic to reconstruct what happened is hard (or impossible?). Eg. for a file that is created and then renamed two times FSEvent will emit 3 events (create | rename, rename, rename). In order to get the old and the new file name of a rename, it is possible to check if the event ids are consecutive. In the example above this would work on the second rename, but not on the first because the event ids are not consecutive. Simply using two successive rename events doesn't work either because a file could be moved out of the monitored directory and another one in. So this is something that should be documented and be left to the API user.
• Changing file attributes (like setting readonly) on Windows fires a FILE_ACTION_MODIFIED event. This cannot be distinguished from writing to a file. This should be documented.
• Renaming a watched directory creates a RENAME event on Linux and OS X, but not on Windows. This should be documented.
• Removing a watched directory creates a REMOVE event on Linux and OS X, but not on Windows. This should be documented. On windows an empty event is generated and if the monitored directory is moved to the trash, the monitor example gets into an infinite loop, reporting empty events. At least on my virtual machine.
• Linux and Windows monitor "inodes", OS X monitors "paths". So the behavior when renaming the watched file or directory differs. This should be documented.

event	platform	native event	current	proposed	comment
create	linux	IN_CREATE	create	create
	windows	FILE_ACTION_ADDED	create	create
	osx	ITEM_CREATED	create	create
remove	linux	IN_DELETE	remove	remove
	windows	FILE_ACTION_REMOVED	remove	remove
	osx	ITEM_REMOVED	remove	remove
write	linux	IN_MODIFY	write	write
	windows	FILE_ACTION_MODIFIED	write	write
	osx	ITEM_MODIFIED	write	write
chmod	linux	IN_ATTRIB	chmod	chmod
	windows	FILE_ACTION_MODIFIED	write	write	eg. readonly flag
	osx	ITEM_CHANGE_OWNER	-	chmod
xattr	linux	IN_ATTRIB	chmod	chmod
	windows	-	-	-
	osx	ITEM_XATTR_MOD	chmod	chmod
rename from	linux	IN_MOVED_FROM	rename	rename	has cookie
	windows	..._RENAMED_OLD_NAME	rename	rename	add cookie
	osx	ITEM_RENAMED	rename	rename	add cookie when possible
rename to	linux	IN_MOVED_TO	create	rename	has cookie
	windows	..._RENAMED_NEW_NAME	rename	rename	add cookie
	osx	ITEM_RENAMED	rename	rename	add cookie when possible
rename in	linux	IN_MOVED_TO	create	create
	windows	FILE_ACTION_ADDED	create	create
	osx	ITEM_RENAMED	rename	rename	must be handled by API user
rename out	linux	IN_MOVED_FROM	rename	remove
	windows	FILE_ACTION_REMOVED	remove	remove
	osx	ITEM_RENAMED	rename	rename	must be handled by API user
remove self directory	linux	IN_DELETE_SELF	remove	remove
	windows			?	infinite loop on move-to-trash
	osx	ITEM_REMOVED	remove	remove
remove self file	linux	IN_DELETE_SELF	remove	remove
	windows	FILE_ACTION_REMOVED	remove	remove
	osx	ITEM_REMOVED	remove	remove
rename self directory	linux	IN_MOVE_SELF	rename	rename	continues monitoring directory
	windows	-	-	-	continues monitoring directory
	osx	ITEM_RENAMED	rename	rename	stops monitoring directory, but continues monitoring path
rename self file	linux	IN_MOVE_SELF	rename	rename	continues monitoring file
	windows	..._RENAMED_OLD_NAME	rename	rename	stops monitoring file, but continues monitoring path
	osx	ITEM_RENAMED	rename	rename	stops monitoring file, but continues monitoring path
ignore	linux	IN_IGNORED	ignore	-
	windows	-	-	-
	osx	-	-	-
overflow	linux	IN_Q_OVERFLOW	-	overflow
	windows	ERROR_NOTIFY_ENUM_DIR	-	overflow
	osx	MUST_SCAN_SUBDIRS	-	overflow

Please let me know what you think.

[passcod]

• Agree with removing ignore.
• I'll have a look at other libraries to see how they handle these, for comparison.
• For renames, would it be possible to aggregate events so we provide a single event that describes the rename entirely, without needing cookies? Or would that be too hard/impossible?
• agree with osx chmod
• tentatively agree with the rest, but pending comparison to other libraries and review when I'm properly awake

[dfaust]

For renames, would it be possible to aggregate events so we provide a single event that describes the rename entirely, without needing cookies?

It would be possible and it was actually my first approach, but I ran into some troubles. This was before I wanted to change move_out events to delete though. So at this point I'm not sure what the better option would be. Either way handling renames on OS X will be a mess.

[passcod]

After a survey of other libraries (Go's fsnotify and notify, Ruby's listen, Facebook's Watchman daemon, Node's chokidar):

• The "consensus" seems to be that rename handling is hard. Some libraries just give up and delegate to the user. Some don't bother and just provide a "change" event. We should do the best we can, but not worry too much if that's not perfect.
• So: let's go with best-effort cookies as a first pass, then see afterwards, as an optional bonus, if it is feasible to have a sane "cookie-less" aggregated event. It looks fairly impossible on OS X.
• As far as I can tell, nobody emits ignore, nor overflow. Ignore being inotify only, let's remove it, but overflow is an important hint, so let's add it. I think the notify vision might be to (in the future) mitigate automagically in case of MUST_SCAN_SUBDIRS, for example.
• Nobody emits chmod events, but as it's fairly well supported, we can.
• What can we do about the infinite loop in Windows? Can we detect that, send a final remove event, and close up?
• For the "rename self file" event on windows, could we get the path the file was renamed to, stop watching the old one, and start watching this new one, to align with linux? Or should we do the reverse, and, on linux, when we get an IN_MOVE_SELF on a file, stop watching the inode and add a watch on the original path?
• As point of interest, how does the polling behaviour compare, when available, on those events?

If not mentioned or addressed, I agree with you on the rest.

[dfaust]

• I found out that Windows can report a ERROR_NOTIFY_ENUM_DIR error when it's buffer overflows. Unfortunately this doesn't seem to work with asynchronous usage of ReadDirectoryChangesW.
• I haven't done any testing on the infinite loop in Windows, yet.

For the "rename self file" event on windows, could we get the path the file was renamed to, stop watching the old one, and start watching this new one, to align with linux?

• That's seems like the best way to do it. Unfortunately this is very tricky to implement. The data structure that currently holds the path of the watched file gets cloned every time the start_read function is called, somehow this path needs to be overwritten though. So we would need some kind of shared memory or something.
• I haven't looked at the polling behavior, yet.

[passcod]

This has been released in 3.0.0! Finally got it all done and released and going.