FileSystemWatcher Deleted event not firing when moving multiple files outside the watched folder on macOS 10.13

[steveisok]

This was opened in Mono and bringing it upstream.

From @jesh182 on Tue, 10 Sep 2019 15:01:54 GMT

Steps to Reproduce

1. Add 3 files to the watched folder (The folder is "SomeFolder" inside user's home folder /Users/username/SomeFolder).
2. Move the 3 files out of the watched folder at the same time (Move to trash also do the same).

Expected Behavior

• Deleted event should be fired for each moved file.

Console:
Deleted FirstFile.jpg
Deleted SecondFile.jpg
Deleted ThirdFile.jpg

Actual Behavior

• Renamed event gets fired, with one file (random of the 3 files inside the folder) as OldName and another file (random of the 3 files inside the folder) as Name.
• Deleted event gets fired, with the last file (the one that was not included by the renamed event)

Console:
Renamed from: FirstFile.jpg to: SecondFile.jpg
Deleted ThirdFile.jpg

Environment

=== Visual Studio Community 2019 for Mac ===

Version 8.2.3 (build 16)
Installation UUID: 4dc24200-d747-41ce-9ca4-dece69112cf5
	GTK+ 2.24.23 (Raleigh theme)
	Xamarin.Mac 5.14.0.85 (d16-2-fix-mac-updates / 86973979)

	Package version: 600000319

=== Mono Framework MDK ===

Runtime:
	Mono 6.0.0.319 (2019-02/52203786470) (64-bit)
	Package version: 600000319

=== NuGet ===

Version: 5.1.0.6013

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Version: 2.1.12
SDK: /usr/local/share/dotnet/sdk/2.1.701/Sdks
SDK Version: 2.1.701
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/6.0.0/lib/mono/msbuild/Current/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.6.11
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Updater ===

Version: 11

=== Xamarin.Android ===

Not Installed

=== Microsoft Mobile OpenJDK ===

Java SDK: Not Found

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Android Device Manager ===

Version: 1.2.0.60
Hash: 686c817
Branch: remotes/origin/d16-2
Build date: 2019-08-07 15:49:58 UTC

=== Xamarin Designer ===

Version: 16.2.0.379
Hash: 3f0652957
Branch: remotes/origin/d16-2
Build date: 2019-08-01 17:32:50 UTC

=== Apple Developer Tools ===

Xcode 10.0 (14320.25)
Build 10A255

=== Xamarin.Mac ===

Version: 5.14.0.114 (Visual Studio Community)
Hash: c6691164
Branch: d16-2
Build date: 2019-07-29 15:54:06-0400

=== Xamarin.iOS ===

Xamarin.iOS not installed.
Can't find mtouch or the Version file at /Library/Frameworks/Xamarin.iOS.framework/Versions/Current.

=== Xamarin Inspector ===

Version: 1.4.3
Hash: db27525
Branch: 1.4-release
Build date: Mon, 09 Jul 2018 21:20:18 GMT
Client compatibility: 1

=== Build Information ===

Release ID: 802030016
Git revision: f438e4b3ac81dbc614da2f2d9ac3c70ea2e5c350
Build date: 2019-08-09 09:07:01+00
Build branch: release-8.2
Xamarin extensions: 67f026710e882e318fb0c90077b429be3dc0e688

=== Operating System ===

Mac OS X 10.13.6
Darwin 17.7.0 Darwin Kernel Version 17.7.0
    Sun Jun  2 20:31:42 PDT 2019
    root:xnu-4570.71.46~1/RELEASE_X86_64 x86_64

Build Logs

FSWatcherBuildLog.txt

Example Project (If Possible)

FSWatcherMac.zip

Copied from original issue mono/mono#16778

[wfurt]

I can reproducer it with 2.1 and 3.0. I will take look.

[aik-jahoda]

The same issue happens when copy files outside the watched folder inside it. In this case, it produces the same events instead of the WatcherChangeTypes.Created.

[aik-jahoda]

The investigation so far:

When a file is renamed the FSEventStreamCallback holds two events. Each event is kFSEventStreamEventFlagItemRenamed. The first contains original file name, the second contains a new file name.
When a file is moved outside the watched folder, only one kFSEventStreamEventFlagItemRenamed event is triggered as the second belongs to the different not watched folder.

The issue happens when we pair new and old names in case of file rename. When we remove 2 files outside the watched folder it triggers kFSEventStreamEventFlagItemRenamed events ad it is wrongly recognized as rename one file to another altought both files were moved outside the folder.

From current observation, those two cases differ only in event id. When a file is renamed, the event number is consecutive. When two files are moved, the event number has a gap.

I didn't find any related doc for macOS and I will spend more time to find if we are able to recognize the correct case.

[wfurt]

It seems like there is already some code trying to deal with pairing and note about rename to file out of watched path.

If testing strongly suggest that "normal" rename always comes together we can possibly disambiguate multiple renames and simply not provide final path. It seems like kernel does provide such info but I see posts about same struggle.

We may need kFSEventStreamCreateFlagWatchRoot according to https://developer.apple.com/library/archive/documentation/Darwin/Conceptual/FSEvents_ProgGuide/UsingtheFSEventsFramework/UsingtheFSEventsFramework.html
It seems like it should deliver events if file is renames outside of watched directory but I don't know if it does anything else.

[aik-jahoda]

The kFSEventStreamCreateFlagWatchRoot flag is already used: https://github.com/dotnet/corefx/blob/a50e056c5a62fb6dfe2e9475be47b93d33661700/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.OSX.cs#L124-L126

However, it doesn't deliver additional events.
Based on the File System Event Security it is not possible to receive events from folder user doesn't have access.

Pairing events based on event id is not documented as reported in apple radar.

The only reliable solution is test each move event with stat and based on the file presence decide if the file was moved in the watched directory or from/to the watched directory.