Back filesystem watchers with @atom/watcher

[smashwilson]

Add a new "experimental" setting for core.fileSystemWatcher that backs watchPath with @atom/watcher. There should be no changes for API consumers.

⚠️ There's a reason this is marked "experimental!" Our intention is to not merge this PR until I'm back from pending paternity leave, then let it percolate its way through master, beta, and stable before we consider flipping the feature flag on for everyone. In the meantime, if you want to see if this fixes a filesystem watching issue that you're having, you'll need to build Atom from this branch. ⚠️

Remaining Work

These are known issues in @atom/watcher that I'd like to address before this merges. Mostly, these avoid regressing on some edge cases that have been reported with the current watcher.

• Translate Windows short names to long names (Translate Windows short names to long names watcher#85)
• Handle symlinks (Handle symlinks watcher#19, Massive memory leak when opening projects with symlink loops #15966)
• Test for filesystem case sensitivity issues (Test for filesystem case sensitivity issues watcher#16)
• Test for unicode paths (Ensure that unicode paths work correctly watcher#12)
• Manual "just poll everything" setting
• @smashwilson back from paternity leave 👶
• Complete the testing matrix below 👇
• Tag and publish @atom/watcher 1.0.0

Features and Limitations

In its current state, the experimental watcher:

• Uses one or two threads for all watcher threads, rather than two threads per watcher.
• Fails gracefully to polling when (a) more than 451 watch roots are created on MacOS; (b) ReadDirectoryChangesW() reports that a directory does not support watching on Windows; or (c) the maximum number of inotify watch descriptors is exceeded on Linux.
• Can log a bunch of diagnostic information to track down edge cases.
• Scales to thousands of watch roots containing tens of thousands of files.
• Watch directories recursively or non-recursively.
• Watch individual files without polling.

It cannot currently:

• Follow symlinks.
• Explicitly include or exclude subpaths of a watched directory.
• Receive notifications on certain network shares.
• Report drive mount and unmount events.

Test Plan

Before we merge, I'd like to verify that we handle or fail gracefully in a number of trickier situations. In each scenario, if we:

• 🔴 crash in native code or lock up, I would consider that a blocker for merging this.
• ⚠️ throw a JavaScript stack or silently fail to deliver any events, I'd consider merging anyway, with an issue filed against @atom/watcher.

These are some of the cases I've thought to try:

• Open a project directory on a network drive.
  • MacOS: sshfs
  • MacOS: Samba
  • Windows: native Windows-to-Windows network share
  • Windows: Samba
  • Linux: sshfs
  • Linux: Samba
• Open a project directory from a flash drive.
  • MacOS
  • Windows
  • Linux
• Unmount a filesystem that contains an open project directory.
  • MacOS
  • Windows
  • Linux
• Mount a new filesystem beneath a watched directory.
  • MacOS
  • Linux
• Cloning a large git repository beneath a watched directory.
  • MacOS
  • Windows
  • Linux

To Opt In

If you want to give this a try before it merges, clone and build Atom from this branch:

git clone https://github.com/atom/atom.git --branch aw-watcher
cd atom
script/build --install

Launch Atom and set core.fileSystemWatcher to "experimental" in your core settings.

[smashwilson]

/cc @ungb I'd ❤️ your input on the test plan in the PR description once you're back in the office 😄 In particular, I have no idea what kinds of network drives we should test, or, like, how we should do that. NFS? Samba? sshfs... ?

/cc @rsese @lee-dohm @50Wliu in case any of you have seen any other filesystem-related issues fly by recently that we should verify this doesn't regress.

/cc @Arcanemagus who has been helping a bunch with the @atom/watcher work. 🙇 You're welcome to check this out in Atom proper, although unfortunately I don't think your NAS will work yet (I haven't done anything to get it out of the "silently deliver no events" category.)

[lee-dohm]

@smashwilson I feel like for network support we should test:

1. Native Windows-to-Windows network shares
2. sshfs
3. Samba

in that priority order. Also, we should test them through a connection with restricted network throughput and latency generator to see what the limits are for our implementation. (In other words, will sshfs over a high-latency 3G cellular connection work as expected?)

[50Wliu]

I'll try to run through flash drive and cloning a large git repository on Windows soon.

[50Wliu]

Got around to checking the scenarios I mentioned above and everything seems all right 👍. Decent performance, no crashes or anything.

[smashwilson]

Got around to checking the scenarios I mentioned above and everything seems all right 👍. Decent performance, no crashes or anything.

✨ Excellent, thank you! You're on Windows, right?

[smashwilson]

On reflection I'm likely going to punt on atom/watcher#70 until after this is merged. It isn't entirely clear to me what the right thing to do is there, and I don't want to hold this up when a better solution might not involve @atom/watcher at all. What's more, nsfw also doesn't handle this well, so we're no worse off than what's currently shipped.

I would like to tackle atom/watcher#19 though.

@ungb: Do we have a samba share or an sshfs mount already available somewhere that we could use for network drive tests? It might be nice to have those available for general testing in any case 🤔

[50Wliu]

You're on Windows, right?

Yep.

[smashwilson]

on Windows

... I can read, really

[ungb]

@ungb: Do we have a samba share or an sshfs mount already available somewhere that we could use for network drive tests? It might be nice to have those available for general testing in any case 🤔

@smashwilson we do have something for sshfs, let me look into samba share.

[smashwilson]

I saw a crash when you disconnect a Samba share on Windows while it contains a watched directory. Tracking it in atom/watcher#118.

[gothicserpent]

#16011 #15966

I just wanted to chime in by saying I recently added github functionality to my atom version, and the default filesystem watcher was constantly at around 5% when i was working on small tasks.

looking forward to this experimental file system watcher and i hope the cpu util will go down! thanks for the effort.