Use fanotify on Linux?

[KyleAMathews]

Curious if y'all have thought about using https://man7.org/linux/man-pages/man7/fanotify.7.html to watch on Linux? It's much more performant than inotify. You can monitor all changes on a fs mount — which I assume that + filtering out any changes not related to the directory of interest would be pretty performant.

[devongovett]

Looks nice. I believe when I started this project it wasn't widely available yet. If we can detect support for it at runtime, I think it's worth having with a fallback to inotify. I imagine some distros don't have it yet.

[mischnic]

I think we'd have to determine the kernel version itself:

The fanotify API was introduced in version 2.6.36 of the Linux
kernel and enabled in version 2.6.37. Fdinfo support was added
in version 3.8.

In the original fanotify API, only a limited set of events was supported. In particular, there
was no support for create, delete, and move events. The support
for those events was added in Linux 5.1

The two fanotify_* functions exist before 5.1 as well, so there shouldn't be problems regarding missing functions at load time even if we would then determine it "unsupported" for our usecase.

(Ubuntu 18.04 LTS uses the 4.15 kernel and is supported until April 2023. All newer versions are on 5.1+.)

[benmccann]

There are a number of details in the fsnotify for Go repo beginning here: fsnotify/fsnotify#114 (comment)

[benmccann]

I believe there are a few possible ways we could call fanotify to recursively watch a directory:

• manually crawl the FS and place a watch on each sub-directory as is done with inotify today
  • works in Docker without issue
  • mirrors existing implementation
  • more complicated and more overhead than other strategies
• use FAN_MARK_MOUNT
  • currently requires CAP_SYS_ADMIN to be granted to Docker (discouraged)
  • requires kernel 6.8 if Docker is run on top of btrfs
• use FAN_MARK_FS
  • best performance
  • currently requires CAP_SYS_ADMIN to be granted to Docker (discouraged)
  • doesn't work in Docker on top of btrfs

All have better support for renames than inotify and higher default limits than inotify, so even the first bullet has advantages when compared to the existing inotify watcher.

Moving this discussion over from fsnotify/fsnotify#114, @amir73il is my summary above correct?

Maybe the nicest thing to do would be to implement these as different backends such as fanotify-crawl and fanotify-fs. @parcel/watcher could then potentially use fanotify-fs if running outside Docker and fanotify-crawl if running inside Docker. The user may manually choose a different backend. This will also allow manually testing out different backends for providing feedback, etc. E.g. we could begin with all off and do a bit of manual testing at first

Also, it would be nice to have an option of a move event so that users can tell if files were moved vs simply created or deleted. I'm not sure if users could rely on this in a generic cross-platform way, but it could be an option to enable to benefit users who are running their application on only a single platform.

What do you think @devongovett ?

[amir73il]

unfortunately, FAN_MARK_MOUNT does not support watching the events that you are interested in (create, delete, rename)

I recommend to watch FAN_RENAME event and not FAN_MOVED_TO/FROM because it is not easy to pair them.
They don't even have the cookie as in inotify FAN_MOVED_FROM/TO events

[benmccann]

Thanks as always for the feedback @amir73il! If we're able to implement fanotify support in this library it will end up being used by lots of popular software like VSCode who consume this library.

@devongovett if you're okay with the plan I laid out above, I will I'll find someone to work on this and sponsor them to send a PR since I know Node, but not C++. And to be clear about my motivations, if I can implement fanotify-crawl with move support, then I will switch Immich from chokidar to @parcel/watcher. This is unrelated to my work on Vite though, so no promises there

[devongovett]

I haven't had a chance to look into fanotify much yet, but seems reasonable to me. Based on your summary above, it seems like fanotify is still not recursive unless you watch the entire filesystem and filter events yourself (which requires special privileges)? Given that, what are the benefits over inotify, which has the same limitation requiring us to crawl the filesystem? I guess you mentioned higher limits by default. Anything else?

I guess the FAN_MARK_FILESYSTEM option would give more benefits, but requiring special permissions might limit its usability. We could probably detect that and use it if available, but not sure how often that really would be available. If not common, is it even worth implementing?

[benmccann]

it seems like fanotify is still not recursive unless you watch the entire filesystem and filter events yourself (which requires special privileges)?

Yes, that's my understanding

Given that, what are the benefits over inotify, which has the same limitation requiring us to crawl the filesystem? I guess you mentioned higher limits by default. Anything else?

Yeah, the two main ones would be the higher default limits (~4x higher on my machine as shown below) and support for reporting file moves. inotify can sort of report file moves, but per their docs it is "inherently racy". fanotify on the otherhand actual has direct support for file moves, which is the killer feature I need. (If a photo on disk is moved, I want to update the location in the Immich index. I can't reliably do this with inotify or trying to match up create/delete events)

$ cat /proc/sys/fs/inotify/max_user_watches
65536
$ cat /proc/sys/fs/fanotify/max_user_marks 
263430

I guess the FAN_MARK_FILESYSTEM option would give more benefits, but requiring special permissions might limit its usability. We could probably detect that and use it if available, but not sure how often that really would be available. If not common, is it even worth implementing?

I had originally thought you'd only need the special permission if you're running inside Docker, which wouldn't be as big of a limitation, but I just looked a bit more and think I may have been mistaken on that. So yeah, it might not be worth implementing. In either case, it's not what I would be interested in implementing because Immich is nearly always deployed within Docker and so I know I can't use it in either case.

[devongovett]

fanotify on the otherhand actual has direct support for file moves, which is the killer feature I need. (If a photo on disk is moved, I want to update the location in the Immich index. I can't reliably do this with inotify or trying to match up create/delete events)

This raises another point: currently renames are reported by @parcel/watcher as two separate events, a delete followed by a create. This is for consistency across platforms, some of which don't have a reliable way to detect renames. I'm not sure if we'd want to support rename events only on linux. If we want to support renames, I'd want to revisit this across all platforms to see if it is possible to support consistently.

[benmccann]

My thought was to make it an option. Continue to report a delete followed by a create by default for cross-platform compatibility, but enable users to opt-in to receive only a single move event on Linux. Many applications are known to run only on Linux because a company runs them on their servers or cloud or because they're deployed in a container. This is a critical functionality for some of those applications (such as Immich). This can't easily be added to most other Linux watching libraries, since it requires a native component to utilize fanotify, so could be a real driver to use @parcel/watcher

You can report file moves on MacOS and Windows by comparing the inode ID of files in create and delete events, but this will only allow you to match up create and delete events within some specified timeout window. fabiospampinato/watcher does this, but it feels potentially unreliable to me. It'd probably be more reliable in a library like this one where you have a separate thread handling the events, but I don't think you could have a 100% guarantee unless you're running on Linux, so any option should probably either remain Linux-only or at least explicitly document the caveats

[devongovett]

Making it opt-in sounds reasonable to me. 👍

[raxyte]

I'd like to take a jab at this (saw the listing from @benmccann), could someone tell me what are the C++ standards available to be used here (or the toolchains you use) and any tips on locally testing the changes. (The documentation leaves something to be desired 😅)