Watch directories for modification

[fheckl]

It may be desirable to have QS watch file system directories for modification to trigger a catalog update for that directory instead of forcing the user to invoke a rescan.
It was among the things I missed most prominently when switching from LaunchBar. E.g. when you install a new application and then want to do something with that application you now have to wait until QS has picked it up into its catalog after some time.

[tiennou]

This is actually already implemented (through QSVoyeur), but there's no interface for setting a catalog entry to be watched in QSFileSystemObjectSource.

[HenningJ]

I the future I'd like have that enabled for all file system catalog entries by default.

[daniels220]

Is there any chance of getting this implemented? It'd be a really nice addition to QS. I'd be happy to test if it could be added as a hidden feature into one of the next few versions of QS.

[skurfer]

It's already in there and enabled for a small number of presets including ~, ~/Documents, ~/Downloads, /Applications/, and a couple of others. You can test this by adding a new file to your home directory and searching for it immediately.

We just haven't added a check-box for it. You can enable it for a custom entry by adding a boolean named "watchTarget" with a value of YES under the "settings" for that entry in Catalog.plist.

It should be used sparingly, though. Only where frequency of changes * number of files = a reasonable number.

[daniels220]

Neat, I'll try it.

Does it scan only the folder that actually changed? I assume it's using FSEvents—if I have ~/Documents/A/Deep/Tree/Of/Folders and I change a file in the deepest one, can it only scan that one folder? If so, I don't have much in my catalog that would change very often, so it doesn't seem like it would ever be an issue. Maybe on an older machine, but I have an SSD—it's totally fine with me if every write operation has a QS-needs-to-scan-now overhead, because I won't even notice, and I will notice if QS can't find a file.

...Actually it looks like it just doesn't scan subfolders at all. touch ~/Documents/TestFolder/ATestFile, invoke, 'atestfile' finds nothing. Could that be changed? I've very carefully managed my Documents folder so I can set the catalog entry for it to infinite depth without breaking QS, and I really want instant access to it if at all possible. I also have some of my applications organized into folders.

[pjrobertson]

You're probably right, all subpaths should be added for scanning.
Something else to add to this issue as a list of things to do.

If you're interested, QS uses https://github.com/bdkjones/VDKQueue to
provide all it's notifications on updates to folders etc.

On 23 August 2012 20:55, daniels220 notifications@github.com wrote:

Neat, I'll try it.

Does it scan only the folder that actually changed? I assume it's using
FSEvents—if I have ~/Documents/A/Deep/Tree/Of/Folders and I change a file
in the deepest one, can it only scan that one folder? If so, I don't have
much in my catalog that would change very often, so it doesn't seem like it
would ever be an issue. Maybe on an older machine, but I have an SSD—it's
totally fine with me if every write operation has a QS-needs-to-scan-now
overhead, because I won't even notice, and I will notice if QS can't
find a file.

...Actually it looks like it just doesn't scan subfolders at all. touch
~/Documents/TestFolder/ATestFile, invoke, 'atestfile' finds nothing.
Could that be changed? I've very carefully managed my Documents folder so I
can set the catalog entry for it to infinite depth without breaking QS, and
I really want instant access to it if at all possible. I also have some of
my applications organized into folders.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/330#issuecomment-7981971.

[daniels220]

So one of these days, I have to learn what FSEvents actually is, what the old /dev/fsevents is, and what else exists. (If someone has written a high-level overview that you know of and could link to I'd be interested.)

I was under the impression that pre-10.4 there was no way to get notified of filesystem changes, period—otherwise why was /dev/fsevents created? And that in 10.5+ that is deprecated in favor of the FSEvents API proper. But it seems like there's this whole other kqueue system, which honestly seems much better than FSEvents anyway.

Actually at that point, is there any reason QS needs to "scan" at all? There's at least one API that provides per-file change notifications, such that QS could just add/remove files based on the events without scanning. The only way to do that might be the unsupported read-from-/dev/fsevents way, though, but if kqueues support it that would solve the performance problem once and for all.

One of these days I'll try to learn Obj-C... Until then, thanks again for all you do.

[pjrobertson]

Actually at that point, is there any reason QS needs to "scan" at all?
There's at least one API that provides per-file change notifications, such
that QS could just add/remove files based on the events without scanning.
The only way to do that might be the unsupported read-from-/dev/fsevents
way, though, but if kqueues support it that would solve the performance
problem once and for all.

A very interesting concept. If it doesn't come with a performance penalty
(most likely why the 'watch directories' option is disabled for most
catalog entries) that that could be the way to go.

One of these days I'll try to learn Obj-C... Until then, thanks again for
all you do.

We look forward to having you on board ;-)

Thanks!

On 23 August 2012 21:15, daniels220 notifications@github.com wrote:

So one of these days, I have to learn what FSEvents actually is, what the
old /dev/fsevents is, and what else exists. (If someone has written a
high-level overview that you know of and could link to I'd be interested.)

I was under the impression that pre-10.4 there was no way to get notified
of filesystem changes, period—otherwise why was /dev/fsevents created? And
that in 10.5+ that is deprecated in favor of the FSEvents API proper. But
it seems like there's this whole other kqueue system, which honestly seems
much better than FSEvents anyway.

Actually at that point, is there any reason QS needs to "scan" at all?
There's at least one API that provides per-file change notifications, such
that QS could just add/remove files based on the events without scanning.
The only way to do that might be the unsupported read-from-/dev/fsevents
way, though, but if kqueues support it that would solve the performance
problem once and for all.

One of these days I'll try to learn Obj-C... Until then, thanks again for
all you do.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/330#issuecomment-7982568.

[skurfer]

There's at least one API that provides per-file change notifications, such that QS could just add/remove files based on the events without scanning.

I think this will be more complicated than that. For starters, all the catalog entry contains is the path to a parent directory. So there'd have to be at least an initial "scan" of some sort to get a list of individual paths to watch. But then, what happens when a new file shows up in the directory? You weren't watching for its path, so how would you ever know about it?

Maybe it could just watch the parent directories and all directories inside it (instead of files). Then, if one of those directories is changed, rescan its contents. That would be quite a bit more efficient than what we have now, since

1. It would only need to rescan the directory that changed instead of everything from the parent folder down.
2. It would only need to scan one level deep (since every sub-directory is being watched individually).

The only way to do that might be the unsupported read-from-/dev/fsevents way

I'm pretty sure there's a supported API for monitoring filesystem changes. I think the only difference is that they don't promise to show you everything. For instance, if the system is too busy or your app isn't consuming the messages fast enough, it can throttle them back.

[daniels220]

I meant an API that watches a directory, but notifies you separately for every file that changes. KQueues do this, but seem to only be able to watch one directory level at once, which is not ideal (although it certainly could be worked around).

The only other supported way to do this AFAIK is FSEvents, which watches an entire tree at once, but will send events only for a particular subfolder if only that folder has changed. The problem is, it just says "something in this directory changed", not "this file changed". Basically it's a way to do exactly what you suggested—but I don't like that way for live apps like Quicksilver (FSEvents was designed for Time Machine and it shows). Ideally you'd have something that tells you every file that changes so you don't have to scan.

On an SSD, the scanning overhead doesn't matter, but it seems like it ought to be more performant to do something like:

1. At startup, do a full scan. Remember all the directories you found.
2. Put a KQueue on each one so you know exactly which files change and don't have to scan.
3. If you get an event indicating a change to an entire directory...to be safe, you might have to rescan it once you've removed the old (now-broken) KQueue and added a new one, just in case an event slipped by.

It occurs to me that if someone unpacks a big tarball, or does a git clone or something, QS miiiight have some trouble with this. The advantage of FSEvents is that it has a latency—so where unpacking a tarball with KQueues gives you 1 event per file/dir, and the overhead of adding a new KQueue for each dir, FSEvents will just wait and at the end say, "directory {tarball output} created, need to scan subdirs".

On the other hand (and I have personal experience with this, having done it with a piece of software that used FSEvents for live scanning which it is not good at), if someone adds ~/Library/Preferences to their catalog, com.apple.Finder.plist changes, oh, every 5 seconds—and that folder has a lot of files in it. If you're scanning the whole thing every 5 seconds instead of knowing exactly which file changed, the machine grinds to a halt.

---

I don't know Objective-C, I swear! :/ Filesystem scanning/notification is a bit of a pet peeve of mine, though...

[pjrobertson]

Adding an option in the catalog prefs to 'watch this folder for modifications' I think would be a great feature to add, and shouldn't be too much code.

[tiennou]

IIRC, we need to wrap some of the lower-level fs primitives with something that better fits the bill (as per the points made by @daniels220 above). For example, having a "watch" checkbox and a "depth" that don't care about each other feels to me like good UX.

Aaand I have something about that too ;-).