can't watch entire C drive using node 0.11 on windows

[reverofevil]

When running the following example on Node 0.11.15

var chokidar = require('chokidar');

var watcher = chokidar.watch('C:\\', {
    persistent: true, ignorePermissionErrors: true
});

var log = console.log.bind(console);

watcher
    .on('add', function(path) { log('File', path, 'has been added'); })
    .on('addDir', function(path) { log('Directory', path, 'has been added'); })
    .on('change', function(path) { log('File', path, 'has been changed'); })
    .on('unlink', function(path) { log('File', path, 'has been removed'); })
    .on('unlinkDir', function(path) { log('Directory', path, 'has been removed'); })
    .on('error', function(error) { log('Error happened', error); })
    .on('ready', function() { log('Initial scan complete. Ready for changes.'); })
    .on('raw', function(event, path, details) { log('Raw event info:', event, path, details); })

watcher.on('change', function(path, stats) {
    if (stats) console.log('File', path, 'changed size to', stats.size);
});

we get something like this in the output

Directory C:\ has been added
Directory C:\ has been removed
Directory C:\$Recycle.Bin has been added
Directory C:\Program Files has been added
Directory C:\ProgramData has been added
Directory C:\Users has been added
Directory C:\Windows has been added
Directory C:\Recovery has been added
Error happened { [Error: watch EINVAL] code: 'EINVAL', errno: 'EINVAL', syscall: 'watch' }
File C:\IO.SYS has been added
Error happened { [Error: watch EINVAL] code: 'EINVAL', errno: 'EINVAL', syscall: 'watch' }
File C:\MSDOS.SYS has been added
Directory C:\Documents and Settings has been added

and then the program exits.

It seems that this is an issue with Node itself, but I'm not proficient to report it. Anyway, it seems that chokidar is a library that was meant to work a bit more stable that default fs.watch, so you may be interested.

[es128]

lol why are you trying to watch the entire C drive?

chokidar works fine with node 0.11 on windows: https://ci.appveyor.com/project/es128/chokidar/build/14/job/qnhg9ij0jykxptce

[reverofevil]

I'm watching an entire drive because

1. I have a file deduplicator;
2. There's an amount of duplicates on my drive;
3. I need something like a month to deal with the results of deduplicator;
4. And I'm not eager to rescan 7 TB of drives every day, because that takes hours.

Anyway, there's nothing in common among the issue and my plans, so I think the title was changed inappropriately.

That test doesn't show anything, because

1. I still get EINVAL error;
2. chokidar fails shortly after that;
3. When I switch to Node 0.10 everything works fine.

[paulmillr]

why 0.11.15 and not 0.12.0?

[es128]

I think the title was changed inappropriately

No it wasn't. You described problems with a very specific outlier use-case. The title you gave the issue is very misleading and generally false for typical usage.

When I switch to Node 0.10 everything works fine.

Really? That's amazing. @bpasero you may interested to hear about this use case.

So have you tried node 0.12 or io.js?

[bpasero]

Hm, I dont think I have seen this specific error under node 0.12. I actually did some testing today for #228 comparing node 0.10 and 0.12 in terms of performance and didnt hit issues. But the folders I tested were also all under my home directory with proper permissions to access.

[reverofevil]

No it wasn't. You described problems with a very specific outlier use-case.

I don't think there's something like outlier use case when one is talking about generic library. I'd prefer to call it a missing test.

So have you tried node 0.12

Error happened { [Error: watch EINVAL] code: 'EINVAL', errno: 'EINVAL', syscall: 'watch' }
File C:\IO.SYS has been added
Error happened { [Error: watch EINVAL] code: 'EINVAL', errno: 'EINVAL', syscall: 'watch' }

After listing several entries from the root of the drive, chokidar stops.

or io.js?

Works relatively fine.

[es128]

What you're doing is very unusual. You should feel proud of it! 😀

I'll give it a try from a VM and see if I can get any insight into what's going on. It is definitely an issue in node. Chokidar might be able to do something to correct for it, but if it can't, it can't - and this will have to be taken up with the node project.

In the meantime, I recommend you stick to node 0.10 or switch to io.js.

[reverofevil]

@es128

The approach with recursive fs.watch taken in most watch-libraries doesn't quite fit my needs yet, because it takes several minutes and consumes almost all system memory. On the other hand inotify-win (C#) can watch an entire drive immediately, so I'm currently trying to connect it to Node with edge. (BTW just have seen @bpasero in issues list there. Same idea, probably?)

Anyway, chokidar helped me a lot previously and I'd be happy to see this issue fixed. Thanks for great work!

[bpasero]

@polkovnikov-ph I am with you (see #228). While I think scanning C: is a bit crazy, I see bad performance from watching large folders in general with chokidar.

As you say, it all boils down in having to scan the entire folder recursively first because of the unacceptable implementation of fs.watch() in libuv. I think trying to solve this on the JS side of things will never really work. The right approach is to fix the mess in libuv. I added pull requests to enable recursive watching natively on windows (libuv/libuv#198) and a workaround for missing filenames during delete (libuv/libuv#199).

Still, this is just the tip of the iceberg. Imho there are more bugs, and the fact that chokidar exists is a proof that lots of manual crafting is needed to get all events properly.

Btw I have experience using node.js with edge and leveraging the C# file watcher class (https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher%28v=vs.110%29.aspx). It works nicely, consumes no memory and is instant. Since it is implemented on top of the same primitives libuv uses, libuv could do the same.

[reverofevil]

Was it fixed?

[es128]

No, but the issue is outdated. If you can reproduce it with the latest versions of chokidar and node and still want attention paid to it, we could consider reopening.

I think the main issue was permissions or special files at the root of the primary drive that would have to be stepped around (you could probably work around this now with some deliberate ignore settings). Aside from that, if you can tolerate the time it takes to get started and the overall system load there shouldn't be anything preventing it from working. But this remains a fairly extreme use-case for chokidar.

[reverofevil]

I will try this again on a weekend. I've finally went on implementing native C++ plugin over system APIs and boost.filesystem library then. It turned out even boost had that same nasty bug when enumerating contents of a directory that contains files with special access rights. I don't think there is much chokidar can do about that without resorting to reimplementing fs.watch from scratch.

[es128]

Right, it just depends. We could maybe find ways to skip over problematic entries seamlessly, but there may be barriers outside our control.

You may also be interested in the watchman project, which now has a Windows implementation in alpha: https://facebook.github.io/watchman/