Performance on Windows is unacceptable for large folders

[bpasero]

I am using chokidar 1.0.0-rc3 watching my Desktop that has 40.000 files, 5765 folders and about 2.6 GB in total. The code is as follows:

var chokidar = require("chokidar");

console.log("before watch");

var watcher = chokidar.watch("C:\\Users\\benjpas\\Desktop", { 
    ignoreInitial: true,
    persistent: true 
});

console.log("after watch");

watcher.on("all", function(type, path) {
    console.log(type, path);
});

watcher.on("error", function(error) {
    console.log("error handler ", error);
});

process.on('uncaughtException', function(error) {
    console.log("uncaughtException: ", error);
});

Watching my process explorer after starting, I see the node.exe process quickly consuming > 400 MB of memory while keeping a steady high CPU load for a long time. I also notice that while chokidar is busy, file events get lost (that is, after startup I was quickly doing a change in some deep nested folder and this event was not showing up in the handlers).

Video of the process going crazy below:

How can we nail this down further? Are you guys using some tools for profiling? Is there something I can try different to see if it has any impact?

[es128]

This is most likely all from the readdirp initial scan, not the watching itself. You should be waiting for the ready event before you expect change events to be detected properly.

ignoreInitial does not prevent the need to walk the file tree ahead of time in order to get accurate events.

[es128]

Same situation in fsevents mode btw

[bpasero]

I see. How come memory goes up so much, you shouldnt be storing more than the list of files per folder, right?

[es128]

That's the consumption of the readdir and stat calls. It should get garbage collected when readdirp is done, although I don't know if/when you'll see it released back to the system in your profiler.

[es128]

Anyway, I'm closing this as it's based upon a false comparison (between chokidar and fs.watch with the recursive flag in the limited situations where that's available).

The per directory fs.watch listeners that chokidar opens are still quite performant. I haven't seen anything so far that makes me think it's worth upheaving the code to take advantage of the recursive flag here.

[bpasero]

@es128 disagree, while my motivation for adding the recursive flag was due to this bug, I still cannot move to another solution (away from chokidar).

In other words, I think this bug is totally valid. Chokidar is nice that it gives me consistent events on all OS, so it should also play nice with large folders if possible.

Could we come back to discussing this issue and understanding the high resource usage? All the objects readdirp() allocates should be very short living. And even if, how come 7000 stat elements end up generating 400 MB of memory? Do you suggest to file a bug against readdirp()?

[es128]

Go ahead. But it isn't a bug. You can't just wish away the resource requirements of necessary operations. Right at the top of this thread you described the expansiveness of the directory being watched. Chokidar needs to know the structure of the file tree to provide accurate events. Obviously, fs.watch's lack of that perspective is why it can't emit events as cleanly.

Perhaps it's worth exploring whether readdirp can add an option for a lighter mode that does not stat for this sort of very atypical use-case.

[bpasero]

I tried the same folder with readdirp() and get up to 150 MB of memory used. I get to 600 MB with chokidar. The memory does not drop back down after I get the ready event.

Printing process.memory: { rss: 637415424, heapTotal: 346279280, heapUsed: 282252376 }