Chokidar uses a lot of memory and CPU for the rest of the execution when using 100k files

[ingles98]

Just like #849

Application performance is highly deteriorated for the whole execution and it seems to be quite exponential. 20k files no issues, 100k and constant 1gb RAM and 50% cpu when using polling. Without using polling, the initial scan takes over 5 minutes and after that it will continue to hang my application.

Any ideas ? Im having these issues on Windows 10.

[JamesS-M]

I'm seeing a similar issue.

[pahan35]

I also faced a similar problem.

In my case, I have a project with 27 000+ files in the watched dirs.

I have a laptop on Ubuntu 22.04 with an Intel Core i5-10210U CPU @ 1.60 GHz.

I faced this issue while using nodemon.

I investigated and noticed possible workarounds in conjunction with watched file extensions.

ATM chokidar watches all the files inside the watched folders, and only later they are filtered by nodemon. The code is here https://github.com/remy/nodemon/blob/main/lib/monitor/watch.js#L170-L174

In my specific case, when all the files were watched, it was 27 000+ files and 50% of processor loading.

However, when I made some internal changes and ignored all the extensions which aren't watched inside the chokidar itself, it started to watch only 17 000+ files, and the processor loading was 35%, which is a 30% reduction.

Unfortunately, I didn't manage to ignore files by extension when providing a Regex like /.+(?<!(ts|tsx|js|json))\/?$/ into the ignored param.

However, I managed to ignore files by their extension, adding some extra code inside the chokidar itself.

I'll prepare a PR to demonstrate how others can achieve the same result.

Potentially, applying to watch only by the specific file extensions should be possible.

[ingles98]

I also faced a similar problem.

In my case, I have a project with 27 000+ files in the watched dirs.

I have a laptop on Ubuntu 22.04 with an Intel Core i5-10210U CPU @ 1.60 GHz.

I faced this issue while using nodemon.

I investigated and noticed possible workarounds in conjunction with watched file extensions.

ATM chokidar watches all the files inside the watched folders, and only later they are filtered by nodemon. The code is here https://github.com/remy/nodemon/blob/main/lib/monitor/watch.js#L170-L174

In my specific case, when all the files were watched, it was 27 000+ files and 50% of processor loading.

However, when I made some internal changes and ignored all the extensions which aren't watched inside the chokidar itself, it started to watch only 17 000+ files, and the processor loading was 35%, which is a 30% reduction.

Unfortunately, I didn't manage to ignore files by extension when providing a Regex like /.+(?<!(ts|tsx|js|json))\/?$/ into the ignored param.

However, I managed to ignore files by their extension, adding some extra code inside the chokidar itself.

I'll prepare a PR to demonstrate how others can achieve the same result.

Potentially, applying to watch only by the specific file extensions should be possible.

I ended up using platform specific bindings for each platform I supported since I could not find a good solution with chokidar. Turns out the software I was making had way more needs for thousands of watched files across networks than anticipated. Ended up working like a charm, meanwhile I left the company. How the time passes without a single update on this issue

[ingles98]

I ended up using the same approach used by Microsoft's visual studio code for Windows, I think they still use chokidar for other platforms, at least that was the case 2 years ago

[paulmillr]

How the time passes without a single update on this issue

Yeah, why didn't you help to resolve this issue?

[ingles98]

How the time passes without a single update on this issue

Yeah, why didn't you help to resolve this issue?

Not everything is an offense on the internet mate. What I meant is how so much have changed since I opened this issue and yet not much on this, surprised that it is still an issue, 100k files isn't that many files but i guess it works for the most that intend on running on Windows. I figured it out back then and started attempting to append the changes into chokidar, however it would change quite a bit while at the same time not working well (or not at all) with the pattern matching for the filePath. There were some issues on linux as well with performance that started showing up which were fixed the same way I did with Windows, therefore I simply forgot about it.

[pahan35]

I opened the PR with PoC for ignoring files by extension.

@paulmillr, please let me know whether I need to work on it more and if is there a chance to get it merged when everything is done.