Windows version has critical performance issue when unwatched files change

[mcclure]

Describe the bug

I am developing with Rust. As such I am using the rust-analyzer LSP server inside of the Sublime Text editor. Sublime's LSP-rust-analyzer plugin makes use of LSP-file-watcher-chokidar, which uses (naturally) chokidar. I dual-boot Windows and Ubuntu and have this same stack in place on both OSes.

On the Windows install only, I see a behavior where during builds, performed outside of Sublime, the Chokidar process goes berzerk and uses very large amounts of CPU time.

Because of the process separation here I can isolate the CPU load to Chokidar: The LSP-rust-analyzer/LSP-file-watcher-chokidar plugins are running in the Sublime Text process, the rust-analyzer LSP and its helpers are running in their own processes, and the node.exe running Chokidar is a process by itself.

This CPU load is a major impediment to development. Because it very precisely runs when I build, it slows down both the build itself and (if I did cargo run instead of cargo build) also the first few seconds of the application running— it happens basically at the worst possible time. At times the one-two punch of a build happening simultaneously with Chokidar going berzerk has been so bad it interferes with the responsiveness of my GUI, music I am streaming etc.

Because this is a build¹, I strongly believe that nothing is changing except for the target (or for web builds dist or pkg) subdirectory. Chokidar has been specifically told by LSP-file-watcher-chokidar not to observe these folders, so my expected behavior is that Chokidar should do nothing when the contents of these directories change and my observed behavior is that it is not only doing something, whatever it is doing is intense enough to make the computer unusable.

I have a loose theory why this is happening and a proposed fix (see below).

Versions:

• Chokidar version: 3.5.1
• Node version: v18.16.1 (current LTS)
• OS version: Windows 10 (Version 10.0.19045 Build 19045)
• Sublime Text version: 4 (Build 4143). Plugins:
  • LSP v1.24
  • LSP-rust-analyzer v1.5.0
  • LSP-file-watcher-chokidar v1.0.2

Notes:

I was running with Node v14.17.3 during most of the period I saw this problem, but while filing this bug I upgraded my Node to v18.16.1 and verified that the Chokidar behavior does not change.

I know the "node" in Task Manager (as in the screenshot above) is running Chokidar, and I know it's system Node, and I know what Chokidar it is running because I can look in Process Explorer and see the invocation is C:\Program Files\nodejs\node.exe C:\Users\Andi\AppData\Local\Sublime Text 3\Package Storage\LSP-file-watcher-chokidar\chokidar\chokidar-cli\index.js". If I look in that "chokidar" folder I find a package.json containing:

"dependencies": {
  "chokidar": "^3.5.1",
  "lodash.debounce": "^4.0.8",
  "lodash.throttle": "^4.1.1"
},

I can get you the various version numbers for Ubuntu (where the problem is not occurring) but I doubt this is important.

Is this Chokidar's bug?

As you can see from the version list above, the stack here is deep.

I believe it is Chokidar that is doing the wrong thing (IE, I believe Chokidar is being told to do the right thing before doing the wrong thing) because before filing here I first filed a bug on the LSP-rust-analyzer plugin, because it seemed possible that my "ignore these directories" directives were not being properly passed through to Chokidar. They showed me a slightly different way to tweak my Sublime configuration which did improve the performance of Sublime but had no noticeable effect on the behavior of the Chokidar subprocess. At the end of that thread, they are able to assure me that my configuration² is correct and show me the specific place in the Sublime LSP system where they claim my Sublime preferences get translated into a Chokidar "ignores" directive.

Since I am running a reasonable, default³ configuration of common software (Sublime, Rust) and that software appears to be using Chokidar in the ordinary way, it seems the bug here does fall on Chokidar. If you can show me that something higher up the stack (such as LSP-file-watcher-chokidar) is using Chokidar incorrectly or is running Chokidar in an unreasonable environment (something has an incorrect version, etc) then I will happily go back to the Sublime/LSP packages and show them they're doing the wrong thing. But if that is the case I lack the background to figure it out on my own…

To Reproduce:

Because unfortunately I am not using Chokidar directly, I can't give you a code sample. However I can give you the steps to reproduce my setup exactly:

• Install Sublime 4 on Windows 10. The unpaid trial version will work fine.

• Install Package Control:

  • In Sublime Text 4, type CTRL-SHIFT-P. Type "Install p" and click the "Install package control" autocomplete.

• Install plugins:

  • In Sublime Text 4, type CTRL-SHIFT-P. Type "Package in" and click the "Package Control: Install Package" autocomplete.
  • Type "ls" and install the "LSP" package.
  • Repeat (CTRL-SHIFT-P, "Package Control: Install Package", "ls ru") and install the LSP-rust-analyzer package.

• Configure Sublime

  • In Sublime Text 4, go to the "Preferences" menu and select "Settings". Between the {} enter

          "binary_file_patterns": ["*.jpg", "*.jpeg", "*.png", "*.gif", "*.ttf", "*.tga", "*.dds", "*.ico", "*.eot", "*.pdf", "*.swf", "*.jar", "*.zip", "target/*", "pkg/*", "node_modules/*"],
          "index_exclude_patterns": ["target/*", "pkg/*", "dist/*", "node_modules/*"],
    

    To maximize similarity with my system, I suggest also adding

    "show_git_status": false,

    But, I don't think this will actually do anything.

  • File->Quit Sublime Text and reopen it.

• Install Rust+Cargo. If this step is onerous, I think LSP-rust-analyzer might install its own separate copy of Rust and you might be able to get it to build. But I don't know how to do that offhand.

• Check out this git repo https://github.com/mcclure/zap

  Note: I say this specific repo because the Chokidar behavior is substantially worse on repos which make use of the "wgpu" library/crate. This library is published by Firefox and its repo contains big chunks of the actual Firefox web browser stack so I theorize that this library triggers the bug because building it happens to create/touch very many intermediate files.

With this configuration/setup complete, the actual repro steps are:

• In Sublime Text 4, use "File"->"Open Folder" and open the "zap" repository⁴.
• Open Task Manager [ctrl-alt-del] and sort by "CPU". Leave the window open.
  • While Sublime Text/LSP-rust-analyzer is doing its initial scan of the repo (messages will appear in the bottom left corner of the Sublime Text window), Sublime Text will jump to the top of the task list. Click the > next to Sublime Text 4 to see its subprocesses. You will see substantial amounts of CPU used by all three of Sublime Text, rust-analyzer.exe, and the Node process running Chokidar (this is expected/"correct" as they are doing actual work). Wait for this work to complete.
• In a CMD window, navigate to the "zap" repository and run cargo build or cargo run.
  • Expected behavior is it builds and Chokidar's node process does nothing at all. Observed behavior at this point is the screenshot up top, with CPU usage going to 100% and the Node.js Javascript Runtime running Chokidar doing enormous amounts of work during the build.
  • If you want to reset conditions to test again, I recommend using "git checkout" to switch to some early commit of the "zap" project, cargo build again, then switch back, which will mess up the timestamps and make cargo build want to rebuild.

Interestingly, disabling the Language Server either for the project or globally (ctrl-shift-p, "la di") does not cause Chokidar to stop its CPU-grabbing behavior once Chokidar has been opened for the zap directory. I don't know what this means.

Analysis

In the LSP-rust-analyzer bug, a project contributor says:

The pattern that rust-analyzer chooses to watch is [...]/rs-hello/**/*.rs which means that the whole rs-hello folder and all subfolders need to be watched recursively. Even if target directory is excluded using ignores, I'm pretty sure that the implemenation might have to watch it anyway and just filter out all events after they happen.

This sounds correct to me. I theorize that chokidar is receiving a recursive watch list and an ignore list, and on Ubuntu (and probably Mac, which I haven't tested) it is interfacing with some file system watcher by giving it the recursive watch list and the ignore list. On Windows I theorize there is no way to tell the OS file watcher to ignore paths, so you're just grabbing the whole recursive directory structure and filtering on the ignore pattern inside the Chokidar process. However, in my use case this is turning out to be unacceptable because the Chokidar process's filtering is turning out to be a larger contribution to CPU load than anything else.

My suggested fix would be, on Windows, if you have a recursively watched directory containing ignores matches, do not watch that directory recursively. Instead, watch that directory non-recursively, and watch its non-ignored constituents recursively. If a directory being watched recursively develops new members, you can check the ignores list to see whether those new members require switching to watching the parent directory non-recursively. This way all files will still be seen, but the ignored directories really will be ignored and Chokidar will not misbehave during activity in an ignored directory.

Footnotes

¹ I believe I am also seeing Chokidar use heavy CPU during operations by my version control system Mercurial, which also should not happen because Chokidar has also been told to ignore .hg, but I haven't gathered as much information on this symptom as it is harder to create an intentional repro.

² Here is my current Sublime configuration:

	"binary_file_patterns": ["*.jpg", "*.jpeg", "*.png", "*.gif", "*.ttf", "*.tga", "*.dds", "*.ico", "*.eot", "*.pdf", "*.swf", "*.jar", "*.zip", "target/*", "pkg/*", "node_modules/*"],
	"index_exclude_patterns": ["target/*", "pkg/*", "dist/*", "node_modules/*"],

"target" is, again, the directory I am most concerned with as it is where Rust Cargo does build.

This would then be merged with these additional default Sublime configuration keys:

	"folder_exclude_patterns": [".svn", ".git", ".hg", "CVS", ".Trash", ".Trash-*"],
	"file_exclude_patterns": ["*.pyc", "*.pyo", "*.exe", "*.dll", "*.obj","*.o", "*.a", "*.lib", "*.so", "*.dylib", "*.ncb", "*.sdf", "*.suo", "*.pdb", "*.idb", ".DS_Store", ".directory", "desktop.ini", "*.class", "*.psd", "*.db", "*.sublime-workspace"],

³ Apparently Sublime Text can run in a mode where it uses Electron instead of Node, I have not explored this.

⁴ A simple maze game. Controls are arrow keys to move or delete/esc to reset maze. No, it isn't very good.

[paulmillr]

@mcclure I really appreciate your report, it's cool and super detailed.

However, I won't be able to assist with it, since chokidar is not in my focus these days. Unfortunately, people don't contribute much to it, so finding a new maintainer has not been successful.

Windows is using shitty watching system, I think, it's possible to speed-up it a lot by switching to fs.watch recursive: true, but that will require some involved work.