I'm developing an app and switching over to using Docker containers for some of it. Since I'm running OS X, I'm running Docker in Vagrant and using NFS to share my working directory, then building containers from the code in that directory.
It can be difficult to find a file watcher that works properly over NFS. Nodemon fails, pm2 eats up CPU, but node-supervisor seems to do well by just doing a slow poll. I get great <1% CPU usage watching with supervisor.
Unfortunately, webpack-dev-server with the old watcher doesn't catch changes at all, so I'm using NewWatchingPlugin. It does catch changes, but it pegs me at about 220% CPU (5 threads)!
I've played around with the invocation of Chokidar in DirectoryWatcher. Setting the polling interval to 5000 gets the CPU usage down quite a bit, to about 30%. It is obviously very slow to react, though, and that's still much higher than I would like.
I stuck a logging statement at the invocation of Chokidar and found that it's watching about 90-100 directories; every single module my project imports, as well as all of the project folders. It's easy to see how, with polling, this could get expensive.
Is it possible to:
The solution doesn't have to be super-elegant; this is just a dev environment.
For reference, regarding working around this, here's what I came up with to get the webpack dev server to play nicely with the remote host:
I serve the frontend on port 80 of docker.dev. I run webpack on the host with:
node_modules/.bin/webpack-dev-server --config webpack-hot-dev-server.config.js --hot --progress \
--colors --port 2002 --content-base http://docker.dev
The webpack config has a publicPath set to http://localhost:2002 so that the UI knows where to get updates.
Additionally when adding the dev server (this code is adapted from one of the webpack boilerplates), I set the client path to localhost:
entry = joinEntry("webpack-dev-server/client?http://localhost:2002, entry);
This feels really hacky but it works and keeps my laptop much cooler since FSEvents is so much more efficient than 100 pollers over NFS.
On OS X, a profiler run shows that the majority of CPU is spent in the kernel:
ticks total ticks total
92978 77.1% 92978 77.1% /usr/lib/system/libsystem_kernel.dylib
Much of this time is spent context switching between the kernel and the application.
If you add
as in this commit b13cefe
into the chokidar.watch options in DirectoryWatcher.js you get a massive reduction in CPU usage when polling.
I did a trace to figure out what's going on so after looking at this
at new DirectoryWatcher (/vagrant/frontend/node_modules/watchpack/lib/DirectoryWatcher.js:49:10)
at WatcherManager.getDirectoryWatcher (/vagrant/frontend/node_modules/watchpack/lib/watcherManager.js:16:33)
at WatcherManager.watchFile (/vagrant/frontend/node_modules/watchpack/lib/watcherManager.js:26:14)
at EventEmitter.<anonymous> (/vagrant/frontend/node_modules/watchpack/lib/watchpack.js:34:49)
at Array.map (native)
at EventEmitter.watch (/vagrant/frontend/node_modules/watchpack/lib/watchpack.js:33:28)
at NodeWatchFileSystem.watch (/vagrant/frontend/node_modules/webpack/lib/node/NodeWatchFileSystem.js:52:15)
at Watching.watch (/vagrant/frontend/node_modules/webpack/lib/Compiler.js:87:47)
at Watching._done (/vagrant/frontend/node_modules/webpack/lib/Compiler.js:83:8)
at Watching.<anonymous> (/vagrant/frontend/node_modules/webpack/lib/Compiler.js:61:18)
it seems that webpack (Compiler.js + NodeWatchFileSystem.js) is requesting a watch on every file that the build depends on, including node_modules.
This is the main reason for high cpu load.
@sokra will the commit referenced in the previous commit make it to a release or do you somehow consider it a "hack".
Would you consider a PR that will allow a config like this?
... other "chokidar.watch" options ...
which basically will "forward" watch options down the stream.
Another way to do it is to have a config option "ignore" specifically for watchpack and do a filter on it somewhere in watchmanager: getDirectoryWatcher.
Which one would you prefer?
Excellent, @kylebyerly-hp . That works great. I agree with @ruslantalpa that it should be an option. Why not try submitting the PR and see how ti goes.
On OS X, I've also noticed that it's easy to break the fsevents module, so you might need a rebuild. If you get it to build then the CPU usage is quite nice, even though it's watching node_modules.
I've created a PR which adds an ignored option: #23
Here's a little bash script https://gist.github.com/aripalo/6d659fefc79dee72e8e3 that fixes the issue until #23 is merged and available in npm registry.