Uses a ton of CPU in polling mode #2

Open
STRML opened this Issue Dec 6, 2014 · 7 comments

Projects

None yet

6 participants

@STRML
STRML commented Dec 6, 2014

Hi,

I'm developing an app and switching over to using Docker containers for some of it. Since I'm running OS X, I'm running Docker in Vagrant and using NFS to share my working directory, then building containers from the code in that directory.

It can be difficult to find a file watcher that works properly over NFS. Nodemon fails, pm2 eats up CPU, but node-supervisor seems to do well by just doing a slow poll. I get great <1% CPU usage watching with supervisor.

Unfortunately, webpack-dev-server with the old watcher doesn't catch changes at all, so I'm using NewWatchingPlugin. It does catch changes, but it pegs me at about 220% CPU (5 threads)!

I've played around with the invocation of Chokidar in DirectoryWatcher. Setting the polling interval to 5000 gets the CPU usage down quite a bit, to about 30%. It is obviously very slow to react, though, and that's still much higher than I would like.

I stuck a logging statement at the invocation of Chokidar and found that it's watching about 90-100 directories; every single module my project imports, as well as all of the project folders. It's easy to see how, with polling, this could get expensive.

Is it possible to:

  1. Make this more efficient by coalescing into a single polling watcher instead of 100?
  2. Do the watching instead from the host OS (which can use FSEvents) and signal to webpack inside the container somehow (create a file, have it listen on a port, etc)? Either the host OS could rebuild, or webpack in the container could.

The solution doesn't have to be super-elegant; this is just a dev environment.

@STRML
STRML commented Dec 6, 2014

For reference, regarding working around this, here's what I came up with to get the webpack dev server to play nicely with the remote host:

I serve the frontend on port 80 of docker.dev. I run webpack on the host with:

node_modules/.bin/webpack-dev-server --config webpack-hot-dev-server.config.js --hot --progress \
                               --colors --port 2002 --content-base http://docker.dev

The webpack config has a publicPath set to http://localhost:2002 so that the UI knows where to get updates.

Additionally when adding the dev server (this code is adapted from one of the webpack boilerplates), I set the client path to localhost:

    entry = joinEntry("webpack-dev-server/client?http://localhost:2002, entry);

I then load the actual app javascript from http://localhost:2002/app.js in development mode, and not from /app.js as in every other mode.

This feels really hacky but it works and keeps my laptop much cooler since FSEvents is so much more efficient than 100 pollers over NFS.

@mikepb
mikepb commented Jun 14, 2015

On OS X, a profiler run shows that the majority of CPU is spent in the kernel:

  ticks   total  ticks   total
  92978   77.1%  92978   77.1%  /usr/lib/system/libsystem_kernel.dylib

Much of this time is spent context switching between the kernel and the application.

@kylebyerly-hp

If you add
ignored: /node_modules/,
as in this commit b13cefe
into the chokidar.watch options in DirectoryWatcher.js you get a massive reduction in CPU usage when polling.

@ruslantalpa

I did a trace to figure out what's going on so after looking at this

Trace
    at new DirectoryWatcher (/vagrant/frontend/node_modules/watchpack/lib/DirectoryWatcher.js:49:10)
    at WatcherManager.getDirectoryWatcher (/vagrant/frontend/node_modules/watchpack/lib/watcherManager.js:16:33)
    at WatcherManager.watchFile (/vagrant/frontend/node_modules/watchpack/lib/watcherManager.js:26:14)
    at EventEmitter.<anonymous> (/vagrant/frontend/node_modules/watchpack/lib/watchpack.js:34:49)
    at Array.map (native)
    at EventEmitter.watch (/vagrant/frontend/node_modules/watchpack/lib/watchpack.js:33:28)
    at NodeWatchFileSystem.watch (/vagrant/frontend/node_modules/webpack/lib/node/NodeWatchFileSystem.js:52:15)
    at Watching.watch (/vagrant/frontend/node_modules/webpack/lib/Compiler.js:87:47)
    at Watching._done (/vagrant/frontend/node_modules/webpack/lib/Compiler.js:83:8)
    at Watching.<anonymous> (/vagrant/frontend/node_modules/webpack/lib/Compiler.js:61:18)

it seems that webpack (Compiler.js + NodeWatchFileSystem.js) is requesting a watch on every file that the build depends on, including node_modules.
This is the main reason for high cpu load.
@sokra will the commit referenced in the previous commit make it to a release or do you somehow consider it a "hack".
Would you consider a PR that will allow a config like this?

watchOptions: {
  poll: true,
  ignore: /node_modules/,
  ... other "chokidar.watch" options ...
}

which basically will "forward" watch options down the stream.
Another way to do it is to have a config option "ignore" specifically for watchpack and do a filter on it somewhere in watchmanager: getDirectoryWatcher.

Which one would you prefer?

@STRML
STRML commented Jan 2, 2016

Excellent, @kylebyerly-hp . That works great. I agree with @ruslantalpa that it should be an option. Why not try submitting the PR and see how ti goes.

On OS X, I've also noticed that it's easy to break the fsevents module, so you might need a rebuild. If you get it to build then the CPU usage is quite nice, even though it's watching node_modules.

@elliottsj
Contributor

I've created a PR which adds an ignored option: #23

@aripalo
aripalo commented Feb 25, 2016

Here's a little bash script https://gist.github.com/aripalo/6d659fefc79dee72e8e3 that fixes the issue until #23 is merged and available in npm registry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment