Mimosa / node very high CPU usage on certain Windows environments

[alextreppass]

Hi,

I'm noticing some odd CPU behaviour with mimosa on my laptop, but not my desktop.

Background: I've been using Mimosa with no real CPU problems for quite some time on my desktop (Windows 7, Quad Core2 Q8450 @ 2.66 GHz, 8GB RAM):

• Node 0.9.9
• Mimosa 0.10.3

I've recently installed mimosa on my laptop (core i5 3320M @ 2.60 GHz, 8GB RAM, 256gb Samsung 830 SSD, which should be beefier than my desktop):

• Node x64, 0.10.5
• Mimosa 0.11.11

...and I'm finding it uses a lot of CPU on my laptop compared to my desktop environment, even when nothing is happening. Node.exe seems to sit at at >50% overall CPU usage across my four cores, even when I am not touching any files that would trigger recompiles / updates.

This is also the case on a number of other company laptops I have tried installing Mimosa on, even going back to node 0.9.9 and Mimosa 0.10.3 to match my desktop.

It is pretty odd, I suspect it's to do with file handles on windows - we've got 300+ js, sass, dust files and server views between our projects, and the project I am running this on is using import-source to bring in the assets of another project.

I've ran a node --prof to dump a v8.log, and left node running for ~20 minutes after startup where the Express server is listening in development mode, and nothing was happening on the system.

I've attached the node-tick-processor output over the v8.log for my laptop where CPU usage was very high: https://gist.github.com/alextreppass/e2b2771625d8c00c7124

Here's the profile for the desktop, exact same Mimosa and Node version where CPU usage was normal (i.e. quite low):
https://gist.github.com/alextreppass/88e7e0b9f4a993d60303

Both node tick reports are using Mimosa 0.11.11 and Node x64 0.10.5.

Any ideas / suggestions for investigation? I'm not sure if the desktop tick profile is significantly different from the laptop one.

Cheers,

[dbashford]

Hrm, I thought I had replied to this at some point, must have neglected to hit submit.

If you remove import-source from the equation, do you still get the CPU burn? So do a mimosa build to get import-source to bring the files in, comment out the module in the list, then fire up watch.

One problem I have had on my Mac is with Growl. Occasionally Growl decides to eat my machine alive and I have to shut it down and restart. It's become such a problem that I usually don't run with it on these days. I need to stop being lazy and reinstall growl as it has nothing to do with the node side of the equation.

[dbashford]

Hrm, I thought I had replied to this at some point, must have neglected to hit submit.

If you remove import-source from the equation, do you still get the CPU burn? So do a mimosa build to get import-source to bring the files in, comment out the module in the list, then fire up watch.

One problem I have had on my Mac is with Growl. Occasionally Growl decides to eat my machine alive and I have to shut it down and restart. It's become such a problem that I usually don't run with it on these days. I need to stop being lazy and reinstall growl as it has nothing to do with the node side of the equation.

[alextreppass]

Will try removing the import source, in case that's it.

We also have a couple of modules watching various file system dirs for certain file types.

Does nodejs do well with file system watchers? Does it hook into OS-level INotify or FindFirstChangeNotification stuff or does it have to keep scanning last modified in each folder it's watching?

Thanks

[alextreppass]

Will try removing the import source, in case that's it.

We also have a couple of modules watching various file system dirs for certain file types.

Does nodejs do well with file system watchers? Does it hook into OS-level INotify or FindFirstChangeNotification stuff or does it have to keep scanning last modified in each folder it's watching?

Thanks

[xixixao]

Not sure if related, but I noticed my node running mimosa (only --server, nothing added) to go to 1GB of memory the other day, over some longer period of time.

[xixixao]

Not sure if related, but I noticed my node running mimosa (only --server, nothing added) to go to 1GB of memory the other day, over some longer period of time.

[xixixao]

There is definitely some memory leak in mimosa, keeps crashing my test server.

[xixixao]

There is definitely some memory leak in mimosa, keeps crashing my test server.

[dbashford]

Is your test server running watch for long stretches?

Once I get a bunch of features out the door I can start profiling things to see what might be happening. I suspect the problem will lie in one of the many dependencies, but it is hard to say for sure.

[dbashford]

Is your test server running watch for long stretches?

Once I get a bunch of features out the door I can start profiling things to see what might be happening. I suspect the problem will lie in one of the many dependencies, but it is hard to say for sure.

[xixixao]

Yes. Agree.

[xixixao]

Yes. Agree.

[dbashford]

I'd be curious if you turned off require verification (assuming you have it on) if you continue to see the CPU increase, or if possibly all that does is slow it down.

[dbashford]

I'd be curious if you turned off require verification (assuming you have it on) if you continue to see the CPU increase, or if possibly all that does is slow it down.

[alextreppass]

I've turned off require, and while that improves the initial startup time, it doesn't consume less CPU.

I suspect it's to do with the file system watching.

[alextreppass]

I've turned off require, and while that improves the initial startup time, it doesn't consume less CPU.

I suspect it's to do with the file system watching.

[dbashford]

Yep, just trying to remove some other things from the equation. I'd imagine if you stripped things down to a bare minimum, so that all you are doing is watching and compiling (no lint, require, live reload etc) that you'd still have the issue?

[dbashford]

Yep, just trying to remove some other things from the equation. I'd imagine if you stripped things down to a bare minimum, so that all you are doing is watching and compiling (no lint, require, live reload etc) that you'd still have the issue?

[SMUnlimited]

Using mimosa 0.12.0 and the mimosa-import-source on lots of files I was getting major cpu issue (80% to 90% consistently) on a win7 64bit machine.

I then stumbled upon this issue for chokidar brunch/brunch#543

I manually modified chokidar index.js to use fs.watch and the cpu usage dropped down to 0 consistently.

Second I reverted this change and modified mimosa-import-source\index.js to specify interval:10000 and binaryInterval:10000 for the chokida watch command. Cpu usage once again was reduced to near 0 cpu usage.
(the defaults are interval:100 and binaryInterval:300 which explains alot)

Suggests we may need to expose these settings in the mimosa-import-source settings to specify the polling rate.

[SMUnlimited]

Using mimosa 0.12.0 and the mimosa-import-source on lots of files I was getting major cpu issue (80% to 90% consistently) on a win7 64bit machine.

I then stumbled upon this issue for chokidar brunch/brunch#543

I manually modified chokidar index.js to use fs.watch and the cpu usage dropped down to 0 consistently.

Second I reverted this change and modified mimosa-import-source\index.js to specify interval:10000 and binaryInterval:10000 for the chokida watch command. Cpu usage once again was reduced to near 0 cpu usage.
(the defaults are interval:100 and binaryInterval:300 which explains alot)

Suggests we may need to expose these settings in the mimosa-import-source settings to specify the polling rate.

[alextreppass]

Great spot.

Can you share info on how you modified chokidar? I'll go try it out, as I'm running mimosa over a large JS codebase here on Windows and are seeing almost 100% CPU usage through the watchers.

[alextreppass]

Great spot.

Can you share info on how you modified chokidar? I'll go try it out, as I'm running mimosa over a large JS codebase here on Windows and are seeing almost 100% CPU usage through the watchers.

[dbashford]

Thx for sniffing this out!

I've pushed import-source v0.4.0 which now exposes interval and intervalBinary, doc updates coming in a sec.

[dbashford]

Thx for sniffing this out!

I've pushed import-source v0.4.0 which now exposes interval and intervalBinary, doc updates coming in a sec.

[dbashford]

New default config:

  importSource:
    interval: 500
    binaryInterval: 1000
    copy:[]

[dbashford]

New default config:

  importSource:
    interval: 500
    binaryInterval: 1000
    copy:[]

[SMUnlimited]

@alextreppass
node_modules\chokidar\lib\index.js

Change this line to always be true: if ((process.platform === 'win32' && nodeVersion === '0.6')) { }
There is alot of debate if using fs.watch is safe or not on the linked post. Using the interval seems to be the safest approach.

[SMUnlimited]

@alextreppass
node_modules\chokidar\lib\index.js

Change this line to always be true: if ((process.platform === 'win32' && nodeVersion === '0.6')) { }
There is alot of debate if using fs.watch is safe or not on the linked post. Using the interval seems to be the safest approach.

[alextreppass]

Changing mimosa's chokidar and mimosa-import-source's chokidar to use fs.watch instead of fs.watchFile has taken my CPU usage down to 0-1% - what a result. It was previously regularly 100% and becoming unusable.

Closing this off :)

[alextreppass]

Changing mimosa's chokidar and mimosa-import-source's chokidar to use fs.watch instead of fs.watchFile has taken my CPU usage down to 0-1% - what a result. It was previously regularly 100% and becoming unusable.

Closing this off :)

[dbashford]

Keep me posted. I'd love to know if that has any negative effects and if passing the intervals helps alleviate any issues.

[dbashford]

Keep me posted. I'd love to know if that has any negative effects and if passing the intervals helps alleviate any issues.

[alextreppass]

So just FYI I'm using this on mimosa 0.12.3, using the chokidar index.js hack as well as a hack to mimosa-import-source to use fs.watch with the following intervals:

watch.watch(copyConfig.to, {
    persistent: true,
                interval: 10000,
                binaryInterval: 10000
  });

I'm pretty happy with the intervals as above - haven't noticed a missed file yet and CPU usage is minimal.

[alextreppass]

So just FYI I'm using this on mimosa 0.12.3, using the chokidar index.js hack as well as a hack to mimosa-import-source to use fs.watch with the following intervals:

watch.watch(copyConfig.to, {
    persistent: true,
                interval: 10000,
                binaryInterval: 10000
  });

I'm pretty happy with the intervals as above - haven't noticed a missed file yet and CPU usage is minimal.

[alextreppass]

I've raised the above in chokidar, suggesting that the watch options can take a some watch vs watchFile settings parameter, defaulting to watchFile.

[alextreppass]

I've raised the above in chokidar, suggesting that the watch options can take a some watch vs watchFile settings parameter, defaulting to watchFile.

[paulmillr]

fixed.

[paulmillr]

fixed.