New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mimosa / node very high CPU usage on certain Windows environments #179

Closed
alextreppass opened this Issue Apr 29, 2013 · 19 comments

Comments

Projects
None yet
5 participants
@alextreppass

alextreppass commented Apr 29, 2013

Hi,

I'm noticing some odd CPU behaviour with mimosa on my laptop, but not my desktop.

Background: I've been using Mimosa with no real CPU problems for quite some time on my desktop (Windows 7, Quad Core2 Q8450 @ 2.66 GHz, 8GB RAM):

  • Node 0.9.9
  • Mimosa 0.10.3

I've recently installed mimosa on my laptop (core i5 3320M @ 2.60 GHz, 8GB RAM, 256gb Samsung 830 SSD, which should be beefier than my desktop):

  • Node x64, 0.10.5
  • Mimosa 0.11.11

...and I'm finding it uses a lot of CPU on my laptop compared to my desktop environment, even when nothing is happening. Node.exe seems to sit at at >50% overall CPU usage across my four cores, even when I am not touching any files that would trigger recompiles / updates.

This is also the case on a number of other company laptops I have tried installing Mimosa on, even going back to node 0.9.9 and Mimosa 0.10.3 to match my desktop.

It is pretty odd, I suspect it's to do with file handles on windows - we've got 300+ js, sass, dust files and server views between our projects, and the project I am running this on is using import-source to bring in the assets of another project.

I've ran a node --prof to dump a v8.log, and left node running for ~20 minutes after startup where the Express server is listening in development mode, and nothing was happening on the system.

I've attached the node-tick-processor output over the v8.log for my laptop where CPU usage was very high: https://gist.github.com/alextreppass/e2b2771625d8c00c7124

Here's the profile for the desktop, exact same Mimosa and Node version where CPU usage was normal (i.e. quite low):
https://gist.github.com/alextreppass/88e7e0b9f4a993d60303

Both node tick reports are using Mimosa 0.11.11 and Node x64 0.10.5.

Any ideas / suggestions for investigation? I'm not sure if the desktop tick profile is significantly different from the laptop one.

Cheers,

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford May 3, 2013

Owner

Hrm, I thought I had replied to this at some point, must have neglected to hit submit.

If you remove import-source from the equation, do you still get the CPU burn? So do a mimosa build to get import-source to bring the files in, comment out the module in the list, then fire up watch.

One problem I have had on my Mac is with Growl. Occasionally Growl decides to eat my machine alive and I have to shut it down and restart. It's become such a problem that I usually don't run with it on these days. I need to stop being lazy and reinstall growl as it has nothing to do with the node side of the equation.

Owner

dbashford commented May 3, 2013

Hrm, I thought I had replied to this at some point, must have neglected to hit submit.

If you remove import-source from the equation, do you still get the CPU burn? So do a mimosa build to get import-source to bring the files in, comment out the module in the list, then fire up watch.

One problem I have had on my Mac is with Growl. Occasionally Growl decides to eat my machine alive and I have to shut it down and restart. It's become such a problem that I usually don't run with it on these days. I need to stop being lazy and reinstall growl as it has nothing to do with the node side of the equation.

@alextreppass

This comment has been minimized.

Show comment
Hide comment
@alextreppass

alextreppass May 9, 2013

Will try removing the import source, in case that's it.

We also have a couple of modules watching various file system dirs for certain file types.

Does nodejs do well with file system watchers? Does it hook into OS-level INotify or FindFirstChangeNotification stuff or does it have to keep scanning last modified in each folder it's watching?

Thanks

alextreppass commented May 9, 2013

Will try removing the import source, in case that's it.

We also have a couple of modules watching various file system dirs for certain file types.

Does nodejs do well with file system watchers? Does it hook into OS-level INotify or FindFirstChangeNotification stuff or does it have to keep scanning last modified in each folder it's watching?

Thanks

@xixixao

This comment has been minimized.

Show comment
Hide comment
@xixixao

xixixao Jun 6, 2013

Contributor

Not sure if related, but I noticed my node running mimosa (only --server, nothing added) to go to 1GB of memory the other day, over some longer period of time.

Contributor

xixixao commented Jun 6, 2013

Not sure if related, but I noticed my node running mimosa (only --server, nothing added) to go to 1GB of memory the other day, over some longer period of time.

@xixixao

This comment has been minimized.

Show comment
Hide comment
@xixixao

xixixao Jun 10, 2013

Contributor

There is definitely some memory leak in mimosa, keeps crashing my test server.

Contributor

xixixao commented Jun 10, 2013

There is definitely some memory leak in mimosa, keeps crashing my test server.

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford Jun 10, 2013

Owner

Is your test server running watch for long stretches?

Once I get a bunch of features out the door I can start profiling things to see what might be happening. I suspect the problem will lie in one of the many dependencies, but it is hard to say for sure.

Owner

dbashford commented Jun 10, 2013

Is your test server running watch for long stretches?

Once I get a bunch of features out the door I can start profiling things to see what might be happening. I suspect the problem will lie in one of the many dependencies, but it is hard to say for sure.

@xixixao

This comment has been minimized.

Show comment
Hide comment
@xixixao

xixixao Jun 10, 2013

Contributor

Yes. Agree.

Contributor

xixixao commented Jun 10, 2013

Yes. Agree.

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford Jun 29, 2013

Owner

I'd be curious if you turned off require verification (assuming you have it on) if you continue to see the CPU increase, or if possibly all that does is slow it down.

Owner

dbashford commented Jun 29, 2013

I'd be curious if you turned off require verification (assuming you have it on) if you continue to see the CPU increase, or if possibly all that does is slow it down.

@alextreppass

This comment has been minimized.

Show comment
Hide comment
@alextreppass

alextreppass Jul 1, 2013

I've turned off require, and while that improves the initial startup time, it doesn't consume less CPU.

I suspect it's to do with the file system watching.

alextreppass commented Jul 1, 2013

I've turned off require, and while that improves the initial startup time, it doesn't consume less CPU.

I suspect it's to do with the file system watching.

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford Jul 1, 2013

Owner

Yep, just trying to remove some other things from the equation. I'd imagine if you stripped things down to a bare minimum, so that all you are doing is watching and compiling (no lint, require, live reload etc) that you'd still have the issue?

Owner

dbashford commented Jul 1, 2013

Yep, just trying to remove some other things from the equation. I'd imagine if you stripped things down to a bare minimum, so that all you are doing is watching and compiling (no lint, require, live reload etc) that you'd still have the issue?

@SMUnlimited

This comment has been minimized.

Show comment
Hide comment
@SMUnlimited

SMUnlimited Aug 7, 2013

Using mimosa 0.12.0 and the mimosa-import-source on lots of files I was getting major cpu issue (80% to 90% consistently) on a win7 64bit machine.

I then stumbled upon this issue for chokidar brunch/brunch#543

I manually modified chokidar index.js to use fs.watch and the cpu usage dropped down to 0 consistently.

Second I reverted this change and modified mimosa-import-source\index.js to specify interval:10000 and binaryInterval:10000 for the chokida watch command. Cpu usage once again was reduced to near 0 cpu usage.
(the defaults are interval:100 and binaryInterval:300 which explains alot)

Suggests we may need to expose these settings in the mimosa-import-source settings to specify the polling rate.

SMUnlimited commented Aug 7, 2013

Using mimosa 0.12.0 and the mimosa-import-source on lots of files I was getting major cpu issue (80% to 90% consistently) on a win7 64bit machine.

I then stumbled upon this issue for chokidar brunch/brunch#543

I manually modified chokidar index.js to use fs.watch and the cpu usage dropped down to 0 consistently.

Second I reverted this change and modified mimosa-import-source\index.js to specify interval:10000 and binaryInterval:10000 for the chokida watch command. Cpu usage once again was reduced to near 0 cpu usage.
(the defaults are interval:100 and binaryInterval:300 which explains alot)

Suggests we may need to expose these settings in the mimosa-import-source settings to specify the polling rate.

@alextreppass

This comment has been minimized.

Show comment
Hide comment
@alextreppass

alextreppass Aug 7, 2013

Great spot.

Can you share info on how you modified chokidar? I'll go try it out, as I'm running mimosa over a large JS codebase here on Windows and are seeing almost 100% CPU usage through the watchers.

alextreppass commented Aug 7, 2013

Great spot.

Can you share info on how you modified chokidar? I'll go try it out, as I'm running mimosa over a large JS codebase here on Windows and are seeing almost 100% CPU usage through the watchers.

dbashford added a commit to dbashford/mimosa-import-source that referenced this issue Aug 7, 2013

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford Aug 7, 2013

Owner

Thx for sniffing this out!

I've pushed import-source v0.4.0 which now exposes interval and intervalBinary, doc updates coming in a sec.

Owner

dbashford commented Aug 7, 2013

Thx for sniffing this out!

I've pushed import-source v0.4.0 which now exposes interval and intervalBinary, doc updates coming in a sec.

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford Aug 7, 2013

Owner

New default config:

  importSource:
    interval: 500
    binaryInterval: 1000
    copy:[]
Owner

dbashford commented Aug 7, 2013

New default config:

  importSource:
    interval: 500
    binaryInterval: 1000
    copy:[]

dbashford added a commit that referenced this issue Aug 7, 2013

@SMUnlimited

This comment has been minimized.

Show comment
Hide comment
@SMUnlimited

SMUnlimited Aug 8, 2013

@alextreppass
node_modules\chokidar\lib\index.js

Change this line to always be true: if ((process.platform === 'win32' && nodeVersion === '0.6')) { }
There is alot of debate if using fs.watch is safe or not on the linked post. Using the interval seems to be the safest approach.

SMUnlimited commented Aug 8, 2013

@alextreppass
node_modules\chokidar\lib\index.js

Change this line to always be true: if ((process.platform === 'win32' && nodeVersion === '0.6')) { }
There is alot of debate if using fs.watch is safe or not on the linked post. Using the interval seems to be the safest approach.

@alextreppass

This comment has been minimized.

Show comment
Hide comment
@alextreppass

alextreppass Aug 8, 2013

Changing mimosa's chokidar and mimosa-import-source's chokidar to use fs.watch instead of fs.watchFile has taken my CPU usage down to 0-1% - what a result. It was previously regularly 100% and becoming unusable.

Closing this off :)

alextreppass commented Aug 8, 2013

Changing mimosa's chokidar and mimosa-import-source's chokidar to use fs.watch instead of fs.watchFile has taken my CPU usage down to 0-1% - what a result. It was previously regularly 100% and becoming unusable.

Closing this off :)

@dbashford

This comment has been minimized.

Show comment
Hide comment
@dbashford

dbashford Aug 8, 2013

Owner

Keep me posted. I'd love to know if that has any negative effects and if passing the intervals helps alleviate any issues.

Owner

dbashford commented Aug 8, 2013

Keep me posted. I'd love to know if that has any negative effects and if passing the intervals helps alleviate any issues.

@alextreppass

This comment has been minimized.

Show comment
Hide comment
@alextreppass

alextreppass Aug 8, 2013

So just FYI I'm using this on mimosa 0.12.3, using the chokidar index.js hack as well as a hack to mimosa-import-source to use fs.watch with the following intervals:

watch.watch(copyConfig.to, {
    persistent: true,
                interval: 10000,
                binaryInterval: 10000
  });

I'm pretty happy with the intervals as above - haven't noticed a missed file yet and CPU usage is minimal.

alextreppass commented Aug 8, 2013

So just FYI I'm using this on mimosa 0.12.3, using the chokidar index.js hack as well as a hack to mimosa-import-source to use fs.watch with the following intervals:

watch.watch(copyConfig.to, {
    persistent: true,
                interval: 10000,
                binaryInterval: 10000
  });

I'm pretty happy with the intervals as above - haven't noticed a missed file yet and CPU usage is minimal.

@alextreppass

This comment has been minimized.

Show comment
Hide comment
@alextreppass

alextreppass Aug 9, 2013

I've raised the above in chokidar, suggesting that the watch options can take a some watch vs watchFile settings parameter, defaulting to watchFile.

alextreppass commented Aug 9, 2013

I've raised the above in chokidar, suggesting that the watch options can take a some watch vs watchFile settings parameter, defaulting to watchFile.

@paulmillr

This comment has been minimized.

Show comment
Hide comment
@paulmillr

paulmillr commented Aug 9, 2013

fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment