Skip to content
This repository

High CPU utilization in 'development' mode on Windows #270

Open
drosen0 opened this Issue August 01, 2012 · 34 comments

6 participants

David Rosen Owen Barnes Paul Miller Benjamin Polidore alan kaiser Paul Jensen
David Rosen

Recently, node.exe is using 30-60% CPU continuously at idle, only while in SS 'development' mode. I don't think this was an issue previously, but I'm not sure at what point this became an issue.

  • Windows 7 32-bit
  • NodeJS 0.8.4
  • SocketStream master

I haven't tested other platforms or versions.

Owen Barnes

Oh dear - sounds like a file watching / polling issue with chokidar. I recently upgraded SocketStream to the latest version, which maybe the cause of the problem.

When you get a moment, could you try running SocketStream using an older version of chokidar and let me know how you get on? I don't get many opportunities to test SocketStream on Windows.

David Rosen

Yes, that's it:

  • chokidar@0.2.6 - ok
  • chokidar@0.3.0 - ok
  • chokidar@0.4.0 - high CPU
David Rosen drosen0 referenced this issue in paulmillr/chokidar August 02, 2012
Closed

High CPU utilization in Windows #14

Paul Miller

should be ok on linux, works ok (<1% CPU) on mac. 0.4 is stable because it uses fs polling, when 0.3 is windows-fast because it uses ReadDirectoryChanges. I don't think you can easily have both of them, likely it's a shitty node fs win32 implementation problem. but I'll see what I can do.

Issues that were fixed with switch to polling fs: paulmillr/chokidar#11, paulmillr/chokidar#7

Paul Miller

actually it worksforme. Just tested it on windows, 0% of cpu load. perhaps something wrong with your configuration.

i'm using node 0.8.4

David Rosen

How can I get more information about what's happening? Using Process Hacker, I can see threads within node.exe getting created every few seconds when using chokidar 0.4.0, but with 0.3.0 the number of threads remains constant. Would a nodetime profile help?

Paul Miller

it's all here.

tl;dr 0.3 uses fs.watch on windows, 0.4 uses fs.watchFile. watchFile polls filesystem every 100ms for changes when watch uses some internal badass api to get changes without polling, when they happen or something like that.

whereas the watch idea is obviously better, node.js made its api very horrible, not mentioning A LOT of bugs inside (you can even now search nodejs bugtracker and see many open watch bugs). watchFile appeared on windows only with nodejs 0.8, so it can be unperformant / not polished on some configs etc.

Paul Miller

what do you use for testing cpu load in this case? I installed socketstream 0.3.0, created new proj and did node app.js

David Rosen

Just tested using node 0.8.5 (just released) with chokidar 0.4.0, and it seems that node's fs.watchFile has been optimized a bit. Now I'm seeing CPU load of 12-18%. With chokidar 0.3.0 the load sits at 0.00-0.01%.

Part of the difference is I have a project built using SocketStream with many files being watched. If I start a new project, the CPU idles at 1.1-1.2%. Now go into the client/css folder and make a lot of copies of app.styl. With 512 copies of app.styl, the CPU hovers at 35-50%.

Paul Miller

On mac it's 10% for 1070 files / 41% for 4280 files (which is still only ⅕ of my macbook's cpu time). Still a lot of space for improvement of win ver, I guess.

Paul Miller

i'll definitely try again to switch to watch, but as i've said this cannot be done until nodejs will fix its 10+ bugs. stability > performance.

David Rosen

At least from my limited experience with chokidar 0.2.6 as used by SocketStream on Windows, it has been stable and reliable. I haven't tried on other platforms, but if there are no major issues, might I suggest that SocketStream use chokidar 0.3.0 until the fs.watch bugs are ironed out?

Paul Miller

I have an idea: what happens if you change options.interval in chokidar source code from 100 to ~300-500 (ms) etc?

Owen Barnes

Hi guys

Thanks for looking into this. I'd like to release SocketStream 0.3.1 this week.

It would be good if we could reach a consensus on either keeping chokidar 0.4.0 as is, or going back to using 0.3.0 - bearing in mind that almost everyone will now be on Node 0.8.

Owen

Paul Miller

I propose to keep 0.4.0 because

  1. Performance problems on the stuff on windows exist partially because of shitty node fs win32 optimization.
  2. Noone +1-d these performance problems.
  3. Again, 0.4 is much more reliable, especially on windows.
David Rosen

I'm using 0.3.0 on Windows and it seems flawless, though I can only vouch for chokidar as it is used within SocketStream. The CPU utilization of 0.4.0 is a blocker for me. But then, I could be the only one doing development on a 4 year old laptop.

Paul Miller

@drosen0

what happens if you change options.interval in chokidar source code from 100 to ~300-500 (ms) etc?

David Rosen

@paulmillr I apologize that I haven't gotten to this. My free time has been limited, but also I admit that I'm not too excited to try this change.

Quintupling the interval to 500ms, I'd expect my CPU to reduce from 15% down to 3%. That's low, but it still bothers me, since the prior version (a) uses virtually no CPU, (b) notices the changes instantly rather than at intervals, and (c) works flawlessly (for me).

While it's true that no one has +1'd this issue, I'm also not aware of any chokidar-related socketstream issues reported while it was on 0.2.6.

Benjamin Polidore

This doesn't happen for me on win7 64bit.

David Rosen

@polidore I'd be interested to know whether the difference is due to the 64-bit vs 32-bit versions. Just to be sure, have you tried with a large number of watched files? I tested using a new SS project, making 512 copies of app.styl, and @paulmillr tested using 4280 files.

When I get a chance, I'll test this on my wife's iMac which has nearly identical hardware specs to my 4-year-old laptop. I'll also give the options.interval change a test when I get a chance.

Benjamin Polidore
alan kaiser

Hi, definitely an issue for me, i have lots of images in our app, so when they're all in, the development mode goes into 150% cpu utilization (unusable)! (MacOS, latest SS 0.3.2, chokidar 0.4.0, nodejs 0.8.12)

Even with moderate numbers of images, the CPU utilization keeps going up pretty dramatically, in the 25% range, which is still usable, once it gets to the 100% range can't use it, so i disable liveReload

Changing the chokidar setting options.interval to 1000 puts it back in the 33% range, which is a little slow on startup, but sort of usable, at 500ms it's running 66%

Owen Barnes

Bah file watching in Node still sucks. If anyone could take a look at this and help improve it, it would be greatly appreciated.

Live Reload will become a 'Websocket Service' module in 0.4. Something that you will explicitly choose to use in your app (just one line of code), however the way it works under-the-hood is essentially the same. One advantage is that it will be easier to pass more options; however I'd much rather fix the underlying issue as many apps are going to contain hundreds of images.

If anyone can help improve this in 0.3, I'll ensure the changes get into 0.4.

David Rosen

When I get a chance, I'll test this on my wife's iMac

Oops I guess I never got around to this, but sounds like it's an issue on MacOS as well. On Windows, file watching works flawlessly (and with virtually zero CPU utilization) as long as I specify chokidar version 0.3.0 in package.json. @rednaxus can you please try chokidar 0.3.0 on MacOS?

alan kaiser

@drosen0, ok, done... tested 0.3.0 chokidar on MacOS, same result 160% CPU.... too bad

Note.. I have lots of image files (>1000), so you may not see the same results

Paul Miller

guys, I have an idea: what do you think about adding a chokidar option that will increase interval for some type of files (images etc) and then test how it’ll work out?

paulmillr/chokidar#29

Owen Barnes

Sounds like a good plan @paulmillr. It will be a lot easier to pass options directly through to chokidar in SocketStream 0.4. Will look something like: https://github.com/socketstream/socketstream-0.4/blob/master/example_app/app.js#L31

Paul Miller

Okay guys, I implemented this. Chokidar (on master) now allows to pass binaryInterval option.

Please try to set this to e.g. 1000 and post if there were any changes.

Paul Miller

chokidar 0.5 was released.

Owen Barnes

Thanks @paulmillr

@drosen0 please let me know if this helps. If so I'll do another release of SocketStream using chokidar 0.5 and document how to deal with large numbers of images.

Owen Barnes

Hi @drosen0

Please can you let me know if this helped. If I don't hear anything in a few days, I will presume it did and close the ticket. Cheers

alan kaiser

hi Owen, testing chokidar 0.5.0 on mac, not a significant improvement, sorry (160% cpu)... SS 0.3.2, is there something else i should be setting to check this?

Paul Jensen

Hi,

I'm going to take a look at this during the week.

Paul Miller

You can also use usePolling: false chokidar option, pretty sure CPU will be very low with that (though it is unreliable as uses fs.watch).

Paul Jensen
Owner

Hi, I bumped chokidar to 0.6.3, though I see 0.7 is out. I'd like to try and have a go at replicating this. I have a Macbook Air with OS X Mavericks and a Desktop with Ubuntu 13.10 available for testing this issue out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.