High CPU utilization in 'development' mode on Windows #270

Closed
drosen0 opened this Issue Aug 1, 2012 · 37 comments

Projects

None yet

6 participants

@drosen0

Recently, node.exe is using 30-60% CPU continuously at idle, only while in SS 'development' mode. I don't think this was an issue previously, but I'm not sure at what point this became an issue.

  • Windows 7 32-bit
  • NodeJS 0.8.4
  • SocketStream master

I haven't tested other platforms or versions.

@owenb

Oh dear - sounds like a file watching / polling issue with chokidar. I recently upgraded SocketStream to the latest version, which maybe the cause of the problem.

When you get a moment, could you try running SocketStream using an older version of chokidar and let me know how you get on? I don't get many opportunities to test SocketStream on Windows.

@drosen0

Yes, that's it:

  • chokidar@0.2.6 - ok
  • chokidar@0.3.0 - ok
  • chokidar@0.4.0 - high CPU
@drosen0 drosen0 referenced this issue in paulmillr/chokidar Aug 2, 2012
Closed

High CPU utilization in Windows #14

@paulmillr

should be ok on linux, works ok (<1% CPU) on mac. 0.4 is stable because it uses fs polling, when 0.3 is windows-fast because it uses ReadDirectoryChanges. I don't think you can easily have both of them, likely it's a shitty node fs win32 implementation problem. but I'll see what I can do.

Issues that were fixed with switch to polling fs: paulmillr/chokidar#11, paulmillr/chokidar#7

@paulmillr

actually it worksforme. Just tested it on windows, 0% of cpu load. perhaps something wrong with your configuration.

i'm using node 0.8.4

@drosen0

How can I get more information about what's happening? Using Process Hacker, I can see threads within node.exe getting created every few seconds when using chokidar 0.4.0, but with 0.3.0 the number of threads remains constant. Would a nodetime profile help?

@paulmillr

it's all here.

tl;dr 0.3 uses fs.watch on windows, 0.4 uses fs.watchFile. watchFile polls filesystem every 100ms for changes when watch uses some internal badass api to get changes without polling, when they happen or something like that.

whereas the watch idea is obviously better, node.js made its api very horrible, not mentioning A LOT of bugs inside (you can even now search nodejs bugtracker and see many open watch bugs). watchFile appeared on windows only with nodejs 0.8, so it can be unperformant / not polished on some configs etc.

@paulmillr

what do you use for testing cpu load in this case? I installed socketstream 0.3.0, created new proj and did node app.js

@drosen0

Just tested using node 0.8.5 (just released) with chokidar 0.4.0, and it seems that node's fs.watchFile has been optimized a bit. Now I'm seeing CPU load of 12-18%. With chokidar 0.3.0 the load sits at 0.00-0.01%.

Part of the difference is I have a project built using SocketStream with many files being watched. If I start a new project, the CPU idles at 1.1-1.2%. Now go into the client/css folder and make a lot of copies of app.styl. With 512 copies of app.styl, the CPU hovers at 35-50%.

@paulmillr

On mac it's 10% for 1070 files / 41% for 4280 files (which is still only ⅕ of my macbook's cpu time). Still a lot of space for improvement of win ver, I guess.

@paulmillr

i'll definitely try again to switch to watch, but as i've said this cannot be done until nodejs will fix its 10+ bugs. stability > performance.

@drosen0

At least from my limited experience with chokidar 0.2.6 as used by SocketStream on Windows, it has been stable and reliable. I haven't tried on other platforms, but if there are no major issues, might I suggest that SocketStream use chokidar 0.3.0 until the fs.watch bugs are ironed out?

@paulmillr

I have an idea: what happens if you change options.interval in chokidar source code from 100 to ~300-500 (ms) etc?

@owenb

Hi guys

Thanks for looking into this. I'd like to release SocketStream 0.3.1 this week.

It would be good if we could reach a consensus on either keeping chokidar 0.4.0 as is, or going back to using 0.3.0 - bearing in mind that almost everyone will now be on Node 0.8.

Owen

@paulmillr

I propose to keep 0.4.0 because

  1. Performance problems on the stuff on windows exist partially because of shitty node fs win32 optimization.
  2. Noone +1-d these performance problems.
  3. Again, 0.4 is much more reliable, especially on windows.
@drosen0

I'm using 0.3.0 on Windows and it seems flawless, though I can only vouch for chokidar as it is used within SocketStream. The CPU utilization of 0.4.0 is a blocker for me. But then, I could be the only one doing development on a 4 year old laptop.

@paulmillr

@drosen0

what happens if you change options.interval in chokidar source code from 100 to ~300-500 (ms) etc?

@drosen0

@paulmillr I apologize that I haven't gotten to this. My free time has been limited, but also I admit that I'm not too excited to try this change.

Quintupling the interval to 500ms, I'd expect my CPU to reduce from 15% down to 3%. That's low, but it still bothers me, since the prior version (a) uses virtually no CPU, (b) notices the changes instantly rather than at intervals, and (c) works flawlessly (for me).

While it's true that no one has +1'd this issue, I'm also not aware of any chokidar-related socketstream issues reported while it was on 0.2.6.

@polidore

This doesn't happen for me on win7 64bit.

@drosen0

@polidore I'd be interested to know whether the difference is due to the 64-bit vs 32-bit versions. Just to be sure, have you tried with a large number of watched files? I tested using a new SS project, making 512 copies of app.styl, and @paulmillr tested using 4280 files.

When I get a chance, I'll test this on my wife's iMac which has nearly identical hardware specs to my 4-year-old laptop. I'll also give the options.interval change a test when I get a chance.

@polidore
@rednaxus

Hi, definitely an issue for me, i have lots of images in our app, so when they're all in, the development mode goes into 150% cpu utilization (unusable)! (MacOS, latest SS 0.3.2, chokidar 0.4.0, nodejs 0.8.12)

Even with moderate numbers of images, the CPU utilization keeps going up pretty dramatically, in the 25% range, which is still usable, once it gets to the 100% range can't use it, so i disable liveReload

Changing the chokidar setting options.interval to 1000 puts it back in the 33% range, which is a little slow on startup, but sort of usable, at 500ms it's running 66%

@owenb

Bah file watching in Node still sucks. If anyone could take a look at this and help improve it, it would be greatly appreciated.

Live Reload will become a 'Websocket Service' module in 0.4. Something that you will explicitly choose to use in your app (just one line of code), however the way it works under-the-hood is essentially the same. One advantage is that it will be easier to pass more options; however I'd much rather fix the underlying issue as many apps are going to contain hundreds of images.

If anyone can help improve this in 0.3, I'll ensure the changes get into 0.4.

@drosen0

When I get a chance, I'll test this on my wife's iMac

Oops I guess I never got around to this, but sounds like it's an issue on MacOS as well. On Windows, file watching works flawlessly (and with virtually zero CPU utilization) as long as I specify chokidar version 0.3.0 in package.json. @rednaxus can you please try chokidar 0.3.0 on MacOS?

@rednaxus

@drosen0, ok, done... tested 0.3.0 chokidar on MacOS, same result 160% CPU.... too bad

Note.. I have lots of image files (>1000), so you may not see the same results

@paulmillr

guys, I have an idea: what do you think about adding a chokidar option that will increase interval for some type of files (images etc) and then test how it’ll work out?

paulmillr/chokidar#29

@owenb

Sounds like a good plan @paulmillr. It will be a lot easier to pass options directly through to chokidar in SocketStream 0.4. Will look something like: https://github.com/socketstream/socketstream-0.4/blob/master/example_app/app.js#L31

@paulmillr

Okay guys, I implemented this. Chokidar (on master) now allows to pass binaryInterval option.

Please try to set this to e.g. 1000 and post if there were any changes.

@paulmillr

chokidar 0.5 was released.

@owenb

Thanks @paulmillr

@drosen0 please let me know if this helps. If so I'll do another release of SocketStream using chokidar 0.5 and document how to deal with large numbers of images.

@owenb

Hi @drosen0

Please can you let me know if this helped. If I don't hear anything in a few days, I will presume it did and close the ticket. Cheers

@rednaxus

hi Owen, testing chokidar 0.5.0 on mac, not a significant improvement, sorry (160% cpu)... SS 0.3.2, is there something else i should be setting to check this?

@paulbjensen

Hi,

I'm going to take a look at this during the week.

@paulmillr

You can also use usePolling: false chokidar option, pretty sure CPU will be very low with that (though it is unreliable as uses fs.watch).

@paulbjensen

Hi, I bumped chokidar to 0.6.3, though I see 0.7 is out. I'd like to try and have a go at replicating this. I have a Macbook Air with OS X Mavericks and a Desktop with Ubuntu 13.10 available for testing this issue out.

@paulbjensen

Hi @drosen0, are you still experiencing this issue. If not, can I close the issue?

@drosen0

Hi @paulbjensen, thank you for following up on this issue. I haven't used SocketStream in quite a long time (and also I switched development platforms), so I can't comment on whether it's fixed. I do use grunt-contrib-watch, which uses gaze. I've noticed that on Mac, some of the projects I work on do use considerable CPU while watching (it seems to be those using an older version of grunt-contrib-watch, and therefore older version of gaze; or it could be that the older projects have more files) and some do not.

@drosen0 drosen0 closed this Jan 3, 2015
@paulmillr

Gaze is not needed. Chokidar has all the needed fixes. I can't reproduce the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment