Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dev-mode file change watching is CPU-intensive #2135

Closed
bdefore opened this issue May 9, 2014 · 14 comments
Closed

dev-mode file change watching is CPU-intensive #2135

bdefore opened this issue May 9, 2014 · 14 comments
Assignees

Comments

@bdefore
Copy link

@bdefore bdefore commented May 9, 2014

I noticed a significant performance hit when I dropped ~800 image files into the public folder of a Meteor 0.8.1.1 test project. On my dev machine (a 2011 Macbook Air), running 'mrt' or 'meteor' would cause CPU pegging, regardless of having any clients open. Activity monitor reported in the range of 20-40%.

Reproduction steps:

Some further observations:

  • This impact would drop off proportionally as I removed files from public. For example, dropping down to about 400 image files dropped CPU load to roughly 12-20% usage.
  • Renaming 'public' to 'foo' did not have any impact.
  • Running 'meteor --release 0.6.5' on the same project leaves my CPU at around 5-7% usage, leading me to believe this is due to something introduced since then.

For now, this means I'll have to host my images outside of the Meteor project.

@bdefore
Copy link
Author

@bdefore bdefore commented May 9, 2014

One more observation:

  • Having four copies of the images directory in the root, for ~3200 images total, caused CPU drag to range from 75-85% on 0.8.1.1. Remained 5-8% on 0.6.5
@bdefore bdefore changed the title high cpu load when many files are in meteor project root high cpu load running server with many files in meteor project root May 9, 2014
@bdefore
Copy link
Author

@bdefore bdefore commented May 9, 2014

→ node -v
v0.10.26

@timhaines
Copy link
Contributor

@timhaines timhaines commented May 9, 2014

@bdefore
Copy link
Author

@bdefore bdefore commented May 10, 2014

thanks @timhaines, useful context. before closing i'd like to have someone on meteor confirm it's only in dev env?

@timhaines
Copy link
Contributor

@timhaines timhaines commented May 10, 2014

@bdefore In that thread, David Glasser who works for Meteor says 'This only affects dev mode, since of course meteor run is not designed for production use.'

@glasser
Copy link
Member

@glasser glasser commented May 14, 2014

This is a known issue. The Node built-in file watching APIs are buggy (we used to use them but then they started crashing) and we'd like to do something better. The Atom editor folks have a new file watching package but it doesn't quite work for us; if atom/node-pathwatcher#24 is addressed I'll look into switching to it.

@glasser glasser changed the title high cpu load running server with many files in meteor project root dev-mode file change watching is CPU-intensive May 14, 2014
@ghost
Copy link

@ghost ghost commented Jul 2, 2014

@glasser Do you already know https://github.com/shama/gaze? It looks very promising and is used by grunt and gulp.

@glasser
Copy link
Member

@glasser glasser commented Jul 22, 2014

@sanjo I hadn't seen that, thanks. I'm not going to have time to look into this before 0.9.0 but hopefully once that's out I can look into it... my laptop battery would appreciate it :)

@ghost
Copy link

@ghost ghost commented Aug 1, 2014

I have done a basic integration of gaze. I replaced the watcher that watches for file changes to restart the app. It reduced the idle CPU usage of my medium size Meteor app from 25.5% to 0.8% on a Intel Core 2 Duo with 2.4 GHz with Mac OS X 10.9.4.

devel: https://github.com/Sanjo/meteor/tree/gaze
0.8.2: https://github.com/Sanjo/meteor/tree/release/0.8.2-gaze
You have to install gaze with npm install gaze inside the dev_bundle/lib/node_modules folder (after it has been created).

@glasser
Copy link
Member

@glasser glasser commented Aug 1, 2014

That looks like a good start! I would want it to only watch the files in our WatchSet data structure, though, instead of every file in the tree.

It also looks like there's no way to tell Gaze "hey, this is the expected hashes of these files, you should fire immediately if they've already changed". We've learned the hard way that there are race conditions if you don't do that; many of the watching methods fail to detect changes that are very soon after the watch started, or between the use of the file by the compiler and the beginning of the watch. (And we can't start the watch before the build, because we won't know ahead of time which files to watch: eg, we need to watch package source that isn't physically inside the app too.)

@ghost
Copy link

@ghost ghost commented Aug 1, 2014

I also tried to integrate gaze into the existing Watcher first, to make it behave exactly the same. But it doesn't work correctly yet. I did the other approach to save time, because I only wanted to see the impact of the switch. Here is the branch: https://github.com/Sanjo/meteor/tree/gaze-full.
The approach of the branch is to use the existing hashing mechanism of Meteor for the first check but use the efficient gaze watcher for watching the files.

@glasser
Copy link
Member

@glasser glasser commented Aug 12, 2014

Looks like pathwatcher did drop their stderr spamming today.

@ghost
Copy link

@ghost ghost commented Nov 16, 2014

Awesome. Thanks for fixing this. :-)

@benjamn
Copy link
Member

@benjamn benjamn commented Nov 26, 2014

This activity log has taught me not to mention an issue in a commit message until I'm ready to push it. Sorry for the noise!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants