Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Improve performance #944

Closed
sindresorhus opened this Issue Feb 23, 2013 · 36 comments

Comments

Projects
None yet
5 participants
Owner

sindresorhus commented Feb 23, 2013

Ways we can improve the build time:

  • Try out using grunt-concurrent
    • Use it for: coffee, compass, imagemin, htmlmin, cssmin, svgmin tasks
    • Open to suggestions on other places it can be useful.
  • Watch can be slow if there's a lot of directories, explore using an alternative to gaze. https://npmjs.org/package/inotify-plusplus https://npmjs.org/package/watch-inotify
  • Make sure all the grunt tasks we use are optimized. Like using async.forEach instead async.forEachSeries
  • Optimize Compass (it's awfully slow). Ideas?
  • Profile the building and reload and get some hard numbers on where we need to optimize
  • Make all the Bower components we use use the ignore prop for faster downloading. (other changes for improved perf there will come to Bower soon).
  • Reduce npm dep tree. Both size and count.
  • Make npm and registry faster :p
Owner

addyosmani commented Feb 23, 2013

Glad to hear you agree it's worth a little exploration. For some insight, I've read that a few people have been able to halve their build times using this. We might run into issues with tasks that depend on each others output, but sure we could trim a little time down regardless.

Owner

sindresorhus commented Mar 7, 2013

Another idea that just came to me is that we can parallelize the npm install && bower install step by using child processes:

(npm install) & (bower install) &

This will execute them at the same time.

Member

passy commented Mar 7, 2013

Ha, I'm already doing this whenever I scaffold a new project. I think it should be safe to suggest that, it's just a bit tedious to type. npm install &; bower install maybe?

Owner

sindresorhus commented Mar 7, 2013

Sure, even better

Owner

sindresorhus commented Mar 7, 2013

@passy any other ideas on how we could improve perf is more than welcome :)

Member

passy commented Mar 7, 2013

@sindresorhus I wish we could replace compass by something faster like sassc, but unfortunately it's not there yet.

Owner

sindresorhus commented Mar 7, 2013

@passy yeah, me too...

Owner

addyosmani commented Mar 7, 2013

Worth adding sassc to the list for future consideration?

Owner

sindresorhus commented Mar 7, 2013

Well, sassc is just a wrapper around libsass (which is where the awesomeness is). I created grunt-sass a while back which uses node-sass, which is a node binding for libsass. We could consider using that in the future, but we would loose the compass functionality (though could be replaced), but the progress on libsass is slow. I have bigger hopes for Stylus to be honest.

Zearin commented Mar 29, 2013

I have bigger hopes for Stylus to be honest.

+1! 👍

Love Stylus. I much prefer it to SASS.

Member

kevva commented Mar 30, 2013

Love Stylus. I much prefer it to SASS.

Both have their pros and cons. @content especially is really nice in Sass.

Owner

sindresorhus commented Apr 6, 2013

We've optimized grunt-contrib-imagemin to do optimizations in parallel which should improve performance there considerably.

@sleeper is also looking into optimizing usemin with the 2.0 work.

Owner

sindresorhus commented Apr 6, 2013

Any other ideas are more than welcome :)

Owner

sindresorhus commented Apr 7, 2013

I've worked on some of the performance stuff today. Pushed some changes to make some of the heavy grunt tasks run concurrently for better perf: https://github.com/yeoman/generator-webapp/commits/master

@sleeper @passy @kevva can you take a look and double-check I didn't screw anything up?

Also added a target to compass which should make Livereload a tiny bit faster.

Owner

sindresorhus commented Apr 7, 2013

Merged yeoman/generator-webapp#32 which will also improve perf.

Owner

sindresorhus commented Apr 7, 2013

I've added some more ideas in the issue description. Feel free to add your own.

The "Make all the Bower components we use use the include prop for faster downloading." could give us a nice boost, but we need help updating all the components.

Owner

sindresorhus commented Apr 7, 2013

@passy landed an awesome patch which significantly improves the startup performance of the generator system: yeoman/generator#201 \o/

Owner

addyosmani commented Apr 7, 2013

Well done, Pascal!

Member

passy commented Apr 7, 2013

Thanks!

Owner

sindresorhus commented Apr 8, 2013

Looked into our dep three and encountered some interesting findings. Looks like grunt-requirejs consumes a whopping 75MB since it requires 3 libs which all require grunt as a dep (which is wrong). Our download should improve significantly when that is fixed.

I also went on an .npmignore rampage today and opened like 15 PRs for big/popular offenders. When those are merged it will reduce the total size of our dependency tree significantly. This also benefits Bower and Grunt.

@sindresorhus sindresorhus referenced this issue in yeoman/generator-webapp_DEPRECATED Apr 8, 2013

Closed

Discussion: 5min+ build time and 122M of dependencies? #40

Owner

addyosmani commented May 12, 2013

@sindresorhus do you have a good feel for the size of the dep tree once your changes are merged into those projects? Right now its still taking quite a while to scaffold out RequireJS projects using the Backbone generator and a few others.

Owner

sindresorhus commented May 12, 2013

@addyosmani It will take awhile for those changes to cripple out. Some of those are 3rd level deps. Though we'll get a huge improvement by doing #1050. We've also been doing many other things to improve the situation both in dep size and performance. Keep in mind that the Backbone generator is not the same as Webapp and might have other dependencies and might not be in complete sync with the webapp generator.

Owner

sindresorhus commented May 12, 2013

Oh, and I spent too many hours this weekend trying to profile the generator system. Turns out most of the node debugging/profiling tools doesn't work. I ended just putting a lot of console.time statements in various places. A big bottleneck is registering the generators, well only some of them: https://gist.github.com/sindresorhus/ec2d23b88beb8f9d2b87 We need to investigate further.

Member

passy commented May 12, 2013

I believe it could make a big difference if we could somehow remove the need to execute the constructors of the generators when registering them. Many contain synchronous calls to the filesystem like

this.indexFile = this.readFileAsString(path.join(this.sourceRoot(), 'index.html'));
this.pkg = JSON.parse(this.readFileAsString(path.join(__dirname, '../package.json')));

Those are executed whenever yo is started and add up quite quickly.

Owner

addyosmani commented May 17, 2013

We're probably going to spend some time getting out 1.0 final in the coming weeks, but I think 1.1 could be a perfect candidate for resolving some of these performance issues. Perhaps tackle 1-2 issues in the list each?

@sindresorhus, why does the include prop offer improvements wrt Bower components? (sorry, I'm stupid) :)

Owner

sindresorhus commented May 17, 2013

@addyosmani I was voted down on the include prop, but I guess you mean the ignore prop. It enables authors to ignore unneeded files which can significantly improve the download speed of a component. Most of the repo size of libs are in their /test folder and other junk. It's also less files for Bower to copy from the cache to the actual components folder.

Owner

addyosmani commented May 18, 2013

Oh, of course! I remember the discussion around ignore. Seeing how many components still pull down their docs, tests and other unnecessary crap +9001 on us taking advantage of this more.

I was wondering - so, after we scaffold out an app we then go and resolve/fetch dependencies which tends to take a while. Is there any way/benefit to us completing that in the background so that developers can actually start working on the code for their app more quickly?

Alternatively, I personally wouldn't mind the time taken for dep fetching to be after I run a watch, build or test for the first time - I usually just care about getting a good baseline setup first. Are any of these ideas feasible?

Member

passy commented May 18, 2013

@addyosmani So you mean we should call bower install from grunt automatically if it hasn't been run before?

Owner

addyosmani commented May 18, 2013

Here's the workflow I'm imagining:

yo webapp - scaffolds out your application, does bower install but doesn't npm install grunt tasks etc.

yo watch/build/test - 'Please wait while we fetch the dependencies needed for building this project' - fetch/install grunt tasks at that point.

it could be really stupid (and I haven't had coffee yet) but sharing the idea in case it has legs.

Member

passy commented May 18, 2013

Gotcha. Why bower upfront and npm in the second step? Wouldn't it be easier to keep them together?

☕️

Owner

addyosmani commented May 18, 2013

That's a good point. I guess I was thinking that bower upfront would allow you to semi-preview your app with basic deps, but you're right - makes sense to keep them together.

Does the idea of installing the deps later on make sense?

Member

passy commented May 18, 2013

One problem I see is that in order to run grunt, we need to install the grunt runtime and the tasks. That means we would need to bootstrap a minimal grunt runtime while scaffolding.

Owner

sindresorhus commented May 18, 2013

Easiest solution is to make the upfront install (bower & npm) faster. There are only downsides to do it later on. I would rather see us trying to get all the Bower packages we use to use the ignore property and continue reducing our deps tree count and size.

Owner

addyosmani commented May 29, 2013

@sindresorhus do we know if there's a plan in place to encourage/evangelize the use of the ignore property more heavily? I agree that this offers more practical gains and it should be done anyway. I'm just trying to think of what else we can do to accelerate adoption of the property beyond filing PRs for all repos.

Owner

sindresorhus commented May 29, 2013

@addyosmani add your thoughts here: bower/bower.github.io#10

Owner

sindresorhus commented May 1, 2014

We've worked hard the last year making everything faster. While there is still places to optimize further I think the performance is much better now.

If you notice something is slower than it should please do open a ticket on the relevant repo ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment