Proposal: Adding Rendering Performance Tests for Components #12112

Merged
merged 0 commits into from Feb 6, 2014

Conversation

Projects
None yet
3 participants
@axemclion

What is in this pull request

The rendering performance tests are based on Chromium telemetry and help check performance regressions across commits.
The useful metrics measured include

  1. Smoothness - measured by way of mean time to draw each frame while scrolling
  2. Page load time - measured using firstPaintTime and window.performance

Adding 'heavy' CSS like blurs or shadows, or javascript that does unnecessary re-painting affect these numbers and indicate performance regressions. The idea is to ensure that as more features are added to the CSS, the components themselves have good CSS and do not become slow. Hence, performance regression testing as a part of continuous integration process

What has been added

A new grunt task has been added that generates temporary files, runs the performance tests for them in saucelabs and reports back the results. The results are stored in a CouchDB Database. The CouchApp in the same database is used to view this data over commits.

This is based on http://axemclion.github.com/bootstrap-perf and is inspired by the work done with http://bench.topcoat.io.
To run the performance task, run grunt connect perf. I have not yet hooked it into travis yet, will do that once this proposal to add rendering tasks is accepted.

Things to note

  1. The couchDB account is under my name and before the final merge, needs to be changed to a couchDB that bootstrap owns.
  2. The graph will be visible at http://coucdbserver/databasename/_design/site/index.html
  3. When running locally, the commit number and job id are not available in environment variable. That is the reason that you do not see any graphs yet. Once integrated with travis, all this information will be populated, allowing for a more useful graph.
@cvrebert

View changes

package.json
@@ -44,7 +44,10 @@
"grunt-jscs-checker": "~0.3.0",
"grunt-saucelabs": "~4.1.2",
"grunt-sed": "~0.1.1",
- "load-grunt-tasks": "~0.2.1"
+ "load-grunt-tasks": "~0.2.1",
+ "cheerio": "~0.13.0",

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Please keep this list alphabetized.

@cvrebert

cvrebert Jan 6, 2014

Member

Please keep this list alphabetized.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 6, 2014

More details about the pull request

  • Chromium Telementry is a page rendering benchmarking tool and is inside the Chromium source code. This pull request runs telemetry tests that were ported to NodeJs in a node module called (browser-perf)[http://github.com/axemclion/browser-perf]. The Chromium smoothness and loading tests are run to calculate the metrics.
  • All data is stored in a CouchDB instance. The perfjankie module is responsible for storing the data inside couchDB in addition to installing the couchapp it the database instance does not already have it. It also runs browser-perf internally.
  • Currently, saucelabs (specifically Chrome 31 on Saucelabs) runs an older version of the chrome driver. When Saucelabs will update this, we would have even richer data (same data in the developer tools timeline) like time taken the parse the HTML, time for paint, rendering, etc.

More details about the pull request

  • Chromium Telementry is a page rendering benchmarking tool and is inside the Chromium source code. This pull request runs telemetry tests that were ported to NodeJs in a node module called (browser-perf)[http://github.com/axemclion/browser-perf]. The Chromium smoothness and loading tests are run to calculate the metrics.
  • All data is stored in a CouchDB instance. The perfjankie module is responsible for storing the data inside couchDB in addition to installing the couchapp it the database instance does not already have it. It also runs browser-perf internally.
  • Currently, saucelabs (specifically Chrome 31 on Saucelabs) runs an older version of the chrome driver. When Saucelabs will update this, we would have even richer data (same data in the developer tools timeline) like time taken the parse the HTML, time for paint, rendering, etc.
@cvrebert

View changes

build/perf.js
+ this.async();
+ });
+ grunt.registerTask('perf', 'Run Performance test cases', function() {
+ var components = grunt.file.read('./_gh_pages/components/index.html', 'utf-8');

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

What about testing the example pages too (/docs/examples)?

@cvrebert

cvrebert Jan 6, 2014

Member

What about testing the example pages too (/docs/examples)?

This comment has been minimized.

@axemclion

axemclion Jan 6, 2014

I wanted to start with components - the idea here was benchmarking components.
The example pages are entire pages with multiple components. Do you see value in benchmarking them too ?

@axemclion

axemclion Jan 6, 2014

I wanted to start with components - the idea here was benchmarking components.
The example pages are entire pages with multiple components. Do you see value in benchmarking them too ?

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

/cc @mdo Your opinion? ^^^

@cvrebert

cvrebert Jan 6, 2014

Member

/cc @mdo Your opinion? ^^^

@cvrebert

View changes

Gruntfile.js
@@ -35,7 +35,7 @@ module.exports = function (grunt) {
jshintrc: 'js/.jshintrc'
},
gruntfile: {
- src: 'Gruntfile.js'
+ src: ['Gruntfile.js', 'build/**/*.js']

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Please also modify the JSCS task config accordingly.

@cvrebert

cvrebert Jan 6, 2014

Member

Please also modify the JSCS task config accordingly.

This comment has been minimized.

@cvrebert

View changes

Gruntfile.js
+ },
+ browsers: [{
+ browserName: 'chrome',
+ version: 31,

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

I believe you can leave out the version number and it will default to the latest available version.

@cvrebert

cvrebert Jan 6, 2014

Member

I believe you can leave out the version number and it will default to the latest available version.

This comment has been minimized.

@axemclion

axemclion Jan 6, 2014

Nope, i have seen that it shuffles between 31 and 28 when I leave out the version number. Wanted to be safe as the perf numbers (from about:tracing and developer tools timelines) are available only in 31.

@axemclion

axemclion Jan 6, 2014

Nope, i have seen that it shuffles between 31 and 28 when I leave out the version number. Wanted to be safe as the perf numbers (from about:tracing and developer tools timelines) are available only in 31.

@cvrebert

View changes

Gruntfile.js
+ port: 80
+ },
+ browsers: [{
+ browserName: 'chrome',

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Shouldn't a platform also be specified?

@cvrebert

cvrebert Jan 6, 2014

Member

Shouldn't a platform also be specified?

This comment has been minimized.

@axemclion

axemclion Jan 6, 2014

Wanted to leave it open - this allows chrome to open in different platforms, giving us better data across platforms when people run it.

or do you think that we are comparing versions, we need the platform should be a part of the standard environment ?

@axemclion

axemclion Jan 6, 2014

Wanted to leave it open - this allows chrome to open in different platforms, giving us better data across platforms when people run it.

or do you think that we are comparing versions, we need the platform should be a part of the standard environment ?

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Erm, so, you're saying that omitting the platform will make Sauce run it on all possible platforms?

@cvrebert

cvrebert Jan 6, 2014

Member

Erm, so, you're saying that omitting the platform will make Sauce run it on all possible platforms?

This comment has been minimized.

@axemclion

axemclion Jan 6, 2014

Nope, omitting the platform (I think) will make it choose a platform at random.

@axemclion

axemclion Jan 6, 2014

Nope, omitting the platform (I think) will make it choose a platform at random.

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Then I'd say we should pin it to one or more popular platform(s).

@cvrebert

cvrebert Jan 6, 2014

Member

Then I'd say we should pin it to one or more popular platform(s).

This comment has been minimized.

@axemclion

axemclion Jan 6, 2014

Sounds good. Will Windows 8.1 and Mac OSX Mavericks work ?

@axemclion

axemclion Jan 6, 2014

Sounds good. Will Windows 8.1 and Mac OSX Mavericks work ?

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Sounds good!

@cvrebert

cvrebert Jan 6, 2014

Member

Sounds good!

This comment has been minimized.

@cvrebert

cvrebert Jan 8, 2014

Member

@axemclion So, can you update the browsers setting accordingly?

@cvrebert

cvrebert Jan 8, 2014

Member

@axemclion So, can you update the browsers setting accordingly?

This comment has been minimized.

@cvrebert

View changes

build/perf.js
+ html.push(code.join('<br/>'), '<!-- Iteration:' + j + ' -->\n<br/>');
+ }
+ html.push('</body></html>')
+ grunt.file.write('./tmp/' + fileName + '.html', html.join(''));

This comment has been minimized.

@cvrebert

cvrebert Jan 6, 2014

Member

Adding a corresponding gitignore entry for this tmp folder would be nice.

@cvrebert

cvrebert Jan 6, 2014

Member

Adding a corresponding gitignore entry for this tmp folder would be nice.

This comment has been minimized.

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 6, 2014

Member

Sounds pretty cool!

Member

cvrebert commented Jan 6, 2014

Sounds pretty cool!

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 6, 2014

Member

I ran the numbers, and http://www.iriscouch.com seems to currently be the cheapest CouchDB SaaS.
@axemclion Any thoughts/recommendations as to which CouchDB provider we should choose?

Member

cvrebert commented Jan 6, 2014

I ran the numbers, and http://www.iriscouch.com seems to currently be the cheapest CouchDB SaaS.
@axemclion Any thoughts/recommendations as to which CouchDB provider we should choose?

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 6, 2014

Member

@mdo Does this sound good to you?

Member

cvrebert commented Jan 6, 2014

@mdo Does this sound good to you?

@mdo

This comment has been minimized.

Show comment
Hide comment
@mdo

mdo Jan 6, 2014

Member

I've wanted CSS performance testing for a long time, so I'm super intrigued by this. Whatever solution we implement should be clearly documented and part of our regular workflow. How does the Grunt task work?

Member

mdo commented Jan 6, 2014

I've wanted CSS performance testing for a long time, so I'm super intrigued by this. Whatever solution we implement should be clearly documented and part of our regular workflow. How does the Grunt task work?

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 6, 2014

Member

IIUC: Parses each example out of the Components docs page and shoves them into separate temporary HTML files. Uses Sauce to run Chrome on a couple platforms and has the Chromes browse each example page. Chrome's telemetry data while rendering the pages is captured and then uploaded (per page/example) to a CouchDB server that hosts a simple web app to browse the stats and compare them over time.
That about right @axemclion?

Member

cvrebert commented Jan 6, 2014

IIUC: Parses each example out of the Components docs page and shoves them into separate temporary HTML files. Uses Sauce to run Chrome on a couple platforms and has the Chromes browse each example page. Chrome's telemetry data while rendering the pages is captured and then uploaded (per page/example) to a CouchDB server that hosts a simple web app to browse the stats and compare them over time.
That about right @axemclion?

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 6, 2014

The grunt task is just a wrapper on top of a node module that I wrote. If you notice, all it does is

  1. Generate the files for testing. It reads the docs/components.html file, extracts out examples and uses that to construct the HTML. This HTML just has a specific component (like progress bar, alert, etc) repeated multiple times to allow for scrolling.
  2. It then calls into perfjankie. PerfJankie is a node which runs the actual chromium-telemetry-translated-to-node test cases to get performance measurements. Perfjankie then stores this data in a couchdb database. Perfjankie is also responsible for deploying a couchapp on the same database to see the data as a graph in a commit timeline.

I wanted to abstract most of this heavy lifting of running tests and saving data, so wrote that module. This way, any framework wanting to to CSS testing can use it.

The grunt task is just a wrapper on top of a node module that I wrote. If you notice, all it does is

  1. Generate the files for testing. It reads the docs/components.html file, extracts out examples and uses that to construct the HTML. This HTML just has a specific component (like progress bar, alert, etc) repeated multiple times to allow for scrolling.
  2. It then calls into perfjankie. PerfJankie is a node which runs the actual chromium-telemetry-translated-to-node test cases to get performance measurements. Perfjankie then stores this data in a couchdb database. Perfjankie is also responsible for deploying a couchapp on the same database to see the data as a graph in a commit timeline.

I wanted to abstract most of this heavy lifting of running tests and saving data, so wrote that module. This way, any framework wanting to to CSS testing can use it.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 6, 2014

@cvrebert yeah, you said it write. Just realized that I was typing the answer at the same time :)

@cvrebert yeah, you said it write. Just realized that I was typing the answer at the same time :)

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 8, 2014

@mdo, @cvrebert Is there a couchdb account for bootstrap that I can replace the account with?

@mdo, @cvrebert Is there a couchdb account for bootstrap that I can replace the account with?

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 13, 2014

@cvrebert Are there any others things required for this Pull request ?

@cvrebert Are there any others things required for this Pull request ?

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 13, 2014

Member

@axemclion Not for me.
@mdo You cool with this?

Member

cvrebert commented Jan 13, 2014

@axemclion Not for me.
@mdo You cool with this?

@mdo

This comment has been minimized.

Show comment
Hide comment
@mdo

mdo Jan 13, 2014

Member

What about documentation for running and viewing the test results? Even something in the readme for now would suffice for accessing the results. This also doesn't show an interface like Topcoat's for viewing these graphs—is that meant for another PR or for someone else to address?

Member

mdo commented Jan 13, 2014

What about documentation for running and viewing the test results? Even something in the readme for now would suffice for accessing the results. This also doesn't show an interface like Topcoat's for viewing these graphs—is that meant for another PR or for someone else to address?

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 13, 2014

Member

This also doesn't show an interface like Topcoat's for viewing these graphs

It does actually; see the previous comments.

Member

cvrebert commented Jan 13, 2014

This also doesn't show an interface like Topcoat's for viewing these graphs

It does actually; see the previous comments.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 13, 2014

@mdo, the graphs and the data, both are stored in CouchDB. This way, we would have one less server to manage :)

I will add documentation to a readme inside tasks folder.

@mdo, the graphs and the data, both are stored in CouchDB. This way, we would have one less server to manage :)

I will add documentation to a readme inside tasks folder.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 16, 2014

@mdo, the documentation has also been added. We just need a CouchDB instance that bootstrap can use (possibly a free account with iriscouch or cloudant) and we should be all set. Should I go ahead and create one ? I can share the credentials with the team.

@mdo, the documentation has also been added. We just need a CouchDB instance that bootstrap can use (possibly a free account with iriscouch or cloudant) and we should be all set. Should I go ahead and create one ? I can share the credentials with the team.

@mdo

This comment has been minimized.

Show comment
Hide comment
@mdo

mdo Jan 16, 2014

Member

@axemclion Great! Thank so much. I'll check this out after v3.1 ships—got a lot on my plate right now and I want to keep things as manageable as possible :). Expect to hear back next week.

Member

mdo commented Jan 16, 2014

@axemclion Great! Thank so much. I'll check this out after v3.1 ships—got a lot on my plate right now and I want to keep things as manageable as possible :). Expect to hear back next week.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 16, 2014

Sounds good, thank you.

Sounds good, thank you.

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 27, 2014

Member

I would like to reuse the individual HTML example files for #11410, so it would be nice if the code for extracting/generating those was its own separate Grunt task.

Member

cvrebert commented Jan 27, 2014

I would like to reuse the individual HTML example files for #11410, so it would be nice if the code for extracting/generating those was its own separate Grunt task.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 27, 2014

Makes sense. Will separate the tasks, and also rebase this. However, note that the components will be repeated multiple times for running the perf tests. I could have a grunt task variables that sets how many times they should be repeated.

Makes sense. Will separate the tasks, and also rebase this. However, note that the components will be repeated multiple times for running the perf tests. I could have a grunt task variables that sets how many times they should be repeated.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 28, 2014

@cvrebert, added the task, the name is probably bad for the task, but how does it look now ? Should I move it to some other place ?

Alternatively, that can be a separate pull request so that you are unblocked?

@cvrebert, added the task, the name is probably bad for the task, but how does it look now ? Should I move it to some other place ?

Alternatively, that can be a separate pull request so that you are unblocked?

@cvrebert

View changes

test-infra/perf.md
+## What does it do
+The `perf` task is defined in the `perf.js` file and does the following things.
+
+1. Generate the HTML files that are required for preformance testing. The HTML files have the markup of the component added multiple times to enable scrolling.

This comment has been minimized.

@cvrebert

cvrebert Jan 28, 2014

Member

typo: "preformance" [sic]

@cvrebert

cvrebert Jan 28, 2014

Member

typo: "preformance" [sic]

@cvrebert

View changes

test-infra/perf.js
+ return;
+ }
+ var cheerio = require('cheerio'),
+ perfjankie = require('perfjankie'),

This comment has been minimized.

@cvrebert

cvrebert Jan 28, 2014

Member

I think this require and the SauceTunnel require should be within the perf task? They're not used within this task.

@cvrebert

cvrebert Jan 28, 2014

Member

I think this require and the SauceTunnel require should be within the perf task? They're not used within this task.

@cvrebert

View changes

test-infra/perf.md
+The performance of various components can measured by running [Chromium Telemetry](http://www.chromium.org/developers/telemetry)'s Smoothness and Loading benchmarks. Running these benchmarks over time tells us about commits that may slow down certain components.
+
+## Usage
+To run the performance metrics, run `grunt connect gen_component_html:perf perf`. The SAUCE_USERNAME and SAUCE_ACCESS_KEY need to be set as environment variables to enable the runs.

This comment has been minimized.

@cvrebert

cvrebert Jan 28, 2014

Member

I think you also need to run the jekyll task before the gen_component_html task since it relies on the Jekyll-generated /_gh_pages/ directory?

@cvrebert

cvrebert Jan 28, 2014

Member

I think you also need to run the jekyll task before the gen_component_html task since it relies on the Jekyll-generated /_gh_pages/ directory?

@cvrebert

View changes

test-infra/perf.js
+ }
+ runPerfTest(0, function() {
+ grunt.verbose.writeln('All perf tests completed');
+ grunt.file.delete('./tmp');

This comment has been minimized.

@cvrebert

cvrebert Jan 28, 2014

Member

I think it'd be better to do the deletion at the start of the gen_component_html task instead.

@cvrebert

cvrebert Jan 28, 2014

Member

I think it'd be better to do the deletion at the start of the gen_component_html task instead.

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 28, 2014

Member

@axemclion I'm not blocked on it, so there's no need to split it into another PR. Thanks for doing the refactor.

Member

cvrebert commented Jan 28, 2014

@axemclion I'm not blocked on it, so there's no need to split it into another PR. Thanks for doing the refactor.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Jan 28, 2014

@cvrebert - Done with your comments. The final 2 remaining things are

  1. Set up couchDB account. Cloudant or IrisCouch?
  2. Add perf to travis

@cvrebert - Done with your comments. The final 2 remaining things are

  1. Set up couchDB account. Cloudant or IrisCouch?
  2. Add perf to travis
@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Jan 28, 2014

Member

@axemclion Yup. We're blocked on @mdo creating the account, which probably won't happen until after v3.1.0 is released later this week.

Member

cvrebert commented Jan 28, 2014

@axemclion Yup. We're blocked on @mdo creating the account, which probably won't happen until after v3.1.0 is released later this week.

@mdo

This comment has been minimized.

Show comment
Hide comment
@mdo

mdo Feb 5, 2014

Member

Created an IrisCouch account, I think. There's no real account it seems—just links from an email. Little odd, but okay. Read to rock when you are: https://bootstrap.iriscouch.com/.

Member

mdo commented Feb 5, 2014

Created an IrisCouch account, I think. There's no real account it seems—just links from an email. Little odd, but okay. Read to rock when you are: https://bootstrap.iriscouch.com/.

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Feb 5, 2014

@mdo, Made the changes, and running the first set of tests to collect the numbers.

The perf dashboard is available at http://bootstrap.iriscouch.com/perf/_design/site/index.html and should have more data as the tests are run over commits.

Should this be made a part of travis runs ? Note that given the number of components, it is taking a good amount of time to generate data for all components.

@mdo, Made the changes, and running the first set of tests to collect the numbers.

The perf dashboard is available at http://bootstrap.iriscouch.com/perf/_design/site/index.html and should have more data as the tests are run over commits.

Should this be made a part of travis runs ? Note that given the number of components, it is taking a good amount of time to generate data for all components.

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Feb 5, 2014

Member

@axemclion I presume we need to do some additional setup to secure the DB against unauthorized creates+updates+deletes?

Member

cvrebert commented Feb 5, 2014

@axemclion I presume we need to do some additional setup to secure the DB against unauthorized creates+updates+deletes?

@cvrebert

This comment has been minimized.

Show comment
Hide comment
@cvrebert

cvrebert Feb 5, 2014

Member

@axemclion About how long does a run take?

Member

cvrebert commented Feb 5, 2014

@axemclion About how long does a run take?

@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Feb 5, 2014

@cvrebert Yes, right now the iriscouch is open to all. That is how I was able to write the design documents, and add data .Ideally, anyone should be allowed to add perf data, but only twbs should have admin access.

The tests took around 15 mins, but note that I do not have any parallel tests on Sauce and I exhausted all my Mac minutes.

@cvrebert Yes, right now the iriscouch is open to all. That is how I was able to write the design documents, and add data .Ideally, anyone should be allowed to add perf data, but only twbs should have admin access.

The tests took around 15 mins, but note that I do not have any parallel tests on Sauce and I exhausted all my Mac minutes.

@cvrebert cvrebert merged commit 81d9a46 into twbs:master Feb 6, 2014

1 check passed

default The Travis CI build passed
Details
@axemclion

This comment has been minimized.

Show comment
Hide comment
@axemclion

axemclion Feb 6, 2014

@cvrebert Were you able to run this on your machine ? How should we go about integrating it with travis?

@cvrebert Were you able to run this on your machine ? How should we go about integrating it with travis?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment