Plots: make measure script more flexible (CLI args) #2183

wwwillchen · 2017-05-08T21:34:21Z

Add flags, help docs, and making it better all around.

patrickhulce · 2017-05-08T22:13:36Z

lighthouse-core/lib/emulation.js

@@ -41,10 +41,13 @@ const NEXUS5X_USERAGENT = {
    '(KHTML, like Gecko) Chrome/59.0.3033.0 Mobile Safari/537.36'
 };

+const LATENCY_FACTOR = 3.75;
+const THROUGHPUT_FACTOR = 1.1;


nit: should we model both of these as multipliers so it's just multiply for both?

+1.

it's equal to throughput * 90/99 but you might as well simplify it to throughput * 0.9

brendankenny · 2017-05-08T22:19:25Z

plots/measure.js

+};
+console.log('Running lighthouse with flag\n disableNetworkThrottling: ', FLAGS.disableNetworkThrottling);
+
+const SUBSET = [


running plots/ this weekend, it was very handy to split this to loading from file so I could quickly switch between a couple of different sets of test sites. If you're adding yargs anyways, it would be great to support loading a list of URLs from a path passed in as a CLI option :)

brendankenny · 2017-05-08T22:19:47Z

plots/measure.js

@@ -150,20 +178,30 @@ main();
 * Returns a promise chain that analyzes all the sites n times.
 * @return {!Promise}
 */
-function runAnalysis() {
+function runAnalysisWithNewChromeInstances() {


brendankenny · 2017-05-15T21:26:08Z

is this ready to review? And will need a rebase with recent changes to chromeLauncher

brendankenny · 2017-05-15T21:41:40Z

is this ready to review?

was that a yes? :)

wwwillchen · 2017-05-22T18:29:54Z

ping @brendankenny for a review :)

brendankenny · 2017-05-23T00:16:25Z

will need a rebase :)

paulirish · 2017-05-24T05:39:32Z

plots/measure.js

-  .catch(err => console.error(err))
-  .then(() => launcher.kill());
+  if (REUSE_CHROME) {
+    ChromeLauncher.launch({port: 9222}).then(launcher => {


I've been running multiple plots at a time recently.. Launcher is getting a unique port by default, but you have to get it off launcher.port and pass into lighthouse().. Here's what I did today:

Can you support this functionality? You could just pass it along like this or create measure instances that hold this state.. wdyt?

paulirish · 2017-05-24T05:40:29Z

plots/measure.js

+// Running it n + 1 times if the first run is deliberately ignored
+// because it has different perf characteristics from subsequent runs
+// (e.g. DNS cache which can't be easily reset between runs)
+const NUMBER_OF_RUNS = (CUSTOM_NUMBER_OF_RUNS || 20) + (KEEP_FIRST_RUN ? 0 : 1);


doesn't seem like an uppercase const as its super conditional.

paulirish · 2017-05-24T05:54:34Z

plots/measure.js

@@ -31,9 +59,29 @@ const ChromeLauncher = require('../chrome-launcher/chrome-launcher.js');
 const Printer = require('../lighthouse-cli/printer');
 const assetSaver = require('../lighthouse-core/lib/asset-saver.js');

-const NUMBER_OF_RUNS = 20;
+const DISABLE_DEVICE_EMULATION = args['disable-device-emulation'];


tbh seems kinda odd to establish these all as uppercase consts as they are dependent on CLI input.

yeah - I removed them and just inlined their usage

paulirish · 2017-05-24T05:55:49Z

plots/measure.js

+  }
+
+  if (SUBSET) {
+    return [


how about we just slice the top of the full sites list instead of repeating here?

It's not DRY but I wanted to both have a set of sites that were useful for cpu/network analysis (basically the most visible / meaningful sites) and preserve the original order of the sites list since it explains in comments how they were sourced.

paulirish · 2017-05-24T05:57:41Z

plots/measure.js

+// Running it n + 1 times if the first run is deliberately ignored
+// because it has different perf characteristics from subsequent runs
+// (e.g. DNS cache which can't be easily reset between runs)
+const NUMBER_OF_RUNS = (CUSTOM_NUMBER_OF_RUNS || 20) + (KEEP_FIRST_RUN ? 0 : 1);


it's unexpected that number_of_runs == 1 will provide two runs for each site. Can we make sure the number here will match # of actual runs done?

OK - I updated it so there's no +1 run. I added a check in main so users don't accidentally try to only do 1 run whose results are discard.

paulirish · 2017-05-24T06:00:07Z

plots/utils.js

+/**
+ * @param {string} filePath
+ */
+function removeRecursive(filePath) {


oy - yeah I was using it in a one-off script. I'll remove it.

…nto network-analysis

wwwillchen

Please take another look! Sorry this took a while :)

wwwillchen · 2017-05-30T18:17:51Z

plots/measure.js

@@ -31,9 +59,29 @@ const ChromeLauncher = require('../chrome-launcher/chrome-launcher.js');
 const Printer = require('../lighthouse-cli/printer');
 const assetSaver = require('../lighthouse-core/lib/asset-saver.js');

-const NUMBER_OF_RUNS = 20;
+const DISABLE_DEVICE_EMULATION = args['disable-device-emulation'];


yeah - I removed them and just inlined their usage

wwwillchen · 2017-05-30T18:18:38Z

plots/measure.js

+// Running it n + 1 times if the first run is deliberately ignored
+// because it has different perf characteristics from subsequent runs
+// (e.g. DNS cache which can't be easily reset between runs)
+const NUMBER_OF_RUNS = (CUSTOM_NUMBER_OF_RUNS || 20) + (KEEP_FIRST_RUN ? 0 : 1);


wwwillchen · 2017-05-30T18:21:06Z

plots/measure.js

+// Running it n + 1 times if the first run is deliberately ignored
+// because it has different perf characteristics from subsequent runs
+// (e.g. DNS cache which can't be easily reset between runs)
+const NUMBER_OF_RUNS = (CUSTOM_NUMBER_OF_RUNS || 20) + (KEEP_FIRST_RUN ? 0 : 1);


OK - I updated it so there's no +1 run. I added a check in main so users don't accidentally try to only do 1 run whose results are discard.

wwwillchen · 2017-05-30T18:27:14Z

plots/measure.js

+  }
+
+  if (SUBSET) {
+    return [


It's not DRY but I wanted to both have a set of sites that were useful for cpu/network analysis (basically the most visible / meaningful sites) and preserve the original order of the sites list since it explains in comments how they were sourced.

wwwillchen · 2017-05-30T18:30:58Z

plots/measure.js

-  .catch(err => console.error(err))
-  .then(() => launcher.kill());
+  if (REUSE_CHROME) {
+    ChromeLauncher.launch({port: 9222}).then(launcher => {


wwwillchen · 2017-05-30T18:31:32Z

plots/utils.js

+/**
+ * @param {string} filePath
+ */
+function removeRecursive(filePath) {


oy - yeah I was using it in a one-off script. I'll remove it.

paulirish · 2017-05-30T19:08:26Z

plots/measure.js

- * then open a fresh Chrome instance for each run.
- */
+function getUrls() {
+  if (args['sites-path']) {


since we already have this, why not toss the default SITES into a file and have sites-path default to that one.

paulirish · 2017-05-30T19:08:59Z

plots/measure.js

+  }
+
+  if (args['subset']) {
+    return [


and it seems like we could just extract this into a diff sites_subset.js file?

paulirish · 2017-05-30T19:10:57Z

plots/measure.js

-function analyzeWithLighthouse(url, outputPath, assetsPath, {ignoreRun}) {
-  const flags = {output: 'json'};
+function analyzeWithLighthouse(launcher, url, outputPath, assetsPath, {ignoreRun}) {
+  const flags = {


if ignoreRun is true then there's no reason to have throttling on. I'd force disable cpu & network throttling in these cases.

wwwillchen

yo @paulirish - updated per your comments.

wwwillchen · 2017-05-30T22:04:22Z

plots/measure.js

-function analyzeWithLighthouse(url, outputPath, assetsPath, {ignoreRun}) {
-  const flags = {output: 'json'};
+function analyzeWithLighthouse(launcher, url, outputPath, assetsPath, {ignoreRun}) {
+  const flags = {


paulirish · 2017-05-30T22:10:32Z

plots/sites.js

+'use strict';
+
+/* eslint-disable no-unused-expressions */
+[


module.exports = [ ...

paulirish · 2017-05-30T22:10:53Z

plots/sites_subset.js

+'use strict';
+
+/* eslint-disable no-unused-expressions */
+[


module.exports = [ ...

paulirish · 2017-05-30T22:15:07Z

plots/measure.js

-  'https://ask.com',
-  'https://bbc.com'
-];
+const keepFirstRun = args['keep-first-run'] || !args['reuse-chrome'];


you can use camelcase instead of this square bracket dance.

args.keepFirstRun etc. yargs does it for you.

paulirish · 2017-05-30T22:15:26Z

plots/measure.js

+    return eval(fs.readFileSync(path.resolve(__dirname, 'sites_subset.js'), 'utf-8'));
+  }
+
+  return eval(fs.readFileSync(path.resolve(__dirname, 'sites.js'), 'utf-8'));


instead. you can set the default on yargs:

.default('sites-path', 'sites.js')

paulirish · 2017-05-30T22:15:49Z

plots/measure.js

+  }
+
+  if (args['sites-path']) {
+    const sitesPath = path.resolve(__dirname, args['sites-path']);


if (args.sitesPath) { return require(path.resolve(__dirname, args.sitesPath)); }

Yeah - this is nice. I did it the more verbose way because the downside is require() will hold onto these files indefinitely in node's module cache. It's definitely OK now since the sites list are needed for the entire lifetime of the script but if we start loading a lot of different sites files, we'll need to watch out for heap usage.

wwwillchen · 2017-05-30T23:43:34Z

@paulirish - please take another look.

paulirish

lgtm % nits

thanks! excited about this

paulirish · 2017-05-30T23:36:50Z

plots/measure.js

+  })
+  .group(
+    ['disable-device-emulation', 'disable-cpu-throttling', 'disable-network-throttling'],
+    'Chrome DevTools settings:')


let's call this Lighthouse settings:

paulirish · 2017-05-30T23:38:33Z

plots/measure.js

@@ -17,11 +17,40 @@
 'use strict';

 /* eslint-disable no-console */
-
+const fs = require('fs');


no longer neccessary

paulirish · 2017-05-30T23:41:43Z

plots/measure.js

-  'https://bbc.com'
-];
+const keepFirstRun = args.keepFirstRun || !args.reuseChrome;
+const numberOfRuns = args.n || 3;


move to yargs default

paulirish · 2017-05-31T00:09:33Z

plots/plot-per-site.html

+  <script src="https://cdn.plot.ly/plotly-1.25.2.min.js"></script>
+</head>
+
+<body>


add this page to the <nav> of the other two html files?

paulirish · 2017-05-31T00:10:15Z

plots/plot-per-site.html

@@ -0,0 +1,29 @@
+<!--


rename this to bars-per-site ?

paulirish · 2017-05-31T00:11:45Z

plots/measure.js

 const path = require('path');
 const parseURL = require('url').parse;

 const mkdirp = require('mkdirp');
+const args = require('yargs')


with the new style of invoking, can you update the readme?

to me, it makes more sense to use the node script.js invocation rather than the npm run one now that we have a CLI. but no big deal.

wwwillchen

Fixed it up!

wwwillchen · 2017-05-31T00:35:53Z

plots/plot-per-site.html

+  <script src="https://cdn.plot.ly/plotly-1.25.2.min.js"></script>
+</head>
+
+<body>


paulirish · 2017-05-31T01:23:52Z

plots/README.md

 # Run lighthouse to collect metrics data
-$ yarn measure
+$ node measure.js


you got a whole CLI for measure.js now!. at the very least you should paste the --help output into the readme

paulirish

Nice stuff all around! Thanks for the back 'n forth!

📉 📈 📊

wwwillchen and others added 11 commits April 27, 2017 13:47

update gitignore

2b7da18

add flags for mobile measurement

5577cf7

top 18 sites

9d11132

remove airbnb - not reliable

4655cbe

adjust measure.js

d38d070

define SUBSET --subset flag

a7eaf8e

fixup measure

adb59a9

make screenshot/analyze.js handle missing results

57b763e

adjustment factors

506dc30

cherry-picked plot-per-site

b8c4577

wrap

ed44705

patrickhulce reviewed May 8, 2017

View reviewed changes

brendankenny reviewed May 8, 2017

View reviewed changes

fix comments

39f6b4c

wwwillchen mentioned this pull request May 12, 2017

Update network throttling to simulate more realistic network conditions #2238

Merged

wwwillchen added 2 commits May 15, 2017 11:58

update

5338f89

undo emulation.js

d6cabdb

wwwillchen changed the title ~~Network analysis - WIP~~ Plots: make measure script more flexible (CLI args) May 15, 2017

wwwillchen added 2 commits May 15, 2017 13:46

small improvements

52ef9ef

merge

7f214eb

fix launcher

72ae90d

brendankenny mentioned this pull request May 17, 2017

give port to plots for new chromelauncher #2284

Merged

paulirish requested changes May 24, 2017

View reviewed changes

wwwillchen added 2 commits May 30, 2017 11:32

paul irish fb

963ec9e

Merge branch 'master' of https://github.com/GoogleChrome/lighthouse i…

da6581b

…nto network-analysis

wwwillchen added 4 commits May 30, 2017 11:35

nit

1447442

fixup measure

922d7ab

fixup ab-screenshot

f637160

fixups

2170a9e

wwwillchen commented May 30, 2017

View reviewed changes

paulirish reviewed May 30, 2017

View reviewed changes

wwwillchen added 2 commits May 30, 2017 12:12

make eslint happy

3d4c291

paul fb

cc59d4d

wwwillchen commented May 30, 2017

View reviewed changes

paulirish reviewed May 30, 2017

View reviewed changes

wwwillchen added 2 commits May 30, 2017 16:23

paul fb

f50a21f

fixup

8555129

paulirish approved these changes May 31, 2017

View reviewed changes

wwwillchen added 2 commits May 30, 2017 17:32

paul fb

e93c300

add links

92a927a

wwwillchen commented May 31, 2017

View reviewed changes

paulirish reviewed May 31, 2017

View reviewed changes

spruce up readme

ebdea6e

paulirish approved these changes May 31, 2017

View reviewed changes

paulirish merged commit 3f7e5a1 into GoogleChrome:master May 31, 2017

Plots: make measure script more flexible (CLI args) #2183

Plots: make measure script more flexible (CLI args) #2183

Conversation

wwwillchen commented May 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented May 15, 2017

brendankenny commented May 15, 2017

wwwillchen commented May 22, 2017

brendankenny commented May 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwwillchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwwillchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwwillchen May 30, 2017 • edited Loading

Choose a reason for hiding this comment

wwwillchen commented May 30, 2017

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwwillchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish left a comment

Choose a reason for hiding this comment

wwwillchen commented May 8, 2017 •

edited

Loading

wwwillchen May 30, 2017 •

edited

Loading