Plots: A/B screenshot viewer #2026

wwwillchen · 2017-04-18T02:25:38Z

Purpose

This enables the lighthouse team to better understand how changes to perf metrics (e.g. the algorithm for FMP) affect the perf results of real world sites.

Mode A: Aligned Screenshot Timeline

This enables easy head-to-head comparison between runs for the same site or even runs across different sites. To make the timeline less cluttered, I only show some of the screenshots.

Example:

Mode B: Non-Aligned Screenshot Timeline

This shows more detail than the other mode and shows every screenshot collected.

Example:

If this tool proves to be useful - we can work on making it easy to use for external users. For example, I can imagine a product manager be very interested in the Aligned Screenshot Timeline View for their company and top 3 competitors.

Closes part of #1980

patrickhulce · 2017-04-18T20:46:16Z

lighthouse-core/test/lib/asset-saver-test.js

+      const ssJSONFilename = 'the_file-0.screenshots.json';
+      const ssFileContents = fs.readFileSync(ssJSONFilename, 'utf8');
+      const expectedScreenshotContent = '"timestamp": 674089419.919,';
+      assert.ok(ssFileContents.includes(expectedScreenshotContent), 'unexpected screenshot json');


maybe we can be a bit stronger with our assertions here, maybe that it parses and it has the timestamp property?

benim sitem çok yavaş : (( bana yardım edermisiniz http://ankaraotokurtarici.com

patrickhulce · 2017-04-18T20:49:26Z

plots/ab-screenshot/analyze.js

+
+  const fcpTiming = getTiming('ttfcp');
+  const fmpTiming = getTiming('ttfmp');
+  const vc100Timing = getTiming('vc100');


can we get 85% too? not sure if it's in pwmetrics yet but we should since we'll be switching up TTI soon enough :)

patrickhulce · 2017-04-18T20:52:51Z

plots/ab-screenshot/analyze.js

+}
+
+/**
+ * Analyzes the screenshots for the first run of a particular site.


hmmm I guess the order doesn't really matter then if it's not an aggregation of any sort? I guess we could just pull any random folder huh...sorting just for consistency?

Yeah, it doesn't really matter but it's nice to know it's always the first run when you're looking up the artifacts for a particular run (e.g. open the trace file for a fishy run).

brendankenny · 2017-04-18T21:22:02Z

love being able to compare, and the screenshots mode makes this amazing for a very visual look at what's going on. For a major switch (like a site going non-PWA to PWA) or

For example, I can imagine a product manager be very interested in the Aligned Screenshot Timeline View for their company and top 3 competitors.

this will be awesome.

Do you have plans for comparing two distributions of runs, too? I think for more minor changes (as opposed to switching to a totally new TTI) that's going to give a better picture of what's going on than comparing just two runs at a time.

Also did you see #2011? Pretty please :)

wwwillchen · 2017-04-18T22:57:01Z

@brendankenny I have it on my to-do list to create a box plot which will show distributions of runs for a particular site over time (e.g. different lighthouse versions). My sense is that this will be more valuable for the Lighthouse team in the longer-term (i.e. post I/O) when the metrics have settled - but if this is a high priority for the team, I can work on it soon.

wwwillchen

Please take another look.

wwwillchen · 2017-04-19T00:36:35Z

lighthouse-core/test/lib/asset-saver-test.js

+      const ssJSONFilename = 'the_file-0.screenshots.json';
+      const ssFileContents = fs.readFileSync(ssJSONFilename, 'utf8');
+      const expectedScreenshotContent = '"timestamp": 674089419.919,';
+      assert.ok(ssFileContents.includes(expectedScreenshotContent), 'unexpected screenshot json');


wwwillchen · 2017-04-19T00:37:51Z

plots/ab-screenshot/analyze.js

+}
+
+/**
+ * Analyzes the screenshots for the first run of a particular site.


Yeah, it doesn't really matter but it's nice to know it's always the first run when you're looking up the artifacts for a particular run (e.g. open the trace file for a fishy run).

wwwillchen · 2017-04-19T19:20:20Z

plots/ab-screenshot/analyze.js

+
+  const fcpTiming = getTiming('ttfcp');
+  const fmpTiming = getTiming('ttfmp');
+  const vc100Timing = getTiming('vc100');


patrickhulce

a-ok by me

patrickhulce · 2017-04-20T21:00:29Z

lighthouse-core/lib/asset-saver.js

+      fs.writeFileSync(screenshotsHTMLFilename, data.screenshotsHTML);
+      log.log('screenshots saved to disk', screenshotsHTMLFilename);
+
+      const screenshotsJSONFilename = `${pathWithBasename}-${index}.screenshots.json`;


make sure to add this to gitignore too :)

patrickhulce · 2017-04-20T21:07:21Z

plots/ab-screenshot/README.md

+$ node analyze.js -a ../out-first -b ../out-second
+
+# Use the screenshot viewer
+# Open index.html in browser


would be dope to use opn and have analyze just open the graphs once it's done :)

wwwillchen · 2017-04-20T22:17:51Z

Re-opening PR to retrigger Travis job

paulirish

awesome stuff. I gave you a PR with a few more goodies that really makes this feature pretty hot:
wwwillchen#1

with that and the other stuff in here sorted, i'm feelin p good!

paulirish · 2017-04-21T07:12:34Z

plots/ab-screenshot/analyze.js

+ * lighthouse can affect perf results in real-world sites.
+ *
+ * Example usage:
+ * node analyze.js -a ./out-first -b ./out-second


i'm thinking there is also the usecase of...

i want to compare two traces that were captured in the same measure run.

so like:

node analyze -a ./out/www.espn.com-./0/ -b ./out/www.espn.com-./2/

however I suppose your approach here is to show multiple sites across these two runs.
should we support both ways?
i'm down with whichever.

I originally didn't think about this use case because then you're comparing two traces with the same configuration (e.g. lighthouse, chrome version) and in that case, I think the end goal could be to either A) visualize the variance between runs when all factors are held constant or B) compare multiple sites (e.g. a PM wants to compare their site vs. 3 competitors).

These seem like nice to have cases but not essential. Let's defer them until later?

Let's defer them until later?

k

paulirish · 2017-04-21T07:21:11Z

plots/ab-screenshot/analyze.js

+ * @return {!SingleRunScreenshots}
+ */
+function analyzeSingleRunScreenshots(sitePath) {
+  const runDir = sortRunFolders(fs.readdirSync(sitePath))[0];


this readdirSync returned a .DS_Store file which ended up setting runPath to NaN. :)

Oy - fixed!

Arkadaşlar ben siteme bunu nasıl ekleyebilirim ? http://ankaraotokurtarici.com
Lütfen bana yardım edermisiniz
Help me pls

wwwillchen

@paulirish ptal.

wwwillchen · 2017-04-20T21:55:30Z

lighthouse-core/lib/asset-saver.js

+      fs.writeFileSync(screenshotsHTMLFilename, data.screenshotsHTML);
+      log.log('screenshots saved to disk', screenshotsHTMLFilename);
+
+      const screenshotsJSONFilename = `${pathWithBasename}-${index}.screenshots.json`;


wwwillchen · 2017-04-25T18:28:09Z

plots/ab-screenshot/analyze.js

+ * lighthouse can affect perf results in real-world sites.
+ *
+ * Example usage:
+ * node analyze.js -a ./out-first -b ./out-second


I originally didn't think about this use case because then you're comparing two traces with the same configuration (e.g. lighthouse, chrome version) and in that case, I think the end goal could be to either A) visualize the variance between runs when all factors are held constant or B) compare multiple sites (e.g. a PM wants to compare their site vs. 3 competitors).

These seem like nice to have cases but not essential. Let's defer them until later?

wwwillchen · 2017-04-25T18:31:18Z

plots/ab-screenshot/analyze.js

+ * @return {!SingleRunScreenshots}
+ */
+function analyzeSingleRunScreenshots(sitePath) {
+  const runDir = sortRunFolders(fs.readdirSync(sitePath))[0];


Oy - fixed!

paulirish · 2017-04-25T19:44:18Z

@wwwillchen wdyt about wwwillchen#1 ?

plots: add kbd nav, more persistent popover.

googlebot · 2017-04-25T20:58:30Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored by someone other than the pull request submitter. We need to confirm that they're okay with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this state. It's up to you to confirm consent of the commit author(s) and merge this pull request when appropriate.

wwwillchen · 2017-04-25T21:08:22Z

@paulirish nice! merged it. Dunno why I didn't see it the first time 😎

wwwillchen added 4 commits April 17, 2017 18:04

Plots: a/b screenshot viewer

bf68461

prettier

55e6f72

fix eslint issues

aaabdae

update asset-saver-test

8b97da4

patrickhulce reviewed Apr 18, 2017

View reviewed changes

address PR feedback

1cbfd6c

wwwillchen commented Apr 19, 2017

View reviewed changes

patrickhulce approved these changes Apr 20, 2017

View reviewed changes

wwwillchen added 4 commits April 20, 2017 14:38

address PR feedback

7e7c469

update readme

12c4de9

update analyze

69266c0

update readme

13cb5b0

wwwillchen closed this Apr 20, 2017

wwwillchen reopened this Apr 20, 2017

plots: add kbd nav, more persistent popover.

39409c3

paulirish requested changes Apr 21, 2017

View reviewed changes

fix .DS_store issue

ced6884

wwwillchen commented Apr 25, 2017

View reviewed changes

Merge pull request #1 from GoogleChrome/ab-tweaks

957cfe4

plots: add kbd nav, more persistent popover.

paulirish approved these changes Apr 25, 2017

View reviewed changes

paulirish merged commit b2eaa08 into GoogleChrome:master Apr 25, 2017

paulirish pushed a commit that referenced this pull request Apr 26, 2017

Plots: A/B screenshot viewer (#2026)

0d67681

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plots: A/B screenshot viewer #2026

Plots: A/B screenshot viewer #2026

wwwillchen commented Apr 18, 2017

patrickhulce Apr 18, 2017

wwwillchen Apr 19, 2017

otoyolyardimm Sep 26, 2020

patrickhulce Apr 18, 2017

wwwillchen Apr 19, 2017

patrickhulce Apr 18, 2017

wwwillchen Apr 19, 2017

brendankenny commented Apr 18, 2017

wwwillchen commented Apr 18, 2017

wwwillchen left a comment

wwwillchen Apr 19, 2017

wwwillchen Apr 19, 2017

wwwillchen Apr 19, 2017

patrickhulce left a comment

patrickhulce Apr 20, 2017

wwwillchen Apr 20, 2017

patrickhulce Apr 20, 2017

wwwillchen commented Apr 20, 2017

paulirish left a comment

paulirish Apr 21, 2017

wwwillchen Apr 25, 2017

paulirish Apr 25, 2017

paulirish Apr 21, 2017

wwwillchen Apr 25, 2017

otoyolyardimm Sep 26, 2020

wwwillchen left a comment

wwwillchen Apr 20, 2017

wwwillchen Apr 25, 2017

wwwillchen Apr 25, 2017

paulirish commented Apr 25, 2017

googlebot commented Apr 25, 2017

wwwillchen commented Apr 25, 2017

Plots: A/B screenshot viewer #2026

Plots: A/B screenshot viewer #2026

Conversation

wwwillchen commented Apr 18, 2017

Purpose

Mode A: Aligned Screenshot Timeline

Mode B: Non-Aligned Screenshot Timeline

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented Apr 18, 2017

wwwillchen commented Apr 18, 2017

wwwillchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwwillchen commented Apr 20, 2017

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wwwillchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulirish commented Apr 25, 2017

googlebot commented Apr 25, 2017

wwwillchen commented Apr 25, 2017