core(metrics): move FMP to computed artifact #4951

patrickhulce · 2018-04-10T22:40:46Z

progress on #4872

moves FMP and also moves all the junk that used to be in FMP to be in a metrics.js audit that will collect all metrics

paulirish

mostly good! just a few ideas.

paulirish · 2018-04-10T23:19:36Z

lighthouse-core/audits/metrics.js

+
+    for (const [traceEventName, timing] of Object.entries(traceOfTab.timings)) {
+      const uppercased = traceEventName.slice(0, 1).toUpperCase() + traceEventName.slice(1);
+      const metricName = `raw${uppercased}`;


raw => trace

paulirish · 2018-04-10T23:20:44Z

lighthouse-core/audits/metrics.js

+      metrics.push(Object.assign({metricName}, values));
+    }
+
+    for (const [traceEventName, timing] of Object.entries(traceOfTab.timings)) {


// Include all timings from traceOftab

paulirish · 2018-04-10T23:21:24Z

lighthouse-core/audits/metrics.js

+    const metricsMap = {firstContentfulPaint, firstMeaningfulPaint, timeToInteractive};
+
+    const metrics = [];
+    for (const [metricName, values] of Object.entries(metricsMap)) {


// Include these specific metrics

paulirish · 2018-04-10T23:22:16Z

lighthouse-core/audits/metrics.js

+    const firstContentfulPaint = await artifacts.requestFirstContentfulPaint(metricComputationData);
+    const firstMeaningfulPaint = await artifacts.requestFirstMeaningfulPaint(metricComputationData);
+    const timeToInteractive = await artifacts.requestConsistentlyInteractive(metricComputationData);
+    const metricsMap = {firstContentfulPaint, firstMeaningfulPaint, timeToInteractive};


move this metricsMap line to right above the for loop. move the const metrics line up higher.

paulirish · 2018-04-10T23:25:41Z

lighthouse-core/lib/traces/pwmetrics-events.js

@@ -24,6 +24,16 @@ function safeGet(object, path) {
  return object;
 }

+function findValueInMetricsAuditFn(metricName, valueName) {


valueName => timingOrTimestamp

patrickhulce · 2018-04-10T23:41:43Z

😷 🗡 💨(that's supposed to be a ninja that's addressing feedback)

paulirish

wfm

brendankenny

quuuuuestions for you

brendankenny · 2018-04-11T22:07:14Z

lighthouse-core/test/results/sample_v2.json

        "items": [
          {
-            "timing": 493,
-            "timestamp": 185603812639.80002,
+            "timing": 479,


why the changes here? Since regenerating from the same source, shouldn't they stay the same?

oh good catch!! this was actually a bug introduced back with TTCI changes which fail when scoring params are not given in our hacky TTCI.audit(artifacts) approach

brendankenny · 2018-04-11T22:09:30Z

lighthouse-core/test/lib/traces/pwmetrics-events-test.js

@@ -8,8 +8,8 @@
 const Metrics = require('../../../lib/traces/pwmetrics-events');
 const assert = require('assert');

-const dbwTrace = require('../../fixtures/traces/dbw_tester.json');
-const dbwResults = require('../../fixtures/dbw_tester-perf-results.json');
+const dbwTrace = require('../../results/artifacts/defaultPass.trace.json');


rename these variables (and in asset-saver) to exampleTrace and sampleV2Results or something like that?

hm, why? they're still on the dbw tester so the names are still relevant :)

ha, somehow I never noticed that's where they were from. Never mind :)

brendankenny · 2018-04-11T22:11:44Z

lighthouse-core/audits/metrics.js

+
+    // Include all timestamps of interest from trace of tab
+    for (const [traceEventName, timing] of Object.entries(traceOfTab.timings)) {
+      const uppercased = traceEventName.slice(0, 1).toUpperCase() + traceEventName.slice(1);


lol no hope of inferring type

sure there is! :D

string -> string -> string + string -> string => string

ha, I meant of metrics :P

oh lol, yes :)

brendankenny · 2018-04-11T22:14:42Z

lighthouse-core/test/audits/first-meaningful-paint-test.js

        assert.equal(result.displayValue, '530\xa0ms');
-        assert.equal(result.rawValue, 529.9);


can we preserve these tests (put them in metrics-test.js, maybe?) Many of them were hard-won results from problematic traces, and having them in our tests make any future deviations from our current trace interpretations more obviously deliberate

by "these tests" do you mean the assertions on the correct timestamp/navstart/fcp rather than just the FMP? AFAICT I didn't regress coverage of any test scenarios

do you mean the assertions on the correct timestamp/navstart/fcp rather than just the FMP?

yes

AFAICT I didn't regress coverage of any test scenarios

is there elsewhere we test for navstart/fCP/fmpFellBack for these problematic traces (lateTracingStartedTrace, badNavStartTrace, preactTrace, etc) specifically?

yes at the level where the problematic logic actually was, trace-of-tab :)
https://github.com/AkshayIyer12/lighthouse/blob/f313424de4f2284392aa758d28416c51f7f59dd2/lighthouse-core/test/gather/computed/trace-of-tab-test.js#L13-L18

ha, well that's perfect then

brendankenny · 2018-04-11T22:17:18Z

lighthouse-core/test/audits/first-meaningful-paint-test.js

  }, computedArtifacts);
 }

 /* eslint-env mocha */
 describe('Performance: first-meaningful-paint audit', () => {
+  const context = {options, settings: {throttlingMethod: 'provided'}};


should these tests include simulated runs or should most of these be kicked up to trace-of-tab tests?

I'm keeping all of the simulated/provided/devtools test distinctions inside the computed artifact versions and keeping all the audit tests on provided to keep things simpler there. does that sound good to you?

I mean, it's not terrible, and reduces churn. I was just thinking if we wrote them today really this unit test would probably just verify that it can get results for the various throttling methods, while the nuts and bolts would be relegated to the computed artifact/lantern computed artifact unit tests.

WDYT? This can also just be some legacy we live with and some day maybe clean up our unit test organization

done, might as well have a real fresh start this time around :D

patrickhulce · 2018-04-12T15:22:13Z

lighthouse-core/audits/screenshot-thumbnails.js

-      TTFI.audit(artifacts).catch(() => ({rawValue: 0})),
-      TTCI.audit(artifacts).catch(() => ({rawValue: 0})),
+      TTFI.audit(artifacts, context).catch(() => ({rawValue: 0})),
+      TTCI.audit(artifacts, context).catch(() => ({rawValue: 0})),


this was the bug you caught @brendankenny

I'll do a followup PR to make these use computed artifacts instead 👍

🤔 I wonder if we should have consider the sample LHR to be a golden file that all changes to must be approved in review to catch these sooner

I wonder if we should have consider the sample LHR to be a golden file that all changes to must be approved in review to catch these sooner

that's a good idea. I wonder if there's some way to enforce that? Maybe have Travis regenerate sample_v2.json and diff with the checked in version? cc @paulirish

yeah that was my thought

yarn update:sample-json git diff path/to/sample_v2.json --exit-code

peek at #4956

patrickhulce · 2018-04-12T17:10:12Z

just as an explicit notice for the purposes of #4934, I'm currently waiting response to review comments :)

brendankenny · 2018-04-12T18:02:33Z

lighthouse-core/gather/computed/trace-of-tab.js

@@ -138,7 +138,8 @@ class TraceOfTab extends ComputedArtifact {
      firstPaintEvt: firstPaint,
      firstContentfulPaintEvt: firstContentfulPaint,
      firstMeaningfulPaintEvt: firstMeaningfulPaint,
-      onLoadEvt: onLoad,
+      loadEvt: load,


will you add the new metrics audit and onLoadEvt -> loadEvt to #4333. We should probably have a list and we haven't been adding "breaking" to our PR titles (oh well)

brendankenny · 2018-04-12T18:07:32Z

lighthouse-core/test/audits/first-meaningful-paint-test.js

  }, computedArtifacts);
 }

 /* eslint-env mocha */
 describe('Performance: first-meaningful-paint audit', () => {
+  const context = {options, settings: {throttlingMethod: 'provided'}};


I mean, it's not terrible, and reduces churn. I was just thinking if we wrote them today really this unit test would probably just verify that it can get results for the various throttling methods, while the nuts and bolts would be relegated to the computed artifact/lantern computed artifact unit tests.

WDYT? This can also just be some legacy we live with and some day maybe clean up our unit test organization

brendankenny · 2018-04-12T18:09:22Z

lighthouse-core/test/audits/first-meaningful-paint-test.js

        assert.equal(result.displayValue, '530\xa0ms');
-        assert.equal(result.rawValue, 529.9);


do you mean the assertions on the correct timestamp/navstart/fcp rather than just the FMP?

yes

AFAICT I didn't regress coverage of any test scenarios

is there elsewhere we test for navstart/fCP/fmpFellBack for these problematic traces (lateTracingStartedTrace, badNavStartTrace, preactTrace, etc) specifically?

patrickhulce · 2018-04-12T18:46:05Z

alright @brendankenny I think all feedback has been addressed

brendankenny

One last request *ducks*

But otherwise LGTM!

brendankenny · 2018-04-12T18:57:46Z

lighthouse-core/test/audits/first-meaningful-paint-test.js

+      traces: {[Audit.DEFAULT_PASS]: {traceEvents}},
+      devtoolsLogs: {[Audit.DEFAULT_PASS]: []},
+    }, computedArtifacts);
+    const context = {options, settings: {throttlingMethod: 'provided'}};


maybe also a simulated test (and devtools? Even though it's currently the same execution path as this) so it checks that it can also successfully call through?

brendankenny · 2018-04-12T20:10:49Z

💥 💥

core(metrics): move FMP to computed artifact

192e659

patrickhulce requested review from brendankenny and paulirish as code owners April 10, 2018 22:40

devtools-bot added the waiting4reviewer label Apr 10, 2018

add metrics test

060179f

paulirish requested changes Apr 10, 2018

View reviewed changes

feedback

91611cc

paulirish approved these changes Apr 11, 2018

View reviewed changes

Merge branch 'master' into fmp_computed_art

63a2f22

patrickhulce added the I/O-Expedite label Apr 11, 2018

brendankenny requested changes Apr 11, 2018

View reviewed changes

fix screenshot bug

339d05e

patrickhulce commented Apr 12, 2018

View reviewed changes

brendankenny reviewed Apr 12, 2018

View reviewed changes

move tests to computed artifact level

74724d4

brendankenny approved these changes Apr 12, 2018

View reviewed changes

paulirish mentioned this pull request Apr 12, 2018

tests: add CI test to confirm sample_v2.json is up to date #4956

Merged

add simulated test

e94a99e

patrickhulce merged commit e5719bd into master Apr 12, 2018

patrickhulce deleted the fmp_computed_art branch April 12, 2018 20:35

patrickhulce mentioned this pull request Apr 13, 2018

core(screenshots): align filmstrip to observed metrics #4965

Merged

paulirish removed the waiting4reviewer label Apr 30, 2018

		assert.equal(result.displayValue, '530\xa0ms');
		assert.equal(result.rawValue, 529.9);

core(metrics): move FMP to computed artifact #4951

core(metrics): move FMP to computed artifact #4951

Conversation

patrickhulce commented Apr 10, 2018

paulirish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Apr 10, 2018 • edited Loading

paulirish left a comment

Choose a reason for hiding this comment

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce Apr 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce Apr 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Apr 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Apr 12, 2018

brendankenny left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented Apr 12, 2018

patrickhulce commented Apr 10, 2018 •

edited

Loading

patrickhulce Apr 12, 2018 •

edited

Loading

patrickhulce Apr 12, 2018 •

edited

Loading