core(gather-runner): use final document when reporting non-HTML error #11620

bradfrosty · 2020-11-02T17:44:44Z

Summary

Fixes a bug where non-HTML page load error would be reported for sites that redirect and resolve to a valid HTML document. The main document's mime type would be checked instead of the "final" document resolved from the redirect.

This change is needed because previously, false positive errors may be reported for valid pages that ultimately respond with HTML via redirects.

Related Issues/PRs

Fixes #11585
Introduced in #11042

patrickhulce

thanks very much @bfrost2893! 🎉 this is great navigating of core Lighthouse code, and excellent tests too 👍

patrickhulce · 2020-11-02T21:04:46Z

lighthouse-core/lib/dependency-graph/simulator/network-analyzer.js

+   * @param {LH.Artifacts.NetworkRequest|undefined} mainDocument
+   * @returns {LH.Artifacts.NetworkRequest}
+   */
+  static findFinalDocument(mainDocument) {


WDYT about...

Suggested change

static findFinalDocument(mainDocument) {

static resolveRedirects(request) {

if (!request) return undefined;

while (request.redirectDestination) request = request.redirectDestination;

return request;

findFinalDocument sounds more like a function that would search through an array of records :)

Definitely agree, given it takes a single NetworkRequest.

patrickhulce · 2020-11-02T22:04:52Z

lighthouse-core/gather/gather-runner.js

+    let finalRecord;
+    try {
+      finalRecord = NetworkAnalyzer.findFinalDocument(mainRecord);
+    } catch (_) {}
+


I don't think we need try/catch around this if we end up going with the resolveRedirects option that can handle undefined :)

Suggested change

let finalRecord;

try {

finalRecord = NetworkAnalyzer.findFinalDocument(mainRecord);

} catch (_) {}

// MIME Type is only set on the final redirected document request. Use this for the HTML check instead of root.

const finalRecord = NetworkAnalyzer.resolveRedirects(mainRecord);

patrickhulce · 2020-11-02T22:05:56Z

lighthouse-core/test/lib/dependency-graph/simulator/network-analyzer-test.js

@@ -385,4 +388,26 @@ describe('DependencyGraph/Simulator/NetworkAnalyzer', () => {
      assert.equal(mainDocument.url, 'https://www.example.com');
    });
  });
+
+  describe('#findFinalDocument', () => {
+    it('should find the final document without redirects', async () => {


one extra one for the undefined case?

bradfrosty · 2020-11-02T23:11:12Z

@patrickhulce thanks for the review! Addressed the changes and added the additional test.

One thing I'm not sure of is why smoke_3_ToT is failing. However it seems to be failing on other branches as well.

PS: Sorry for the name change, been itching to get rid of those numbers...

patrickhulce

LGTM! over to @Beytoven for the final seal of 👍

lighthouse-core/gather/gather-runner.js

Beytoven

Everything LGTM as well! Thanks.

patrickhulce · 2020-11-03T20:48:46Z

The test failure is a result of a Chrome change, we can land this once https://github.com/GoogleChrome/lighthouse/pull/11625/files lands 👍

Beytoven · 2020-11-04T16:00:04Z

The test failure is a result of a Chrome change, we can land this once https://github.com/GoogleChrome/lighthouse/pull/11625/files lands 👍

It has landed now.

patrickhulce · 2020-11-04T16:08:45Z

Awesome, @bradfrosty if you can rebase we should be able to get this landed 👍

This also handles the undefined case inside the fn, instead of outside.

bradfrosty · 2020-11-05T17:29:43Z

@patrickhulce @Beytoven any idea about how to fix the CLA check?

connorjclark · 2020-11-05T17:50:11Z

CLA is blocked on "patrick.hulce@gmail.com" for some reason .... We can just override. Yours went through. Thanks!

connorjclark · 2020-11-05T17:51:55Z

lighthouse-core/lib/dependency-graph/simulator/network-analyzer.js

+   * @param {LH.Artifacts.NetworkRequest|undefined} request
+   * @returns {LH.Artifacts.NetworkRequest|undefined}
+   */
+  static resolveRedirects(request) {


nit: @patrickhulce would this method be better on NetworkRequest?

Marginally yes, but we have network analysis somewhat scattered at the moment, so I don't think it makes a large difference.

connorjclark · 2020-11-05T17:52:52Z

lighthouse-core/lib/dependency-graph/simulator/network-analyzer.js

+   * @returns {LH.Artifacts.NetworkRequest|undefined}
+   */
+  static resolveRedirects(request) {
+    if (!request) return undefined;


Can we handle the undefined case where this is called instead?

I'm good with either, but this came from this suggestion

cc @patrickhulce

I suggested that so it would be drop-in replacement with the rest :)

I would also have a slight preference to handle the undefined at the caller, but I don't think it matters much honestly.

Alrighty, I leave it to you @bradfrosty, we're good with w/e!

Moved the undefined case to the caller -- let me know if that looks alright! Instead of handling the possible exception (like with the call to findMainDocument()), I type checked the mainDocument .

I also updated the undefined case test in network-analyzer-test.js to expect an exception to throw.

google-cla · 2020-11-05T19:21:17Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2020-11-05T19:23:08Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

patrickhulce · 2020-11-05T19:27:31Z

@googlebot I consent.

connorjclark · 2020-11-05T19:32:11Z

lighthouse-core/gather/gather-runner.js

@@ -259,9 +261,15 @@ class GatherRunner {
      mainRecord = NetworkAnalyzer.findMainDocument(networkRecords, passContext.url);
    } catch (_) {}

+    // MIME Type is only set on the final redirected document request. Use this for the HTML check instead of root.
+    let finalRecord;
+    if (mainRecord instanceof NetworkRequest) {


A simple truthy check is good enough, other than that this LGTM.

connorjclark · 2020-11-05T19:32:28Z

lighthouse-core/gather/gather-runner.js

@@ -259,9 +261,15 @@ class GatherRunner {
      mainRecord = NetworkAnalyzer.findMainDocument(networkRecords, passContext.url);
    } catch (_) {}

+    // MIME Type is only set on the final redirected document request. Use this for the HTML check instead of root.
+    let finalRecord;
+    if (mainRecord instanceof NetworkRequest) {


Suggested change

if (mainRecord instanceof NetworkRequest) {

if (mainRecord) {

connorjclark · 2020-11-05T19:33:39Z

lighthouse-core/test/lib/dependency-graph/simulator/network-analyzer-test.js

+      assert.equal(finalDocument.url, 'https://m.vk.com/');
+    });
+
+    it('should throw if not given a request', () => {


Since the parameter is typechecked to not allow undefined, we can consider this case covered trivially by the typesystem, so let's delete this test case.

patrickhulce · 2020-11-05T19:48:13Z

heads up @bradfrosty there's a lint error to fix based on one of the github applied suggestions :) (unused variable)

looks great though, can merge after that!

bradfrosty requested a review from a team as a code owner November 2, 2020 17:44

bradfrosty requested review from Beytoven and removed request for a team November 2, 2020 17:44

google-cla bot added the cla: yes label Nov 2, 2020

vercel bot deployed to Preview November 2, 2020 17:44 View deployment

devtools-bot assigned Beytoven Nov 2, 2020

devtools-bot added the waiting4reviewer label Nov 2, 2020

bradfrosty force-pushed the non-html-final-document branch from 5a2781a to e3675bf Compare November 2, 2020 17:55

vercel bot deployed to Preview November 2, 2020 17:55 View deployment

patrickhulce self-assigned this Nov 2, 2020

patrickhulce suggested changes Nov 2, 2020

View reviewed changes

patrickhulce added waiting4committer and removed waiting4reviewer labels Nov 2, 2020

vercel bot deployed to Preview November 2, 2020 23:06 View deployment

patrickhulce approved these changes Nov 2, 2020

View reviewed changes

lighthouse-core/gather/gather-runner.js Show resolved Hide resolved

patrickhulce removed their assignment Nov 2, 2020

patrickhulce added waiting4reviewer and removed waiting4committer labels Nov 2, 2020

vercel bot deployed to Preview November 2, 2020 23:37 View deployment

Beytoven approved these changes Nov 3, 2020

View reviewed changes

patrickhulce added land-when-ci-is-green chillin ❄ and removed waiting4reviewer land-when-ci-is-green labels Nov 3, 2020

bradfrosty added 2 commits November 4, 2020 11:12

core(gather-runner): use final document when reporting non-HTML error

bb7a6a1

Rename findFinalDocument to resolveRedirects

70c77d9

This also handles the undefined case inside the fn, instead of outside.

connorjclark added cla: yes and removed cla: no labels Nov 5, 2020

connorjclark reviewed Nov 5, 2020

View reviewed changes

vercel bot deployed to Preview November 5, 2020 19:21 View deployment

google-cla bot added cla: no and removed cla: yes labels Nov 5, 2020

bradfrosty force-pushed the non-html-final-document branch from 054b42f to 47d788b Compare November 5, 2020 19:23

vercel bot deployed to Preview November 5, 2020 19:23 View deployment

google-cla bot added cla: yes and removed cla: no labels Nov 5, 2020

patrickhulce self-assigned this Nov 5, 2020

patrickhulce added waiting4reviewer and removed chillin ❄ labels Nov 5, 2020

connorjclark reviewed Nov 5, 2020

View reviewed changes

bradfrosty force-pushed the non-html-final-document branch from 47d788b to bc2422d Compare November 5, 2020 19:38

vercel bot deployed to Preview November 5, 2020 19:38 View deployment

Move undefined case of resolveRedirects to caller

d8a2b11

bradfrosty force-pushed the non-html-final-document branch from bc2422d to d8a2b11 Compare November 5, 2020 19:49

vercel bot deployed to Preview November 5, 2020 19:49 View deployment

patrickhulce merged commit 4d3bda1 into GoogleChrome:master Nov 5, 2020

bradfrosty deleted the non-html-final-document branch November 5, 2020 21:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core(gather-runner): use final document when reporting non-HTML error #11620

core(gather-runner): use final document when reporting non-HTML error #11620

bradfrosty commented Nov 2, 2020

patrickhulce left a comment

patrickhulce Nov 2, 2020

bradfrosty Nov 2, 2020

patrickhulce Nov 2, 2020

patrickhulce Nov 2, 2020

bradfrosty commented Nov 2, 2020 •

edited

Loading

patrickhulce left a comment

Beytoven left a comment

patrickhulce commented Nov 3, 2020

Beytoven commented Nov 4, 2020

patrickhulce commented Nov 4, 2020

bradfrosty commented Nov 5, 2020

connorjclark commented Nov 5, 2020

connorjclark Nov 5, 2020

patrickhulce Nov 5, 2020

connorjclark Nov 5, 2020 •

edited

Loading

bradfrosty Nov 5, 2020

patrickhulce Nov 5, 2020

connorjclark Nov 5, 2020

bradfrosty Nov 5, 2020

google-cla bot commented Nov 5, 2020

google-cla bot commented Nov 5, 2020

patrickhulce commented Nov 5, 2020

connorjclark Nov 5, 2020

connorjclark Nov 5, 2020

connorjclark Nov 5, 2020

bradfrosty Nov 5, 2020

patrickhulce commented Nov 5, 2020

	if (mainRecord instanceof NetworkRequest) {
	if (mainRecord) {

core(gather-runner): use final document when reporting non-HTML error #11620

core(gather-runner): use final document when reporting non-HTML error #11620

Conversation

bradfrosty commented Nov 2, 2020

patrickhulce left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bradfrosty commented Nov 2, 2020 • edited Loading

patrickhulce left a comment

Choose a reason for hiding this comment

Beytoven left a comment

Choose a reason for hiding this comment

patrickhulce commented Nov 3, 2020

Beytoven commented Nov 4, 2020

patrickhulce commented Nov 4, 2020

bradfrosty commented Nov 5, 2020

connorjclark commented Nov 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

connorjclark Nov 5, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

google-cla bot commented Nov 5, 2020

google-cla bot commented Nov 5, 2020

patrickhulce commented Nov 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Nov 5, 2020

bradfrosty commented Nov 2, 2020 •

edited

Loading

connorjclark Nov 5, 2020 •

edited

Loading