core(preload): only allow same origin (domain + subdomains) #5065

wardpeet · 2018-04-29T15:23:46Z

Fixes nr 1 of #4425
Restrict results to same-origin (as a rough filter for predictable and unchanging URLs). Same origin as in subdomains and rootdomains.

patrickhulce · 2018-04-30T16:41:26Z

@wardpeet FYI I had to rework considerable portions of this for lantern (#5062, would love your review!), wdyt about holdin' tight for a cycle to get that in first? :)

wardpeet · 2018-05-02T16:27:21Z

sure thing no worries

patrickhulce · 2018-05-03T00:23:13Z

alright sorry for the conflicts @wardpeet, but this should be ready to rebase now!

paulirish · 2018-05-14T22:16:02Z

lighthouse-core/lib/url-shim.js

+   * @param {string} urlA
+   * @param {string} urlB
+   */
+  static rootDomainsMatch(urlA, urlB) {


can you add some tests in url-shim-test?

yeah forgot about that one :)

patrickhulce · 2018-05-16T17:30:48Z

lighthouse-core/lib/url-shim.js

+    }
+
+    const urlARootDomain = urlAInfo.hostname.split('.').slice(-2).join('.');
+    const urlBRootDomain = urlBInfo.hostname.split('.').slice(-2).join('.');


https://en.wikipedia.org/wiki/Second-level_domain is a good starting point

paulirish · 2018-05-16T17:30:24Z

lighthouse-core/lib/url-shim.js

+      return false;
+    }
+
+    const urlARootDomain = urlAInfo.hostname.split('.').slice(-2).join('.');


this is called "tld plus one". let's use that term for the method name and here.

let's add a few whitelisted exceptions like '.co.uk'. (country-code second level domains).

paulirish · 2018-05-23T17:16:42Z

lighthouse-core/lib/url-shim.js

@@ -17,6 +17,24 @@ const Util = require('../report/html/renderer/util.js');
 const URL = /** @type {!Window["URL"]} */ (typeof self !== 'undefined' && self.URL) ||
    require('url').URL;

+const tldPlusOne = {


let's hardcode just the keys of this list (and add .co.uk)

add comment saying that https://publicsuffix.org/ is the real source of this data. and hypothetically we could pull in a dep like https://npms.io/search?q=public+suffix+list to do this forreal, but at significant byte cost.

and admit we're taking the lazy way out.

shared a sheet with you on the most popular 2-level TLDs, but summary: if we did a regex with /.(co|com|org|edu|gov).TLD$/ that should cover most cases

I added a new list to the regex coming from http archive. Not sure how I should comment it in the code. Also i'm not matching all 1000 entries

patrickhulce · 2018-05-30T21:18:22Z

lighthouse-core/lib/url-shim.js

@@ -17,6 +17,13 @@ const Util = require('../report/html/renderer/util.js');
 const URL = /** @type {!Window["URL"]} */ (typeof self !== 'undefined' && self.URL) ||
    require('url').URL;

+const tldPlusOneRegex = new RegExp(`\\.(com|co|gov|org|ac|or|edu|go|nic|in|gouv|gob|web|spb|net|` +


I think we can trim down this list a bit based on the coverage percentages in that sheet, doing just the top 10 already gives us 94% coverage

com 2321232 co 1714813 gov 435538 edu 304490 ac 222746 org 202231 go 82772 gob 73721 or 72214 net 43587 ---------------- sum 5473344 total 5841097 coverage ~94%

we could also just go to top 15, everything beyond that is <10000 occurrences

patrickhulce

thanks @wardpeet !

patrickhulce · 2018-05-30T21:19:22Z

lighthouse-core/lib/url-shim.js

+  static getTld(hostname) {
+    const tld = hostname.split('.').slice(-1)[0];
+
+    const tldRegex = new RegExp(`${tldPlusOneRegex.source}\\.${tld}$`);


let's just do the .split and a tlds.has instead :)

wardpeet · 2018-06-01T23:34:12Z

Updated not sure this is what you had in mind

patrickhulce

this looks good to me! thanks @wardpeet!! 👏 👏

(got some lint things to fix it seems)

patrickhulce · 2018-06-01T23:39:56Z

lighthouse-core/test/audits/uses-rel-preload-test.js

@@ -50,14 +50,18 @@ describe('Performance: uses-rel-preload audit', () => {

  it('should suggest preload resource', () => {
    const rootNode = buildNode(1, 'http://example.com');
-    const mainDocumentNode = buildNode(2, 'http://www.example.com');
+    const mainDocumentNode = buildNode(2, 'http://www.example.com:3000');


we're gonna need some graph creation helpers soon to make all this stuff :)

patrickhulce · 2018-06-01T23:41:35Z

lighthouse-core/test/audits/uses-rel-preload-test.js

      },
    };

    const mainResource = Object.assign({}, defaultMainResource, {
-      url: 'http://www.example.com',
+      url: 'http://www.example.com:3000',


is this just to exercise extra hostname checks?

yes it's to make sure that origins with a port number are handled correct by the rootDomainMatch

patrickhulce · 2018-06-01T23:42:01Z

lighthouse-core/test/audits/uses-rel-preload-test.js

@@ -131,7 +165,7 @@ describe('Performance: uses-rel-preload audit', () => {
    return UsesRelPreload.audit(mockArtifacts(networkRecords, chains, mainResource), {}).then(
      output => {
        assert.equal(output.rawValue, 1250);
-        assert.equal(output.details.items.length, 1);
+        assert.equal(output.details.items.length, 2);


maybe overkill but assert the URL matches too?

patrickhulce · 2018-06-01T23:47:51Z

lighthouse-core/lib/url-shim.js

@@ -17,6 +17,10 @@ const Util = require('../report/html/renderer/util.js');
 const URL = /** @type {!Window["URL"]} */ (typeof self !== 'undefined' && self.URL) ||
    require('url').URL;

+const listOfTlds = [


Add a comment that somewhat documents what happened here re: coverage and whatnot? maybe just link to the best comment thread on this PR :)

patrickhulce · 2018-06-11T20:37:10Z

so I just realized in #5471, we don't want to hide URLs based on if they were 3rd party, we want to hide them if their initiator was a 3rd party 🤔 nearly all of this should still apply, but I'll prob make an update soon :)

wardpeet requested review from brendankenny, patrickhulce and paulirish as code owners April 29, 2018 15:23

devtools-bot added the waiting4reviewer label Apr 29, 2018

wardpeet force-pushed the feature/preload-subdomains branch from 32a4b20 to 9d95eeb Compare April 29, 2018 19:35

paulirish added chillin ❄ and removed waiting4reviewer labels Apr 30, 2018

patrickhulce removed the chillin ❄ label May 3, 2018

wardpeet force-pushed the feature/preload-subdomains branch from f9b3442 to 6c8c56d Compare May 4, 2018 18:04

paulirish reviewed May 14, 2018

View reviewed changes

patrickhulce reviewed May 16, 2018

View reviewed changes

paulirish reviewed May 16, 2018

View reviewed changes

paulirish changed the title ~~Only allow same origin (domain + subdomains) for preload~~ core(preload): only allow same origin (domain + subdomains) May 18, 2018

paulirish changed the title ~~core(preload): only allow same origin (domain + subdomains)~~ only allow same origin (domain + subdomains) May 18, 2018

paulirish changed the title ~~only allow same origin (domain + subdomains)~~ core(preload): only allow same origin (domain + subdomains) May 18, 2018

paulirish changed the title ~~core(preload): only allow same origin (domain + subdomains)~~ core(preload): only allow same origin (domain + subdomains). May 18, 2018

paulirish changed the title ~~core(preload): only allow same origin (domain + subdomains).~~ core(preload): only allow same origin (domain + subdomains).. May 18, 2018

paulirish changed the title ~~core(preload): only allow same origin (domain + subdomains)..~~ core(preload): only allow same origin (domain + subdomains) May 18, 2018

paulirish had a problem deploying to PR staging May 20, 2018 21:43 Error

paulirish approved these changes May 23, 2018

View reviewed changes

wardpeet added 5 commits May 30, 2018 22:25

Only allow same origin (domain + subdomains) for preload

af8118e

Use hostname instead of host

5a7c041

Fix codecov

71f6153

Create tld plus one helper

7b697ad

Update get tld

a681ce7

wardpeet force-pushed the feature/preload-subdomains branch from a66ea70 to a681ce7 Compare May 30, 2018 20:25

patrickhulce reviewed May 30, 2018

View reviewed changes

update tld check

295d016

patrickhulce approved these changes Jun 1, 2018

View reviewed changes

Review changes

acc39e1

patrickhulce merged commit d385645 into master Jun 6, 2018

patrickhulce deleted the feature/preload-subdomains branch June 6, 2018 16:39

paulirish mentioned this pull request Mar 24, 2019

docs(url-shim): add comment about public suffix lst #7671

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core(preload): only allow same origin (domain + subdomains) #5065

core(preload): only allow same origin (domain + subdomains) #5065

wardpeet commented Apr 29, 2018 •

edited

Loading

patrickhulce commented Apr 30, 2018 •

edited

Loading

wardpeet commented May 2, 2018

patrickhulce commented May 3, 2018

paulirish May 14, 2018

wardpeet May 16, 2018

patrickhulce May 16, 2018

paulirish May 16, 2018

paulirish May 23, 2018

patrickhulce May 23, 2018

wardpeet May 30, 2018

patrickhulce May 30, 2018

patrickhulce May 30, 2018

patrickhulce left a comment

patrickhulce May 30, 2018

wardpeet commented Jun 1, 2018

patrickhulce left a comment

patrickhulce Jun 1, 2018

patrickhulce Jun 1, 2018

wardpeet Jun 2, 2018

patrickhulce Jun 1, 2018

patrickhulce Jun 1, 2018

patrickhulce commented Jun 11, 2018

core(preload): only allow same origin (domain + subdomains) #5065

core(preload): only allow same origin (domain + subdomains) #5065

Conversation

wardpeet commented Apr 29, 2018 • edited Loading

patrickhulce commented Apr 30, 2018 • edited Loading

wardpeet commented May 2, 2018

patrickhulce commented May 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wardpeet commented Jun 1, 2018

patrickhulce left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickhulce commented Jun 11, 2018

wardpeet commented Apr 29, 2018 •

edited

Loading

patrickhulce commented Apr 30, 2018 •

edited

Loading