Removes hash fragments from url in offline audit #1319

wardpeet · 2016-12-26T21:00:44Z

Reviewers: ALL
We store the url inside options.url without a fragment so we should do this check without fragment as well.
Make sure that I don't break anything crucial :p

Fixes PR #1296

ebidel · 2016-12-27T00:53:07Z

lighthouse-core/gather/gatherers/offline.js

@@ -34,4 +36,8 @@ class Offline extends Gatherer {
  }
 }

+function cleanHash(url) {
+  return url.href.removeURLFragment();


I'm not familiar withremoveURLFragment. Where does it come from?

If it's not part of a spec, url.origin + url.pathname + url.search would be clear.

To be honest autocomplete let me to it :p. (https://cs.chromium.org/chromium/src/third_party/WebKit/Source/devtools/front_end/platform/utilities.js?q=removeURLFragment&sq=package:chromium&l=215)

I can just copy that function instead so we are sure it never disappears. Reason I didn't want to build the url myself is that I keep the href as pure as it can be but maybe it's not necessary at all 😄

Oh wow. Yea, let's go with our own implementation instead of a function added to String's prototoype.

How about url.origin + url.pathname + url.search? It's obvious and pure 👍

maybe do the new URL() part inside here too? Can also do url.hash = ''; return url.href;

okay the subclass of URL is now in place, so we can move it there. :)

brendankenny · 2017-01-03T23:38:09Z

lighthouse-core/gather/gatherers/offline.js

@@ -25,7 +26,8 @@ class Offline extends Gatherer {

  afterPass(options, tracingData) {
    const navigationRecord = tracingData.networkRecords.filter(record => {
-      return record._url === options.url && record._fetchedViaServiceWorker;
+      return cleanHash(new URL(record._url)) === cleanHash(new URL(options.url)) &&


This seems a little worrying at first (how do we account for all the crazy things SPAs do with URL fragments?), but this is in line with how service worker caching of the page works (the SW Cache explicitly ignores the URL fragment on requests), so seems like it may be the right thing to do.

@jeffposnick for his opinion and for any cases where this may go awry

I'm not sure I understand the full context, but in general, yeah, I think ignoring the hash would make sense from the point of view of both the Cache Storage API and navigation requests.

brendankenny · 2017-01-03T23:40:28Z

lighthouse-core/gather/gatherers/offline.js

@@ -34,4 +36,8 @@ class Offline extends Gatherer {
  }
 }

+function cleanHash(url) {
+  return url.href.removeURLFragment();


maybe do the new URL() part inside here too? Can also do url.hash = ''; return url.href;

brendankenny · 2017-01-03T23:41:27Z

lighthouse-core/test/gather/gatherers/offline-test.js

+        assert.strictEqual(artifact, 200);
+      }),
+      offlineGather.afterPass(optionsWithQueryString, tracingData).then(artifact => {
+        assert.strictEqual(artifact, -1);


this is a failure test, so maybe move to the other it()?

brendankenny · 2017-01-03T23:52:46Z

We store the url inside options.url without a fragment so we should do this check without fragment as well.

To clarify what's happening here:

In the CLI (based on the example in "URL responds with a 200 when offline" fails incorrectly #1296), if you test https://unindented.github.io/little-yam/, the offline test will also load https://unindented.github.io/little-yam/ which matches the response URL that comes back from the SW.
In the extension, if you load https://unindented.github.io/little-yam/ it gets redirected to https://unindented.github.io/little-yam/#/login?_k=oaybjy before testing by LH even starts, which we then compare against the offline response URL https://unindented.github.io/little-yam/ from the SW and fail the test.

In this example, it's actually sw-precache that's stripping the fragment and requesting https://unindented.github.io/little-yam/, but that's what would be done anyway if you weren't using sw-precache and requested a match for https://unindented.github.io/little-yam/#/login?_k=oaybjy from the cache directly.

This is kind of the opposite problem of #715, where a real redirect (not just adding a fragment to the URL) makes the CLI fail the offline test because it tests the original URL but the extension passes since it always tests the redirected URL.

paulirish · 2017-01-04T03:19:27Z

I think this could sit on top of the subclassed URL proposal I made over here:
#1390 (comment)

so it could be new URL(record._url)).stripHash() === new URL(options._url)).stripHash() && ...

wardpeet · 2017-01-04T16:18:21Z

@paulirish that make sense :)
I could change this to @ebidel suggestion and @patrickhulce could fix the url abstraction inside #1390 or i can just close this PR :p

@brendankenny maybe update the audit to check url with fragment and without?

brendankenny · 2017-01-04T23:35:59Z

#1390 landed, so good to go on adding method to the url shim

wardpeet · 2017-01-08T11:42:09Z

I've 2 options. I went for option 1

Option 1:

/**
 * Removes hash from URL
 * @return {URL} Url object chaining
 */
URL.prototype.stripHash = function stripHash() {
  this.hash = '';

  return this;
};

Option 2:

/**
 * Removes hash from an url and returns it
 * @return {string} Url without the hash
 */
URL.prototype.stripHash = function stripHash() {
  const currentHash = this.hash;
  this.hash = '';

  const urlWithoutHash = this.href;
  this.hash = currentHash;

  return urlWithoutHash;
};

brendankenny · 2017-01-09T18:58:40Z

a third option would mirror the URL spec's equality with the exclude fragments flag set. We could name it something awkward like

/**
 * Determine if url1 equals url2, ignoring URL fragments.
 * @param {string} url1
 * @param {string} url2
 * @return {boolean}
 */
URL.equalWithExcludedFragments = function(url1, url2) {
  url1 = new URL(url1);
  url1.hash = '';

  url2 = new URL(url2);
  url2.hash = '';

  return url1.href === url2.href;
};

(or pick a much better name :)

wardpeet · 2017-01-09T22:04:34Z

@brendankenny do you rather have option 3?

brendankenny · 2017-01-09T22:07:49Z

I would, but don't know what others think

brendankenny · 2017-01-09T22:08:34Z

lighthouse-core/lib/url-shim.js

+ * Removes hash from URL
+ * @return {URL} Url object chaining
+ */
+URL.prototype.stripHash = function stripHash() {


need test(s) in url-shim-test for whatever we end up putting in here

patrickhulce · 2017-01-12T00:25:52Z

+1 to a static method over a prototype extension, but I'm not terribly invested in it and fine with this too.

wardpeet · 2017-01-12T06:09:10Z

equalWithExcludedFragments it will be :)

brendankenny · 2017-01-12T18:37:09Z

yay awkward naming :)

brendankenny · 2017-01-18T18:50:13Z

@wardpeet :):) what do you think?

wardpeet · 2017-01-19T05:58:44Z

i'll try to update today :)

ebidel · 2017-01-21T00:48:47Z

@wardpeet can you rebase?

…ne check. We store the url inside options.url without a fragment so we should do this check without fragment as well. Fixes PR GoogleChrome#1296

wardpeet · 2017-01-21T19:00:09Z

Updated finally, sorry for the wait!

ebidel · 2017-01-23T20:28:48Z

Can you add a test for equalWithExcludedFragments in url-shim-test.js?

brendankenny

@wardpeet I'm going to add the tests for you :P

wardpeet · 2017-01-28T15:54:14Z

@brendankenny thanks

ebidel suggested changes Dec 27, 2016

View reviewed changes

brendankenny requested changes Jan 3, 2017

View reviewed changes

wardpeet force-pushed the offline-queryfragment branch from 11229bc to 5aeabec Compare January 8, 2017 11:41

ebidel approved these changes Jan 9, 2017

View reviewed changes

brendankenny reviewed Jan 9, 2017

View reviewed changes

wardpeet added 3 commits January 21, 2017 19:56

Removes hash fragments from url so serviceworker does a correct offli…

b72d90e

…ne check. We store the url inside options.url without a fragment so we should do this check without fragment as well. Fixes PR GoogleChrome#1296

Updated review

d729f47

Added equalWithExcludedFragments to check urls without fragments

4e85708

wardpeet force-pushed the offline-queryfragment branch from b466513 to 4e85708 Compare January 21, 2017 19:02

brendankenny approved these changes Jan 28, 2017

View reviewed changes

brendankenny merged commit 385dc9b into GoogleChrome:master Jan 28, 2017

brendankenny mentioned this pull request Jan 28, 2017

add tests for url-shim's equalWithExcludedFragments #1566

Merged

wardpeet deleted the offline-queryfragment branch May 11, 2017 18:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removes hash fragments from url in offline audit #1319

Removes hash fragments from url in offline audit #1319

wardpeet commented Dec 26, 2016

ebidel Dec 27, 2016

wardpeet Dec 27, 2016 •

edited

Loading

ebidel Dec 29, 2016

brendankenny Jan 3, 2017

paulirish Jan 4, 2017

brendankenny Jan 3, 2017 •

edited

Loading

jeffposnick Jan 4, 2017

brendankenny Jan 3, 2017

brendankenny Jan 3, 2017

brendankenny commented Jan 3, 2017 •

edited

Loading

paulirish commented Jan 4, 2017

wardpeet commented Jan 4, 2017 •

edited

Loading

brendankenny commented Jan 4, 2017

wardpeet commented Jan 8, 2017

brendankenny commented Jan 9, 2017

wardpeet commented Jan 9, 2017 •

edited

Loading

brendankenny commented Jan 9, 2017

brendankenny Jan 9, 2017

patrickhulce commented Jan 12, 2017

wardpeet commented Jan 12, 2017

brendankenny commented Jan 12, 2017

brendankenny commented Jan 18, 2017

wardpeet commented Jan 19, 2017

ebidel commented Jan 21, 2017

wardpeet commented Jan 21, 2017

ebidel commented Jan 23, 2017

brendankenny left a comment

wardpeet commented Jan 28, 2017

Removes hash fragments from url in offline audit #1319

Removes hash fragments from url in offline audit #1319

Conversation

wardpeet commented Dec 26, 2016

Choose a reason for hiding this comment

wardpeet Dec 27, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny Jan 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brendankenny commented Jan 3, 2017 • edited Loading

paulirish commented Jan 4, 2017

wardpeet commented Jan 4, 2017 • edited Loading

brendankenny commented Jan 4, 2017

wardpeet commented Jan 8, 2017

brendankenny commented Jan 9, 2017

wardpeet commented Jan 9, 2017 • edited Loading

brendankenny commented Jan 9, 2017

Choose a reason for hiding this comment

patrickhulce commented Jan 12, 2017

wardpeet commented Jan 12, 2017

brendankenny commented Jan 12, 2017

brendankenny commented Jan 18, 2017

wardpeet commented Jan 19, 2017

ebidel commented Jan 21, 2017

wardpeet commented Jan 21, 2017

ebidel commented Jan 23, 2017

brendankenny left a comment

Choose a reason for hiding this comment

wardpeet commented Jan 28, 2017

wardpeet Dec 27, 2016 •

edited

Loading

brendankenny Jan 3, 2017 •

edited

Loading

brendankenny commented Jan 3, 2017 •

edited

Loading

wardpeet commented Jan 4, 2017 •

edited

Loading

wardpeet commented Jan 9, 2017 •

edited

Loading