Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: add inline scripts to Scripts artifact #7065

Merged
merged 27 commits into from
Mar 5, 2019
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
fac5bed
core: add inline scripts to Scripts artifact (#7060)
connorjclark Jan 19, 2019
a67cccb
make Scripts artifact an array of stuffs
connorjclark Jan 22, 2019
a8debfb
update golden
connorjclark Jan 22, 2019
8535f0b
rename var. update artifacts.json
connorjclark Jan 25, 2019
a959a03
revert changes to test artifact trace/devtools log
connorjclark Jan 28, 2019
13067d4
make Scripts.requestId optional
connorjclark Jan 28, 2019
0c28f25
meta...
connorjclark Jan 28, 2019
7a2a75b
warn instead of throw error
connorjclark Jan 28, 2019
554d398
add test for missing request id script
connorjclark Jan 28, 2019
f02e0b6
update golden lhr
connorjclark Jan 28, 2019
9321d6a
fix artifacts.json, code->content
connorjclark Jan 29, 2019
0fe823a
pr changes
connorjclark Jan 29, 2019
80a062e
update tets
connorjclark Jan 29, 2019
b5a0912
refactor syntax
connorjclark Jan 29, 2019
a38a548
Merge remote-tracking branch 'origin/master' into issue-7060-inline-s…
connorjclark Jan 31, 2019
9fc4e6d
Merge branch 'master' into issue-7060-inline-scripts
connorjclark Feb 13, 2019
cc8e401
elide inline content for js minified audit, add inline attribute to s…
connorjclark Feb 27, 2019
e78d8b5
update expected url
connorjclark Feb 27, 2019
98509ba
off by one bcuz idk
connorjclark Feb 27, 2019
8610f78
fix test
connorjclark Feb 27, 2019
c01ca17
Merge branch 'master' into issue-7060-inline-scripts
connorjclark Feb 27, 2019
54f416c
reorder
connorjclark Feb 27, 2019
9abd2f9
Merge remote-tracking branch 'origin/master' into issue-7060-inline-s…
connorjclark Mar 4, 2019
5c133cb
displayUrl, todo comment, pr changes
connorjclark Mar 4, 2019
28bfe98
update type comment
connorjclark Mar 4, 2019
b9f82ae
reverse ternary
connorjclark Mar 4, 2019
0ecd754
move const outside of try block
connorjclark Mar 4, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ module.exports = [
overallSavingsBytes: '>45000',
overallSavingsMs: '>500',
items: {
length: 1,
length: 3,
Copy link
Collaborator Author

@connorjclark connorjclark Jan 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now takes into account these two inline scripts

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we assert more in here to give insight about which 3 these should be? (shouldn't be exhaustive, but presumably the sort should be stable so could do urls)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. the data was stable too so I include it.

},
},
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,13 +77,12 @@ class UnminifiedJavaScript extends ByteEfficiencyAudit {
/** @type {Array<LH.Audit.ByteEfficiencyItem>} */
const items = [];
const warnings = [];
for (const requestId of Object.keys(artifacts.Scripts)) {
const scriptContent = artifacts.Scripts[requestId];
for (const {requestId, code} of artifacts.Scripts) {
const networkRecord = networkRecords.find(record => record.requestId === requestId);
if (!networkRecord || !scriptContent) continue;
if (!networkRecord || !code) continue;

try {
const result = UnminifiedJavaScript.computeWaste(scriptContent, networkRecord);
const result = UnminifiedJavaScript.computeWaste(code, networkRecord);
// If the ratio is minimal, the file is likely already minified, so ignore it.
// If the total number of bytes to be saved is quite small, it's also safe to ignore.
if (result.wastedPercent < IGNORE_THRESHOLD_IN_PERCENT ||
Expand Down
35 changes: 31 additions & 4 deletions lighthouse-core/gather/gatherers/scripts.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

const Gatherer = require('./gatherer');
const NetworkRequest = require('../../lib/network-request');
const getElementsInDocumentString = require('../../lib/page-functions.js').getElementsInDocumentString; // eslint-disable-line max-len
const URL = require('../../lib/url-shim.js');

/**
* @fileoverview Gets JavaScript file contents.
Expand All @@ -20,21 +22,46 @@ class Scripts extends Gatherer {
async afterPass(passContext, loadData) {
const driver = passContext.driver;

/** @type {Object<string, string>} */
const scriptContentMap = {};
/** @type {LH.Artifacts['Scripts']} */
brendankenny marked this conversation as resolved.
Show resolved Hide resolved
const scripts = [];

/** @type {string[]} */
const inlineScripts = await driver.evaluateAsync(`(() => {
${getElementsInDocumentString};

return getElementsInDocument('script')
.filter(meta => !meta.src && meta.text.trim())
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like this one isn't a meta ;)

.map(meta => meta.text);
})()`, {useIsolation: true});

if (inlineScripts.length) {
const mainResource = loadData.networkRecords.find(
request => URL.equalWithExcludedFragments(request.url, passContext.url));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the other instances of finding the main resource mention URL.equalWithExcludedFragments being slow and so they have a quicker startsWith first pass. Seems like this should do the same if the speed is really a concern?

Agreed it'd be much preferable to call a centralized version of this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. after the two merge PRs merge maybe we should just move that check into URL.equalWithExcludedFragments as an early return

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should just move that check into URL.equalWithExcludedFragments as an early return

yeah it's a bit different to do generically because in these cases we know for a fact there are no fragments on the network records but agreed we should not have to do these things, the URL.* methods should just be fast by default :)

if (!mainResource) {
throw new Error('could not locate mainResource');
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a teensy bit worried that this is going to start throwing errors into our perf category on edge cases we previously didn't really care about. We don't actually need a request ID for the whole thing to work. WDYT about making requestId optional?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds alright, but what are these edge cases?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I knew them I would fix them :)

I just meant that we have seen this error popup in peoples reports before and it's mostly 🤷‍♂️and move on since it wasn't that big a deal. Now it'll be a bit more user-facing.

I actually had to tackle this for canonical though and think it'll be solid.

https://github.com/GoogleChrome/lighthouse/pull/7080/files#diff-2113a8215d43931339dbb50287d89dcbR450

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. thoughts on handling reporting of unminified scripts in the JS audit? I just put ? as a placeholder url :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, one of us should extract that function to somewhere else (whoever merges last)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract it off of network-analyzer? that actually seems like a good place for it :)

though if you mean we should finally move network-analyzer out of the depdency-graph/simulator folder then I agree with you :)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. thoughts on handling reporting of unminified scripts in the JS audit? I just put ? as a placeholder url

Ideally we'd try to provide a snippet of the code like we do for CSS, but I'm OK with this for now until we see how common it is.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe when the fancy code snippet preview (#6901) lands we can do something cool here :D

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah having it in network-analyzer is perfect

💯 to snippets

}
scripts.push(...inlineScripts.map(code => ({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we've generally preferred the explicit return if an arrow function is going to span multiple lines since the return behavior gets easier to miss the longer the code goes on. I think that counts doubly for the returning-an-object case because the signal for reading it (and parsing it) as a block is so strong :)

cool with adding an explicit return ?

code,
requestId: mainResource.requestId,
})));
}

const scriptRecords = loadData.networkRecords
.filter(record => record.resourceType === NetworkRequest.TYPES.Script);

for (const record of scriptRecords) {
try {
const content = await driver.getRequestContent(record.requestId);
if (content) {
scriptContentMap[record.requestId] = content;
scripts.push({
code: content,
requestId: record.requestId,
});
}
} catch (e) {}
}

return scriptContentMap;
return scripts;
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,35 +16,44 @@ const resourceType = 'Script';
describe('Page uses optimized responses', () => {
it('fails when given unminified scripts', () => {
const auditResult = UnminifiedJavascriptAudit.audit_({
Scripts: {
'123.1':
`
Scripts: [
{
requestId: '123.1',
code: `
var foo = new Set();
foo.add(1);
foo.add(2);

if (foo.has(2)) {
console.log('hello!')
}
`,
'123.2':
`
`,
},
{
requestId: '123.2',
code: `
const foo = new Set();
foo.add(1);

async function go() {
await foo.has(1)
console.log('yay esnext!')
}
`,
'123.3':
/* eslint-disable no-useless-escape */
`,
},
{
requestId: '123.3',
code: /* eslint-disable no-useless-escape */
`
const foo = 1
/Edge\/\d*\.\d*/.exec('foo')
`,
'123.4': '#$*%dense',
},
},
{
requestId: '123.4',
code: '#$*%dense',
},
],
}, [
{requestId: '123.1', url: 'foo.js', transferSize: 20 * KB, resourceType},
{requestId: '123.2', url: 'other.js', transferSize: 50 * KB, resourceType},
Expand All @@ -58,30 +67,36 @@ describe('Page uses optimized responses', () => {
}));

expect(results).toMatchObject([
{url: 'foo.js', wastedPercent: 57, wastedKB: 11},
{url: 'other.js', wastedPercent: 53, wastedKB: 27},
{url: 'foo.js', wastedPercent: 56, wastedKB: 11},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did this change?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace changes in the provided network record contents. the content: eats up some of the ws :)

{url: 'other.js', wastedPercent: 53, wastedKB: 26},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

{url: 'valid-ish.js', wastedPercent: 39, wastedKB: 39},
]);
});

it('passes when scripts are already minified', () => {
const auditResult = UnminifiedJavascriptAudit.audit_({
Scripts: {
'123.1':
'var f=new Set();f.add(1);f.add(2);if(f.has(2))console.log(1234)',
'123.2':
`
const foo = new Set();
foo.add(1);
Scripts: [
{
requestId: '123.1',
code: 'var f=new Set();f.add(1);f.add(2);if(f.has(2))console.log(1234)',
},
{
requestId: '123.2',
code: `
const foo = new Set();
foo.add(1);

async function go() {
await foo.has(1)
console.log('yay esnext!')
}
`,
'123.3':
'for{(wtf',
},
async function go() {
await foo.has(1)
console.log('yay esnext!')
}
`,
},
{
requestId: '123.3',
code: 'for{(wtf',
},
],
}, [
{requestId: '123.1', url: 'foo.js', transferSize: 20 * KB, resourceType},
{requestId: '123.2', url: 'other.js', transferSize: 3 * KB, resourceType}, // too small
Expand Down
72 changes: 16 additions & 56 deletions lighthouse-core/test/results/sample_v2.json
Original file line number Diff line number Diff line change
Expand Up @@ -1998,41 +1998,10 @@
"id": "unminified-javascript",
"title": "Minify JavaScript",
"description": "Minifying JavaScript files can reduce payload sizes and script parse time. [Learn more](https://developers.google.com/speed/docs/insights/MinifyResources).",
"score": 0.88,
"scoreDisplayMode": "numeric",
"rawValue": 150,
"displayValue": "Potential savings of 30 KB",
"warnings": [],
"details": {
"type": "opportunity",
"headings": [
{
"key": "url",
"valueType": "url",
"label": "URL"
},
{
"key": "totalBytes",
"valueType": "bytes",
"label": "Size (KB)"
},
{
"key": "wastedBytes",
"valueType": "bytes",
"label": "Potential Savings (KB)"
}
],
"items": [
{
"url": "http://localhost:10200/zone.js",
"totalBytes": 71654,
"wastedBytes": 30470,
"wastedPercent": 42.52388078488413
}
],
"overallSavingsMs": 150,
"overallSavingsBytes": 30470
}
"score": null,
"scoreDisplayMode": "error",
"rawValue": null,
"errorMessage": "artifacts.Scripts is not iterable"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update the artifacts.json?

},
"unused-css-rules": {
"id": "unused-css-rules",
Expand Down Expand Up @@ -4549,7 +4518,6 @@
"audits[uses-long-cache-ttl].details.headings[0].text",
"audits[total-byte-weight].details.headings[0].text",
"audits[render-blocking-resources].details.headings[0].label",
"audits[unminified-javascript].details.headings[0].label",
"audits[uses-webp-images].details.headings[1].label",
"audits[uses-text-compression].details.headings[0].label"
],
Expand Down Expand Up @@ -4819,7 +4787,6 @@
"audits[uses-long-cache-ttl].details.headings[2].text",
"audits[total-byte-weight].details.headings[1].text",
"audits[render-blocking-resources].details.headings[1].label",
"audits[unminified-javascript].details.headings[1].label",
"audits[uses-webp-images].details.headings[2].label",
"audits[uses-text-compression].details.headings[1].label"
],
Expand Down Expand Up @@ -4872,13 +4839,19 @@
"lighthouse-core/audits/byte-efficiency/unminified-javascript.js | description": [
"audits[unminified-javascript].description"
],
"lighthouse-core/audits/byte-efficiency/unused-css-rules.js | title": [
"audits[unused-css-rules].title"
],
"lighthouse-core/audits/byte-efficiency/unused-css-rules.js | description": [
"audits[unused-css-rules].description"
],
"lighthouse-core/audits/byte-efficiency/uses-webp-images.js | title": [
"audits[uses-webp-images].title"
],
"lighthouse-core/audits/byte-efficiency/uses-webp-images.js | description": [
"audits[uses-webp-images].description"
],
"lighthouse-core/lib/i18n/i18n.js | displayValueByteSavings": [
{
"values": {
"wastedBytes": 30470
},
"path": "audits[unminified-javascript].displayValue"
},
{
"values": {
"wastedBytes": 8526
Expand All @@ -4893,22 +4866,9 @@
}
],
"lighthouse-core/lib/i18n/i18n.js | columnWastedBytes": [
"audits[unminified-javascript].details.headings[2].label",
"audits[uses-webp-images].details.headings[3].label",
"audits[uses-text-compression].details.headings[2].label"
],
"lighthouse-core/audits/byte-efficiency/unused-css-rules.js | title": [
"audits[unused-css-rules].title"
],
"lighthouse-core/audits/byte-efficiency/unused-css-rules.js | description": [
"audits[unused-css-rules].description"
],
"lighthouse-core/audits/byte-efficiency/uses-webp-images.js | title": [
"audits[uses-webp-images].title"
],
"lighthouse-core/audits/byte-efficiency/uses-webp-images.js | description": [
"audits[uses-webp-images].description"
],
"lighthouse-core/audits/byte-efficiency/uses-optimized-images.js | title": [
"audits[uses-optimized-images].title"
],
Expand Down
39 changes: 4 additions & 35 deletions proto/sample_v2_round_trip.json
Original file line number Diff line number Diff line change
Expand Up @@ -2114,42 +2114,11 @@
},
"unminified-javascript": {
"description": "Minifying JavaScript files can reduce payload sizes and script parse time. [Learn more](https://developers.google.com/speed/docs/insights/MinifyResources).",
"details": {
"headings": [
{
"key": "url",
"label": "URL",
"valueType": "url"
},
{
"key": "totalBytes",
"label": "Size (KB)",
"valueType": "bytes"
},
{
"key": "wastedBytes",
"label": "Potential Savings (KB)",
"valueType": "bytes"
}
],
"items": [
{
"totalBytes": 71654.0,
"url": "http://localhost:10200/zone.js",
"wastedBytes": 30470.0,
"wastedPercent": 42.52388078488413
}
],
"overallSavingsBytes": 30470.0,
"overallSavingsMs": 150.0,
"type": "opportunity"
},
"displayValue": "Potential savings of 30\u00a0KB",
"errorMessage": "artifacts.Scripts is not iterable",
"id": "unminified-javascript",
"score": 0.88,
"scoreDisplayMode": "numeric",
"title": "Minify JavaScript",
"warnings": []
"score": null,
"scoreDisplayMode": "error",
"title": "Minify JavaScript"
},
"unused-css-rules": {
"description": "Remove unused rules from stylesheets to reduce unnecessary bytes consumed by network activity. [Learn more](https://developers.google.com/web/tools/lighthouse/audits/unused-css).",
Expand Down
4 changes: 2 additions & 2 deletions types/artifacts.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,8 @@ declare global {
RobotsTxt: {status: number|null, content: string|null};
/** Set of exceptions thrown during page load. */
RuntimeExceptions: Crdp.Runtime.ExceptionThrownEvent[];
/** The content of all scripts loaded by the page, keyed by networkRecord requestId. */
Scripts: Record<string, string>;
/** The content of all scripts loaded by the page, and the networkRecord requestId that loaded them. Note, HTML documents will have one entry for the same requestId per script tag. */
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe reverse these clauses, Note, HTML documents will have one entry per script tag, all with the same requestId.

Scripts: Array<{code: string, requestId: string}>;
/** Version information for all ServiceWorkers active after the first page load. */
ServiceWorker: {versions: Crdp.ServiceWorker.ServiceWorkerVersion[], registrations: Crdp.ServiceWorker.ServiceWorkerRegistration[]};
/** The status of an offline fetch of the page's start_url. -1 and a explanation if missing or there was an error. */
Expand Down