Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: gather source maps #9101

Merged
merged 90 commits into from Jul 25, 2019
Merged
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
3ce3602
source maps gather first pass
connorjclark Jun 1, 2019
7e9b440
own gatherer
connorjclark Jun 1, 2019
66bc6c6
smoke test
connorjclark Jun 1, 2019
fceb88c
no jest test
connorjclark Jun 1, 2019
c721cbd
more smoke
connorjclark Jun 1, 2019
e217b37
snap
connorjclark Jun 1, 2019
baa97c2
coverage fix
connorjclark Jun 1, 2019
5da78cf
get with the program
connorjclark Jun 1, 2019
8719ccb
remove source-maps from default config
connorjclark Jun 2, 2019
5886359
revert snapshot
connorjclark Jun 3, 2019
a6666fe
load expected map
connorjclark Jun 3, 2019
2b0a98f
tweak smoke config
connorjclark Jun 3, 2019
8708432
filter script parsed events
connorjclark Jun 3, 2019
203c934
errorMessage
connorjclark Jun 3, 2019
c3183b6
bind
connorjclark Jun 3, 2019
d6a51e2
httpz so fresh
connorjclark Jun 3, 2019
a6be4c4
one eval per map fetch
connorjclark Jun 3, 2019
61963aa
one at a time
connorjclark Jun 3, 2019
b0bc06b
remove bang hack
connorjclark Jun 4, 2019
ddf4e10
fix double error
connorjclark Jun 5, 2019
59d6414
edit comment
connorjclark Jun 7, 2019
dd46ccf
remove source map dep
connorjclark Jun 18, 2019
cb33c1f
Merge remote-tracking branch 'origin/master' into gather-source-maps
connorjclark Jun 18, 2019
dac0523
revert smokehouse report changes
connorjclark Jun 18, 2019
fea9706
eslint
connorjclark Jun 18, 2019
65be82f
ts
connorjclark Jun 18, 2019
6efbe2d
scripturl, types
connorjclark Jun 18, 2019
3c53626
fix smokes
connorjclark Jun 18, 2019
cb3ad96
reduce try catch
connorjclark Jun 18, 2019
7036618
doc strig
connorjclark Jun 18, 2019
c203eec
doc strring
connorjclark Jun 18, 2019
f48656a
..
connorjclark Jun 18, 2019
c184fe6
types
connorjclark Jun 18, 2019
75f3be6
assert on key
connorjclark Jun 18, 2019
7fea7e1
rename fixture
connorjclark Jun 18, 2019
d80a6d4
nits
connorjclark Jun 18, 2019
65ed824
unit test
connorjclark Jun 19, 2019
fe31b29
bug fix
connorjclark Jun 19, 2019
39357b6
250ms timeout
connorjclark Jun 25, 2019
c401d75
kick
connorjclark Jun 25, 2019
e6c8655
move try catch to for loop
connorjclark Jun 25, 2019
b8c4114
steal patricks driver mocks
connorjclark Jun 28, 2019
86b02a7
mock-driver
connorjclark Jun 28, 2019
b75b39a
destructuring in loops is 🔥
connorjclark Jun 28, 2019
05e0b58
comment
connorjclark Jun 28, 2019
ecbbcd1
docs
connorjclark Jul 1, 2019
e3954ac
file comment
connorjclark Jul 1, 2019
568c73e
rm var
connorjclark Jul 1, 2019
d5dc80c
remove smokes
connorjclark Jul 1, 2019
aed3532
fix test
connorjclark Jul 1, 2019
1562659
Apply suggestions from code review
connorjclark Jul 10, 2019
2d6d85e
varios test changes
connorjclark Jul 10, 2019
12f7a11
test names
connorjclark Jul 10, 2019
a5f035e
comment
connorjclark Jul 10, 2019
1fa1ec2
log statements
connorjclark Jul 11, 2019
4f87472
brendan patch
connorjclark Jul 11, 2019
07a878d
add failing test b/c of resolving
connorjclark Jul 11, 2019
84a3216
comment
connorjclark Jul 11, 2019
4175f5d
resolve to base url
connorjclark Jul 11, 2019
546915c
pr changes
connorjclark Jul 12, 2019
804b24c
lint
connorjclark Jul 12, 2019
28dff4b
check fetch status
connorjclark Jul 12, 2019
789971f
let error be thrown from fetch
connorjclark Jul 12, 2019
1b35120
print errors
connorjclark Jul 12, 2019
6b64879
parellize
connorjclark Jul 12, 2019
ed0a2d0
add sourceMapUrl to artifact
connorjclark Jul 13, 2019
1200696
names is optional
connorjclark Jul 13, 2019
1fb9133
fewer try catches
connorjclark Jul 13, 2019
038cee2
forgot sourcemapurl
connorjclark Jul 13, 2019
c582b13
filter
connorjclark Jul 13, 2019
7dcd360
sourceMapUrl undefined if data uri
connorjclark Jul 15, 2019
331df48
more isSourceMapADataUri
connorjclark Jul 17, 2019
0b84c2e
elide. type sourceMapOrError
connorjclark Jul 18, 2019
7979c04
Update lighthouse-core/gather/gatherers/source-maps.js
connorjclark Jul 18, 2019
14613d0
fix resolving
connorjclark Jul 23, 2019
06bd78a
_retrieveMapFromScriptParsedEvent
connorjclark Jul 23, 2019
4777243
throw for less tsc
connorjclark Jul 23, 2019
574801e
put it back
connorjclark Jul 23, 2019
c84e2ae
types
connorjclark Jul 23, 2019
6a29f0d
rm testArgs
connorjclark Jul 23, 2019
2a46c4e
scriptParsedEvent
connorjclark Jul 23, 2019
416fe3a
resolvedSourceMapUrl
connorjclark Jul 23, 2019
bf0293c
doc
connorjclark Jul 23, 2019
ac943f6
Merge remote-tracking branch 'origin/master' into gather-source-maps
connorjclark Jul 24, 2019
755ac81
brendan changes
connorjclark Jul 25, 2019
ba09eff
Merge branch 'gather-source-maps' of github.com:GoogleChrome/lighthou…
connorjclark Jul 25, 2019
052e86b
Merge remote-tracking branch 'origin/master' into gather-source-maps
connorjclark Jul 25, 2019
fe24ee2
json, do not use gatherer for css lol
connorjclark Jul 25, 2019
a4aef48
proto
connorjclark Jul 25, 2019
bbaafbd
update
connorjclark Jul 25, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

Next

source maps gather first pass

  • Loading branch information...
connorjclark committed Jun 1, 2019
commit 3ce360279e570d12434da2ba85dc57b5e54e2118
@@ -10,8 +10,10 @@ const NetworkAnalyzer = require('../../lib/dependency-graph/simulator/network-an
const NetworkRequest = require('../../lib/network-request.js');
const getElementsInDocumentString = require('../../lib/page-functions.js').getElementsInDocumentString; // eslint-disable-line max-len
const pageFunctions = require('../../lib/page-functions.js');
const Driver = require('../driver.js'); // eslint-disable-line no-unused-vars
const LHError = require('../../lib/lh-error.js');

/* global getNodePath */
/* global getNodePath fetch */

/**
* @return {LH.Artifacts['ScriptElements']}
@@ -37,10 +39,85 @@ function collectAllScriptElements() {
});
}

/**
* This function is careful not to parse the response as JSON, as it will
* just need to be serialized again over the protocol, and source maps can
* be huge.
*
* If an error occurs, the first character is '!'.
*
* @param {string[]} urls
* @return {Promise<string[]>}
*/
async function fetchSourceMaps(urls) {
const responses = urls.map(async (url) => {
try {
const response = await fetch(url);
return response.text();
} catch (err) {
return '!' + err.toString();
}
});
return Promise.all(responses);
}

/**
* @fileoverview Gets JavaScript file contents.
*/
class ScriptElements extends Gatherer {
constructor() {
super();
/** @type {LH.Crdp.Debugger.ScriptParsedEvent[]} */
this._scriptParsedEvents = [];
}

/**
* @param {Driver} driver
* @param {string[]} urls
* @return {Promise<LH.Artifacts.SourceMap[]>}
*/
async fetchSourceMaps(driver, urls) {
const urlsParam = JSON.stringify(urls);
// TODO: change default protocol timeout?
// driver.setNextProtocolTimeout(250);
/** @type {string[]} */
const sourceMapJsons =
await driver.evaluateAsync(`(${fetchSourceMaps.toString()})(${urlsParam})`);

/** @type {LH.Artifacts.SourceMap[]} */
const sourceMaps = [];
for (const json of sourceMapJsons) {
if (json.startsWith('!')) {
sourceMaps.push({error: json.substring(1)});
continue;
}

try {
sourceMaps.push(JSON.parse(json));
} catch (error) {
// Without this catch, this silently fails and the gatherer returns an empty object... why no visible error?
sourceMaps.push({error});
}
}
return sourceMaps;
}

/**
* @param {LH.Crdp.Debugger.ScriptParsedEvent} event
*/
onScriptParsed(event) {
this._scriptParsedEvents.push(event);
}

/**
* @param {LH.Gatherer.PassContext} passContext
*/
async beforePass(passContext) {
const driver = passContext.driver;
driver.on('Debugger.scriptParsed', this.onScriptParsed.bind(this));
await driver.sendCommand('Debugger.enable');
}

/**
* @param {LH.Gatherer.PassContext} passContext
* @param {LH.Gatherer.LoadData} loadData
@@ -91,6 +168,36 @@ class ScriptElements extends Gatherer {
} catch (e) {}
}

driver.off('Debugger.scriptParsed', this.onScriptParsed);
await driver.sendCommand('Debugger.disable');

/** @type {Array<{script: LH.Artifacts.ScriptElement, sourceMapURL: string}>} */
const toFetch = [];
for (const event of this._scriptParsedEvents) {
if (event.sourceMapURL) {
const script = scripts.find(script => script.src === event.url);
if (!script) continue;
if (event.sourceMapURL.startsWith('data:')) {
const buffer = new Buffer(event.sourceMapURL.split(',')[1], 'base64');
script.sourceMap = JSON.parse(buffer.toString());
} else {
toFetch.push({script, sourceMapURL: event.sourceMapURL});
}
}
}

try {
const sourceMaps = await this.fetchSourceMaps(driver, toFetch.map(obj => obj.sourceMapURL));
for (const [i, sourceMap] of sourceMaps.entries()) {
toFetch[i].script.sourceMap = sourceMap;
}
} catch (err) {
// If we timeout, we timeout.
if (err.code !== LHError.errors.PROTOCOL_TIMEOUT) {
throw err;
}
}

return scripts;
}
}
@@ -129,6 +129,7 @@
"prettier": "^1.14.3",
"pretty-json-stringify": "^0.0.2",
"puppeteer": "^1.10.0",
"source-map": "^0.7.3",

This comment has been minimized.

Copy link
@brendankenny

brendankenny Jun 18, 2019

Member

seems unused except for the types? Remove until we need it?

This comment has been minimized.

Copy link
@connorjclark

connorjclark Jun 18, 2019

Author Collaborator

done

"typescript": "3.2.2",
"uglify-es": "3.0.15",
"url-search-params": "0.6.1",
@@ -207,12 +207,16 @@ declare global {
devtoolsNodePath: string;
/** Where the script was discovered, either in the head, the body, or network records. */
source: 'head'|'body'|'network'
/** The ID of the network request that matched the URL of the src or the main document if inline, null if no request could be found. */
sourceMap?: SourceMap;
/** The content of the inline script or the network record with the matching URL, null if the script had a src and no network record could be found. */
content: string | null
/** The ID of the network request that matched the URL of the src or the main document if inline, null if no request could be found. */
requestId: string | null
}

export type SourceMap = { map: import('source-map').RawSourceMap } | { error: string }

/** @see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/a#Attributes */
export interface AnchorElement {
rel: string
@@ -7592,6 +7592,11 @@ source-map@^0.6.0, source-map@^0.6.1, source-map@~0.6.1:
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.6.1.tgz#74722af32e9614e9c287a8d0bbde48b5e2f1a263"
integrity sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==

source-map@^0.7.3:
version "0.7.3"
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.7.3.tgz#5302f8169031735226544092e64981f751750383"
integrity sha512-CkCj6giN3S+n9qrYiBTX5gystlENnRW5jZeNLHpe6aue+SrHcG5VYwujhW9s4dY31mEGsxBDrHR6oI69fTXsaQ==

source-map@~0.5.1:
version "0.5.6"
resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.5.6.tgz#75ce38f52bf0733c5a7f0c118d81334a2bb5f412"
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.