Chapter 5: Add first third party scripting query

[patrickhulce]

The first third party query for the scripting time analysis.

ref #86

[patrickhulce]

I'm almost positive you know a better way to do this @rviscomi than the copypasta I have here. My stumbling block was always needing a .map function in order to flatten this result and match it with the join index

[rviscomi]

Ahhhh yeah BigQuery's JSON function don't play well with arrays. For best results I recommend a JS UDF to do the parsing and data processing.

Here's an example of a query that creates a UDF, which parses the JSON payload, preprocesses it, and hands it back to the SQL for aggregation: https://github.com/HTTPArchive/almanac.httparchive.org/blob/master/sql/2019/03_Markup/03_03c.sql

[rviscomi]

#standardSQL
# Percentage of script execution time that are from third party requests broken down by third party category.
CREATE TEMPORARY FUNCTION getExecutionTimes(report STRING)
RETURNS ARRAY<STRUCT<url STRING, execution_time FLOAT64>> LANGUAGE js AS '''
try {
  var $ = JSON.parse(report);
  return $.audits['bootup-time'].details.items.map(item => ({
    url: item.url,
    execution_time: item.scripting
  }));
} catch (e) {
  return [];
}
''';

SELECT
  category AS third_party_category,
  SUM(item.execution_time) AS total_execution_time,
  ROUND(SUM(item.execution_time) * 100 / SUM(SUM(item.execution_time)) OVER (), 4) AS pct_execution_time
FROM
  `httparchive.lighthouse.2019_07_01_mobile`,
  UNNEST(getExecutionTimes(report)) AS item
LEFT JOIN
  `lighthouse-infrastructure.third_party_web.2019_07_01`
ON
  NET.HOST(item.url) = domain
GROUP BY
  third_party_category
ORDER BY
  pct_execution_time DESC

(65.3s elapsed, 1.22 TB processed)

third_party_category	total_execution_time	pct_execution_time
	3.42E+09	48.3414
ad	1.01E+09	14.2466
social	4.75E+08	6.7085
hosting	4.70E+08	6.6399
video	4.59E+08	6.4946
analytics	3.46E+08	4.8858
cdn	3.16E+08	4.4615
utility	2.76E+08	3.9068
tag-manager	1.02E+08	1.4358
customer-success	1.01E+08	1.4253
marketing	5.49E+07	0.7756
content	3.48E+07	0.4925
other	1.32E+07	0.1859

Could probably remove the total field, doesn't seem to be very insightful.

Hope that helps.

[rviscomi]

For reference here are the results from your original query:

(448.7s elapsed, 1.22 TB processed)

thirdPartyCategory	totalExecutionTime	percentExecutionTime
	3.42E+09	48.3633
ad	1.01E+09	14.2296
social	4.74E+08	6.6972
hosting	4.70E+08	6.6429
video	4.59E+08	6.4984
analytics	3.45E+08	4.8847
cdn	3.16E+08	4.4631
utility	2.76E+08	3.9067
tag-manager	1.01E+08	1.4348
customer-success	1.01E+08	1.4261
marketing	5.48E+07	0.7757
content	3.48E+07	0.4919
other	1.31E+07	0.1858

Identical results for the most part. The slight differences are probably attributed to pages with 35+ items.

[patrickhulce]

oh thanks Rick! didn't expect a full query from you thank you very much for the help :)

[rviscomi]

#standardSQL
# Percentage of script execution time that are from third party requests broken down by third party category.
CREATE TEMPORARY FUNCTION getExecutionTimes(report STRING)
RETURNS ARRAY<STRUCT<url STRING, execution_time FLOAT64>> LANGUAGE js AS '''
try {
  var $ = JSON.parse(report);
  return $.audits['bootup-time'].details.items.map(item => ({
    url: item.url,
    execution_time: item.scripting
  }));
} catch (e) {
  return [];
}
''';

SELECT
  category AS third_party_category,
  SUM(item.execution_time) AS total_execution_time,
  ROUND(SUM(item.execution_time) * 100 / SUM(SUM(item.execution_time)) OVER (), 4) AS pct_execution_time
FROM
  `httparchive.lighthouse.2019_07_01_mobile`,
  UNNEST(getExecutionTimes(report)) AS item
LEFT JOIN
  `lighthouse-infrastructure.third_party_web.2019_07_01`
ON
  NET.HOST(item.url) = domain
GROUP BY
  third_party_category
ORDER BY
  pct_execution_time DESC

(65.3s elapsed, 1.22 TB processed)

third_party_category	total_execution_time	pct_execution_time
	3.42E+09	48.3414
ad	1.01E+09	14.2466
social	4.75E+08	6.7085
hosting	4.70E+08	6.6399
video	4.59E+08	6.4946
analytics	3.46E+08	4.8858
cdn	3.16E+08	4.4615
utility	2.76E+08	3.9068
tag-manager	1.02E+08	1.4358
customer-success	1.01E+08	1.4253
marketing	5.49E+07	0.7756
content	3.48E+07	0.4925
other	1.32E+07	0.1859

Could probably remove the total field, doesn't seem to be very insightful.

Hope that helps.