Custom metric for content-visibility #171
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tunetheweb
reviewed
Jul 1, 2025
dist/css.js
Outdated
Comment on lines
81
to
87
| const elements = document.querySelectorAll('*'); | ||
| const contentVisibilityElements = []; | ||
| const contentVisibilityValues = {}; | ||
|
|
||
| for (const element of elements) { | ||
| try { | ||
| const computedStyle = getComputedStyle(element); |
There was a problem hiding this comment.
@pmeenan any concerns with running getComputedStyle on every element in the DOM?
It sounds like a terrible idea for performance but custom metrics are executed off page load so I guess it doesn't really matter for those does it?
There was a problem hiding this comment.
I'm on the fence. I assume it will only force a layout once if one is pending and then can return the element position for everything else without having to recalculate anything expensive but doing anything on ALL page elements freaks me out a bit.
My query selector foo isn't great, but is there a way to query based on content-visibility and only retrieve the elements with the "interesting" content visibility types directly?
I'd recommend trying it on some extreme edge-case pages (like CNN) to see how bad it is.
There was a problem hiding this comment.
My query selector foo isn't great, but is there a way to query based on content-visibility and only retrieve the elements with the "interesting" content visibility types directly
@burakguneli did suggest doing this for inline styles. But that's very limited. We really need the calculated style, which won't be directly queryable.
The other option is to look at the parsed_css data. I'm not totally familiar with that to be honest and whether that's just "we have a selector that uses content-visibility but no idea if it's used?" or if it actually gives the usage. @burakguneli was going to look further in this.
There was a problem hiding this comment.
@tunetheweb @pmeenan I worked on an alternative way instead of doing expensive calls like getComputedStyle. I will send my commit and try to see if that works as expected
There was a problem hiding this comment.
I'm n ot sure what manually processing the CSS gains over using parsed_css (which basically does this already)?
There was a problem hiding this comment.
@tunetheweb thanks for checking and your feedback. I think you are right, the best way would be to use already parsed css. I'm kind of trying to learn how these custom metrics work, sorry for the many commits and notifications you are getting because of it. I'm not clear on one thing. Do you know how I can use the already parsed CSS in my custom metric file content_visibility.js?
There was a problem hiding this comment.
The idea would be you wouldn’t need this custom metric and instead would get the data you need by selecting from that table.
Try it out for last month for a single URL (make sure to bill the httparchive project).
As I said earlier the downside is that I think it only says if a selector exists using that property. Not how much it’s actually used (or even if at all!). But not familiar with that table so not 100% sure about that and maybe it is better than that.
There was a problem hiding this comment.
oh thanks I got it now! For that, I tried running a couple of queries on the 30th of June like that: https://console.cloud.google.com/bigquery?inv=1&invt=Ab19zQ&project=httparchive&ws=!1m10!1m4!4m3!1shttparchive!2ssample_data!3sparsed_css_10k!1m4!1m3!1shttparchive!2sbquxjob_69d0dc73_197c1eb5200!3sUS
but after running this query I got the impression that we don't have it and we need to create a custom metric. Maybe it was my bad and I couldn't write a proper SQL query. I will try to improve it and retry!
There was a problem hiding this comment.
This seems to work:
#standardSQL
CREATE TEMPORARY FUNCTION hasContentVisibility(css STRING)
RETURNS ARRAY<STRUCT<property STRING, freq INT64>>
LANGUAGE js
OPTIONS (library = "gs://httparchive/lib/css-utils.js")
AS '''
try {
var ast = JSON.parse(css);
let ret = {};
walkDeclarations(ast, ({property}) => {
// Strip hacks like *property, _property etc and normalize to lowercase
property = property.replace(/[^a-z-]/g, "").toLowerCase();
if (matches(property, 'content-visibility')) {
incrementByKey(ret, property);
}
});
return Object.entries(ret).map(([property, freq]) => {
return {property, freq};
});
} catch (e) {
return [];
}
''';
WITH totals AS (
SELECT
client,
COUNT(distinct root_page) AS total_pages
FROM
`httparchive.crawl.parsed_css`
WHERE
date = '2025-06-01' AND
rank <= 1000 AND -- remove to run on whole dataset
is_root_page
GROUP BY
client
),
content_visibility_pages AS (
SELECT
client,
COUNT(distinct root_page) AS pages_with_content_visibility
FROM
`httparchive.crawl.parsed_css`,
UNNEST (hasContentVisibility(css))
WHERE
date = '2025-06-01' AND
rank <= 1000 AND -- remove to run on whole dataset
is_root_page
GROUP BY
client
)
SELECT
totals.client,
IFNULL(content_visibility_pages.pages_with_content_visibility, 0) AS pages_with_content_visibility,
totals.total_pages,
ROUND(IFNULL(content_visibility_pages.pages_with_content_visibility, 0) * 100.0 / totals.total_pages, 2) AS pct_pages
FROM
totals
LEFT JOIN
content_visibility_pages
USING (client)
ORDER BY
totals.clientThere was a problem hiding this comment.
that works amazing thanks!!! @tunetheweb
https://almanac.httparchive.org/en/2022/Changed custom metrics values: {
"_content_visibility": {
"used": false,
"count": 0,
"values": [],
"uniqueValues": []
},
"_performance": {
"lcp_elem_stats": {
"startTime": 452,
"nodeName": "IMG",
"url": "https://almanac.httparchive.org/static/images/home-hero.png",
"size": 161505,
"loadTime": 311.19999999552965,
"renderTime": 452,
"attributes": [
{
"name": "src",
"value": "/static/images/home-hero.png"
},
{
"name": "alt",
"value": ""
},
{
"name": "width",
"value": "820"
},
{
"name": "height",
"value": "562"
},
{
"name": "fetchpriority",
"value": "high"
}
],
"boundingClientRect": {
"x": 866,
"y": 339.109375,
"width": 485,
"height": 332.390625,
"top": 339.109375,
"right": 1351,
"bottom": 671.5,
"left": 866
},
"naturalWidth": 820,
"naturalHeight": 562,
"styles": {
"background-image": "none",
"pointer-events": "auto",
"position": "relative",
"width": "485px",
"height": "332.391px"
},
"percentOfViewport": "0.173",
"cover90viewport": false
},
"raw_lcp_element": {
"nodeName": "IMG",
"attributes": [
{
"name": "src",
"value": "/static/images/home-hero.png"
},
{
"name": "alt",
"value": ""
},
{
"name": "width",
"value": "820"
},
{
"name": "height",
"value": "562"
},
{
"name": "fetchpriority",
"value": "high"
}
]
},
"lcp_resource": {
"id": "7555.10",
"sequence": 10,
"body": "/home/pmeenan/wptagent/work/wptagent-v6-manual-20241002-10.20.0.3/250705_NJ_4.1.0/bodies/7555.10",
"url": "https://almanac.httparchive.org/static/images/home-hero.png",
"status": 200,
"connectionId": 119,
"protocol": "h2",
"connectionReused": true,
"fromServiceWorker": false,
"timing": {
"requestTime": 63090.89679,
"proxyStart": -1,
"proxyEnd": -1,
"dnsStart": -1,
"dnsEnd": -1,
"connectStart": -1,
"connectEnd": -1,
"sslStart": -1,
"sslEnd": -1,
"workerStart": -1,
"workerReady": -1,
"workerFetchStart": -1,
"workerRespondWithSettled": -1,
"sendStart": 5.238,
"sendEnd": 6.073,
"pushStart": 0,
"pushEnd": 0,
"receiveHeadersStart": 58.957,
"receiveHeadersEnd": 59.121
},
"fromDiskCache": false,
"remoteIPAddress": "[2607:f8b0:4004:c19::79]",
"remotePort": 443,
"securityState": "secure",
"securityDetails": {
"protocol": "TLS 1.3",
"keyExchange": "",
"keyExchangeGroup": "X25519",
"cipher": "AES_128_GCM",
"certificateId": 0,
"subjectName": "almanac.httparchive.org",
"sanList": [
"almanac.httparchive.org"
],
"issuer": "WR3",
"validFrom": 1747802989,
"validTo": 1755582041,
"signedCertificateTimestampList": [
{
"status": "Verified",
"origin": "Embedded in certificate",
"logDescription": "Let's Encrypt 'Oak2025h2'",
"logId": "0DE1F2302BD30DC140621209EA552EFC47747CB1D7E930EF0E421EB47E4EAA34",
"timestamp": 1747806589725,
"hashAlgorithm": "SHA-256",
"signatureAlgorithm": "ECDSA",
"signatureData": "3046022100F01F1EEC48A8367228365F844D9F436AD4DFCAD4DC3FDC6B90B00C7DD4A7F643022100CD560F6DAE12D3EC7E93275CE003A21CAB46C8CB876AF4BF58B3BDCBBC8C8F9C"
},
{
"status": "Verified",
"origin": "Embedded in certificate",
"logDescription": "Google 'Xenon2025h2' log",
"logId": "DDDCCA3495D7E11605E79532FAC79FF83D1C50DFDB003A1412760A2CACBBC82A",
"timestamp": 1747806589823,
"hashAlgorithm": "SHA-256",
"signatureAlgorithm": "ECDSA",
"signatureData": "304502200BF49F0A1D1AAAC58AAAF49F63516A4AFA6C54FBC0966BA7484D62865680B3E20221008D8FABB31E8E2A36B30ECE8F9A6E183274DFCBC628DF5D2D5935125BF5381026"
}
],
"certificateTransparencyCompliance": "compliant",
"serverSignatureAlgorithm": 2052,
"encryptedClientHello": false
},
"fromPrefetchCache": false,
"response_headers": {
"cache-control": "public, max-age=10800",
"content-type": "image/png",
"date": "Sat, 05 Jul 2025 15:07:23 GMT",
"etag": "\"68ujDQ\"",
"expires": "Sat, 05 Jul 2025 18:07:23 GMT",
"server": "Google Frontend",
"strict-transport-security": "max-age=31556926; includeSubDomains",
"vary": "Accept-Encoding",
"x-cloud-trace-context": "a710665a85223f24ab1fb0119f14b67e",
"x-content-type-options": "nosniff"
},
"request_headers": {
":authority": "almanac.httparchive.org",
":method": "GET",
":path": "/static/images/home-hero.png",
":scheme": "https",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip, deflate, br, zstd",
"accept-language": "en-US,en;q=0.9",
"priority": "u=1, i",
"referer": "https://almanac.httparchive.org/en/2022/",
"sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"138\", \"Google Chrome\";v=\"138\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Unknown\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "same-origin",
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 PTST/250702.151337"
},
"initiator": {
"type": "parser",
"url": "https://almanac.httparchive.org/en/2022/",
"lineNumber": 1425,
"columnNumber": 99
},
"documentURL": "https://almanac.httparchive.org/en/2022/",
"timestamp": 63090.881636,
"frameId": "6674D54CF7950D3E78485340C9294147",
"hasUserGesture": false,
"type": "Image",
"wallTime": 1751728043.655701,
"initialPriority": "High",
"priority": "High",
"transfer_size": 38173
},
"is_lcp_statically_discoverable": true,
"is_lcp_preloaded": false,
"lcp_preload": [],
"web_vitals_js": [
"https://almanac.httparchive.org/static/js/web-vitals.js?v=8d892f61f132b9a5889c5811203fa593"
],
"gaming_metrics": [],
"speculation_rules": [
{
"prerender": [
{
"source": "document",
"where": {
"and": [
{
"href_matches": "/*"
},
{
"not": {
"href_matches": "/static/*"
}
}
]
},
"eagerness": "moderate"
}
]
}
]
}
}https://w3c.github.io/sustainableweb-wsgChanged custom metrics values: {
"_content_visibility": {
"used": false,
"count": 0,
"values": [],
"uniqueValues": []
},
"_performance": {
"lcp_elem_stats": {
"startTime": 264,
"nodeName": "P",
"url": "",
"size": 119250,
"loadTime": 0,
"renderTime": 264,
"attributes": [],
"boundingClientRect": {
"x": 495.5,
"y": 927.59375,
"width": 800,
"height": 216,
"top": 927.59375,
"right": 1295.5,
"bottom": 1143.59375,
"left": 495.5
},
"styles": {
"background-image": "none",
"pointer-events": "auto",
"position": "static",
"width": "800px",
"height": "216px"
},
"percentOfViewport": "0.00",
"cover90viewport": false
},
"raw_lcp_element": null,
"lcp_resource": null,
"is_lcp_statically_discoverable": true,
"is_lcp_preloaded": null,
"lcp_preload": null,
"web_vitals_js": [],
"gaming_metrics": [],
"speculation_rules": []
}
}https://www.hurriyet.com.trChanged custom metrics values: {
"_content_visibility": {
"used": false,
"count": 0,
"values": [],
"uniqueValues": []
},
"_performance": {
"lcp_elem_stats": {
"startTime": 1412,
"nodeName": "IMG",
"url": "https://image.hurimg.com/i/hurriyet/75/383x217/6868f7b6fa5b0aec3e4a1375.jpg",
"size": 52228,
"loadTime": 1149,
"renderTime": 1412,
"attributes": [
{
"name": "src",
"value": "https://image.hurimg.com/i/hurriyet/75/383x217/6868f7b6fa5b0aec3e4a1375.jpg"
},
{
"name": "alt",
"value": "Son dakika... CHPli 3 belediye başkanı gözaltına alındı"
},
{
"name": "title",
"value": "Son dakika... CHPli 3 belediye başkanı gözaltına alındı"
},
{
"name": "class",
"value": ""
},
{
"name": "width",
"value": "383"
},
{
"name": "height",
"value": "217"
},
{
"name": "pinger-seen",
"value": "true"
}
],
"boundingClientRect": {
"x": 200.015625,
"y": 493,
"width": 303.65625,
"height": 172.03125,
"top": 493,
"right": 503.671875,
"bottom": 665.03125,
"left": 200.015625
},
"naturalWidth": 383,
"naturalHeight": 217,
"styles": {
"background-image": "none",
"pointer-events": "auto",
"position": "static",
"width": "303.656px",
"height": "172.031px"
},
"percentOfViewport": "0.0562",
"cover90viewport": false
},
"raw_lcp_element": {
"nodeName": "IMG",
"attributes": [
{
"name": "src",
"value": "https://image.hurimg.com/i/hurriyet/75/383x217/6868f7b6fa5b0aec3e4a1375.jpg"
},
{
"name": "alt",
"value": "Son dakika... CHPli 3 belediye başkanı gözaltına alındı"
},
{
"name": "title",
"value": "Son dakika... CHPli 3 belediye başkanı gözaltına alındı"
},
{
"name": "class",
"value": ""
},
{
"name": "width",
"value": "383"
},
{
"name": "height",
"value": "217"
}
]
},
"lcp_resource": {
"id": "9076.24",
"sequence": 22,
"body": "/home/pmeenan/wptagent/work/wptagent-v6-manual-20241002-10.20.0.3/250705_9K_6.1.0/bodies/9076.24",
"url": "https://image.hurimg.com/i/hurriyet/75/383x217/6868f7b6fa5b0aec3e4a1375.jpg",
"status": 200,
"connectionId": 342,
"protocol": "h2",
"connectionReused": false,
"fromServiceWorker": false,
"timing": {
"requestTime": 63173.179138,
"proxyStart": -1,
"proxyEnd": -1,
"dnsStart": 0.369,
"dnsEnd": 77.149,
"connectStart": 80.806,
"connectEnd": 336.133,
"sslStart": 136.183,
"sslEnd": 336.125,
"workerStart": -1,
"workerReady": -1,
"workerFetchStart": -1,
"workerRespondWithSettled": -1,
"sendStart": 336.713,
"sendEnd": 337.302,
"pushStart": 0,
"pushEnd": 0,
"receiveHeadersStart": 522.059,
"receiveHeadersEnd": 522.27
},
"fromDiskCache": false,
"remoteIPAddress": "51.81.107.96",
"remotePort": 443,
"securityState": "secure",
"securityDetails": {
"protocol": "TLS 1.3",
"keyExchange": "",
"keyExchangeGroup": "X25519",
"cipher": "AES_256_GCM",
"certificateId": 0,
"subjectName": "*.hurimg.com",
"sanList": [
"*.hurimg.com",
"hurimg.com"
],
"issuer": "GlobalSign GCC R6 AlphaSSL CA 2025",
"validFrom": 1750177266,
"validTo": 1784478065,
"signedCertificateTimestampList": [
{
"status": "Verified",
"origin": "Embedded in certificate",
"logDescription": "Let's Encrypt 'Oak2026h2'",
"logId": "ACAB30706CEBEC8431F413D2F4915F111E422443B1F2A68C4F3C2B3BA71E02C3",
"timestamp": 1750177269146,
"hashAlgorithm": "SHA-256",
"signatureAlgorithm": "ECDSA",
"signatureData": "3045022100D7B5DC5DC225E85258AA5413DA4767DC9F0588E624DBBE95A8242C64FF4440E102202CECEA2E9615F91421911BE9B5B4C132D0219789BEB9572DC63222E5979093F3"
},
{
"status": "Verified",
"origin": "Embedded in certificate",
"logDescription": "DigiCert 'Wyvern2026h2'",
"logId": "C2317E574519A345EE7F38DEB29041EBC7C2215A22BF7FD5B5AD769AD90E52CD",
"timestamp": 1750177269137,
"hashAlgorithm": "SHA-256",
"signatureAlgorithm": "ECDSA",
"signatureData": "3045022100E3775048DAAA429FA9D256FDCE00A1183F612F4353DC9727A3CD088461098E0F022011EC5EE1A7F34EDB69F41E8C2E66DF1549A87A06D3BC85C58A165C983E943A54"
},
{
"status": "Verified",
"origin": "Embedded in certificate",
"logDescription": "Sectigo 'Mammoth2026h2'",
"logId": "94B1C18AB0D057C47BE0AC040E1F2CBC8DC375727BC951F20A526126863BA73C",
"timestamp": 1750177269163,
"hashAlgorithm": "SHA-256",
"signatureAlgorithm": "ECDSA",
"signatureData": "3045022100B9CD82E1EE09EA014253F65AA2F21A5ED2BE4F8426ADC8D4A9314295CB62692F02206963AB0EBA3376FDA8DAB6C8B4DDC22463D8F01C4E81F1F683E1E0818B492FBA"
}
],
"certificateTransparencyCompliance": "compliant",
"serverSignatureAlgorithm": 2052,
"encryptedClientHello": false
},
"fromPrefetchCache": false,
"response_headers": {
"accept-ranges": "bytes",
"age": "18370",
"allow": "GET, HEAD",
"cache-control": "max-age=31556926",
"content-length": "40515",
"content-type": "image/gif",
"date": "Sat, 05 Jul 2025 15:08:42 GMT",
"etag": "W/\"14c2312bc666441be5bdbdb5cb149763\"",
"last-modified": "Sat, 05 Jul 2025 10:00:23 GMT",
"server": "MerlinCDN",
"via": "HTTP/2.0 Merlin CDN",
"x-amz-request-id": "tx00000e2463161d392e4ea-006868f821-f3f5925-eu-tr",
"x-cache-status": "HIT",
"x-edge": "us-vga-ovc-s04",
"x-midtier": "nl-naw4-ws-s35",
"x-rgw-object-type": "Normal"
},
"request_headers": {
":authority": "image.hurimg.com",
":method": "GET",
":path": "/i/hurriyet/75/383x217/6868f7b6fa5b0aec3e4a1375.jpg",
":scheme": "https",
"accept": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
"accept-encoding": "gzip, deflate, br, zstd",
"accept-language": "en-US,en;q=0.9",
"priority": "u=2, i",
"referer": "https://www.hurriyet.com.tr/",
"sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"138\", \"Google Chrome\";v=\"138\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Unknown\"",
"sec-fetch-dest": "image",
"sec-fetch-mode": "no-cors",
"sec-fetch-site": "cross-site",
"sec-fetch-storage-access": "active",
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 PTST/250702.151337"
},
"initiator": {
"type": "parser",
"url": "https://www.hurriyet.com.tr/",
"lineNumber": 0,
"columnNumber": 63787
},
"documentURL": "https://www.hurriyet.com.tr/",
"timestamp": 63173.161553,
"frameId": "9A99A967785C795FB674E3C6172E4929",
"hasUserGesture": false,
"type": "Image",
"wallTime": 1751728125.93562,
"initialPriority": "Medium",
"priority": "High",
"transfer_size": 40903
},
"is_lcp_statically_discoverable": true,
"is_lcp_preloaded": false,
"lcp_preload": [],
"web_vitals_js": [],
"gaming_metrics": {
"fidIframeOverlaySoft": false
},
"speculation_rules": []
}
} |
|
This implementation is unnecessary so closing the PR |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds support for tracking and analyzing elements using the content-visibility CSS property, which is critical for performance optimization analysis. The content-visibility property allows developers to skip rendering of off-screen content, significantly improving page load performance and user experience.
Changes
Example Output
Test websites: