CSS Scraper improvements

[bartveneman]

Planned changes for 2027

Area	2026	2027
Scraper	Static-HTML fetch	Headless browser (see below)
URL list	Majestic Million	Same dataset Google uses (also raised independently in a podcast conversation with Igalia)
Analyzer	@projectwallace/css-analyzer v7.6.3	Newer version (TBD)
Scraping behavior	—	Expected to differ, not yet detailed

Headless scraping, no hard Playwright dependency

Build the scraper as a library that accepts a browser context as input,
rather than depending directly on Playwright. That way it can run against a
hosted headless-browser vendor (e.g. Browserless) instead of only running
Playwright in-process.

Why go headless at all:

1. Unlocks CSS coverage analysis (used vs. unused CSS) — a library for
   this already exists in-house.
2. Needed for adoptedStyleSheets access — constructed stylesheets
   adopted into shadow roots via JS, invisible to static-HTML scraping.
3. Allows us to run Wappalyzer to detect use of frameworks, which may help during analysis later on.

Scraper pipeline (proposed, multi-stage)

1. Fetch the target URL, verify it responds successfully.
2. In parallel: fetch robots.txt and check robots meta tags to confirm
   scraping is actually allowed. Sites that opt out via robots directives
   are skipped entirely.
3. If allowed: load the page, start CSS coverage, and collect all CSS
   regardless of origin — linked stylesheets, <style> tags, inline
   style= attributes, adoptedStyleSheets, and @import-ed stylesheets.
4. Run tech detection (CMS/framework identification, via
   Wappalyzer) alongside the CSS collection.
5. Store one entry per URL in the scraping database, with every CSS piece
   found tagged by its origin type (link / adopted-stylesheet /
   style-tag / inline).

Also capture inline styles

Not just linked/embedded stylesheets — style="..." attributes too. Still
need to decide how inline styles are represented in the analysis: merged
into the main CSS corpus, or tracked separately, since they behave
differently (always maximum specificity, no selectors).

Running Playwright at scale

This is probably going to run on my personal laptop for a couple of weeks, but let's try to make it agnostic so that people can run it in GitHub actions using a service like browserless or whatever .

[bartveneman]

Use URI's from the Chrome UX report, as the Web Almanac does as well: https://almanac.httparchive.org/en/2025/methodology#chrome-ux-report

[bartveneman]

Need to decide if we want to crawl secondary pages as well, like Google does. This was also mentioned in the Igalia podcast episode.