Document parsing and returning content (scraping) using the HTMLRewriter

[kristianfreeman]

in the workers video that wes bos put out yesterday, he was looking for a tool to parse some HTML coming in via response, and return a subsection of it as the new response inside of the worker. he ultimately did some manual string parsing, but it turns out this would have been a perfect use-case for htmlrewriter's parser!

i synced up with wes and gave him my example code, which sets up an htmlrewriter looking for an img with a specific id, and setting the img src as part of the worker's local request state. if that state is found (an image has been parsed), it gets returned directly, otherwise, the worker falls back to the original request:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

const BASE = 'wes.io'

function buildURL(url) {
  const urlObject = new URL(url)
  const newURL = urlObject.href.replace(urlObject.host, BASE)
  return newURL
}

class ImageHandler {
  element(element) {
    this.foundImage = element.getAttribute('src')
  }
}

async function handleRequest(request) {
  const parts = request.url.split('/')
  
  if (parts.length !== 5 || !parts[4].includes('content')) {
    return fetch(buildURL(request.url))
  }

  const dropURL = parts.slice(0, 4).join("/")
  const resp = await fetch(buildURL(dropURL))
  const handler = new ImageHandler()
  const body = await new HTMLRewriter()
    .on('img#image-item', handler)
    .transform(resp)
    .text()

  if (handler.foundImage) {
    return fetch(handler.foundImage)
  } else {
    return new Response(body, resp)
  }
}

this is a super useful feature of the htmlrewriter that we don't have any docs for! i synced up with @harrishancock to make sure there wasn't a better way to do this – maybe long-term we can propose an API improvement to handle this use-case, but for now, this should probably be added as a template and talked more generally about in the reference part of the docs ("Parsing and returning data from a response using HTMLRewriter")

[kristianfreeman]

cc @adamschwartz who has built a scraper tool using HTML Rewriter, and may have thoughts on how to present this info well

[adamschwartz]

@signalnerve thx for letting me know. Your solution is great! We should definitely document this.

As of today, Web Scraper only knows how to return the text content of the nodes matching the selector you set. However I’ve always wanted to add a way to scrape attributes as well, and this was a great reminder of that.

With that framing, maybe we would consider documenting this use-case just as you have above, but with slightly different naming. Something like this:

class AttributeScraper {
  constructor(attribute) {
    this.attribute = attribute
  }

  element(element) {
    this.value = element.getAttribute(this.attribute)
  }
}

const scraper = new AttributeScraper(attribute)
const body = await new HTMLRewriter()
  .on(selector, scraper)
  .transform(response)
  .text()

if (scraper.value) {
  // ...
}

Then from Wes’s original example, selector and attribute could be 'img#image-item' and 'src' (as you have) or 'meta[property="og:image:secure_url"]' and 'content' (as he had in his video).

[harrishancock]

Couple quick notes: prefer .arrayBuffer() to drive the parser, to avoid redundant text decoding/encoding. The Response constructor will later handle the resulting ArrayBuffer just fine.

Also, buildUrl() could probably use urlObject.host = BASE to mutate the URL in place. Though it doesn't matter too much.

[adamschwartz]

Last night I took a pass at refactoring Web Scraper to make it easier to support attribute scraping as well as text content scraping. I ended up with a Scraper class which mostly just adds some syntactic sugar, but I found it helpful in the end, and others may too.

Here’s how it’s ultimately used:

async function handleAPIRequest({ url, selector, attr, spaced, pretty }) {
  let scraper, result

  try {
    scraper = await new Scraper().fetch(url)
  } catch (error) {
    return generateErrorJSONResponse(error, pretty)
  }

  try {
    if (!attr) {
      result = await scraper.querySelector(selector).getText({ spaced })
    } else {
      result = await scraper.querySelector(selector).getAttribute(attr)
    }

  } catch (error) {
    return generateErrorJSONResponse(error, pretty)
  }

  return generateJSONResponse({ result }, pretty)
}

Source

Currently the API is such that for each new Scraper().fetch(url), you can only .getText() or .getAttribute(attr) on it once.

I believe this is because once HTML Rewriter has transformed the response it cannot be transformed again. However I have not tried looking very far into this.

Ultimately there may be only so much we can do to improve the API when HTML Rewriter fundamentally works differently from the browser engine (which allows for unlimited querying of an in-memory DOM by selector).

One thought I had to allow for multiple queries is to change the API slightly by adding something like an execute().

So rather than doing this:

const result = await scraper.querySelector(selector).getText()

(And therefore being limited to only one getText() per scraper instance.)

Instead you would do something like this:

// Scraper prepares multiple ElementHandlers
scraper.querySelector(selectorA).getText()
scraper.querySelector(selectorB).getText()
scraper.querySelector(selectorC).getAttribute(attr)

// HTML Rewriter transforms the response and returns results for all .getText() and .getAttribute(attr) calls.
const result = await scraper.execute()

One downside of this approach is that the single query case would them become this:

const result = await scraper.querySelector(selector).getText().execute()

In order to avoid having to call .execute() in this case, the default behavior could be that all .getText() or .getAttribute() calls get executed automatically unless an option is passed into the constructor. Something like this:

const scraper = await new Scraper({ multiple: true }).fetch(url)

As this is uncharted territory, and the mechanics and asynchronous nature of this stuff is pretty confusing to me, I’d certainly appreciate any thoughts from @signalnerve and @harrishancock on this.

---

Despite the drawbacks of the approach, I will say that for single queries the API is pretty satisfying to use. For example, for Wes’s original use case, here’s all he’d have to do after importing the Scraper class:

async function handleRequest(request) {
  const parts = request.url.split('/')
  
  if (parts.length !== 5 || !parts[4].includes('content')) {
    return fetch(buildURL(request.url))
  }

  const dropURL = parts.slice(0, 4).join("/")
  const scraper = await new Scraper().fetch(buildURL(dropURL))
  const result = await scraper.querySelector('img#image-item').getAttribute('src')

  if (result && result.result) {
    return fetch(result.result)
  } else {
    return new Response(body, resp)
  }
}

[harrishancock]

Very cool!

I'll have to dig into it more later, but one thought I had: what about getting rid of querySelector() and instead provide an API like scraper.getText(selector1).getAttribute(selector2, attrName).execute()? Simply to cut down on the number of calls required to set up the scraper.