Some websites block scrapers/bots, so policy content isn't retrieved successfully

[joshwallerr]

I'll edit background.js to use Zyte API for content grabbing, instead of JS fetch API

[joshwallerr]

This is nearly finished. Update coming within days.

[joshwallerr]

The issue I've ran into, is that Zyte API browser requests can take around 30 seconds to process - way too long for production.

The plan now is to fetch the html by default with the Javascript fetch API, and if the returned page text is smaller than a certain amount of characters (meaning the page wasn't loaded, and a small error message was shown), then we will use Zyte API.

It will significantly extend response times if a site does require javascript, but I think it's worth it:

A successful summary with an extra 30 second wait > An unsuccessful summary that saves 30 seconds