Prevent being blacklisted and/or report exceptions

[jochemstoel]

As the respective page layout may change, this approach is pretty fragile. Keep in mind that the respective platform may rate-limit / blacklist you in case of "excessive" usage.
Source: readme

I create unofficial APIs using request and cheerio all the time in the exact same way. What you can do to prevent getting blacklisted is cache the response body to the local file system and set a minimum delay time required before fetching it again. If cache is less than x period old, resolve local file and otherwise fetch it again and overwrite. This also protects the user if he or she is stupid and the script accidentally gets executed in a loop.

You don't want to periodically check if all of your modules like this 'still work'. You can implement a sort of event handler/callback at the end of your HTTP request logic that will report back to you if something failed. In other words, if successful do nothing but otherwise send the error somewhere using webhooks or websockets or something. Whether you want this depends on the situation or should be optional I guess but in this way when something changes that breaks your script, you will know about it in seconds from all users and applications using it.