This repository contains a group of templates for performing web scraping with nodejs, these templates are designed to be used in a Google Cloud environment as a Google Cloud function in a node.js environment, however you can certainly take this code and tweak it to work in different environments.
The three main templates this repository contains is as follows:
- Web Scraping from cloudflare sites
- Web Scraping from regular sites
- Web Scraping from sites that require client side JavaScript to run
You can find more information about these templates at my blog including detailed guides to get them setup at Lukes Toolkit