website-scraper-puppeteer

Plugin for website-scraper which returns html for dynamic websites using puppeteer.

Maintenance of this project is made possible by all the contributors and sponsors. If you'd like to sponsor this project and have your avatar or company logo appear below click here. 💖

• nodejs version >= 20
• website-scraper version >= 5

npm install website-scraper website-scraper-puppeteer

Puppeteer normally downloads a compatible version of Chrome automatically during installation via a postinstall script. Starting with npm v12, lifecycle scripts (including postinstall) are disabled by default, so this automatic download no longer runs. You may then see an error like Could not find Chrome (ver. ...) when scraping.

If you use npm v12 or newer (or another package manager that blocks install scripts, e.g. recent pnpm), install Chrome manually after installing the packages:

npx puppeteer browsers install chrome

Alternatives:

• Re-enable install scripts by adding puppeteer to the allowScripts field in your package.json (or .npmrc), so the browser is downloaded automatically again.
• Use a Chrome/Chromium already installed on your system by pointing Puppeteer to it via launchOptions.executablePath (or the PUPPETEER_EXECUTABLE_PATH environment variable).

See Puppeteer's guide on why automatic downloads can be blocked for more details.

import scrape from 'website-scraper';
import PuppeteerPlugin from 'website-scraper-puppeteer';

await scrape({
    urls: ['https://www.instagram.com/gopro/'],
    directory: '/path/to/save',
    plugins: [ 
      new PuppeteerPlugin({
        launchOptions: { headless: "new" }, /* optional */
        gotoOptions: { waitUntil: "networkidle0" }, /* optional */
        scrollToBottom: { timeout: 10000, viewportN: 10 }, /* optional */
      })
    ]
});

Puppeteer plugin constructor accepts next params:

• launchOptions - (optional) - puppeteer launch options, can be found in puppeteer docs
• gotoOptions - (optional) - puppeteer page.goto options, can be found in puppeteer docs
• scrollToBottom - (optional) - in some cases, the page needs to be scrolled down to render its assets (lazyloading). Because some pages can be really endless, the scrolldown process can be interrupted before reaching the bottom when one or both of the bellow limitations are reached:
  • timeout - in milliseconds
  • viewportN - viewport height multiplier

It starts Chromium in headless mode which just opens page and waits until page is loaded. It is far from ideal because probably you need to wait until some resource is loaded or click some button or log in. Currently this module doesn't support such functionality.