Search Engine Land: Scripting and Scraping with Headless Chrome

Revisions: SMX Master Class

• Tested and Updated Scripts and dependencies
• Includes updated slides

Table of Contents

• Getting Started
  • Prerequisites
  • Installation
• Usage
• License
• Contact

Getting Started

To get a local copy up and running follow these simple steps.

Prerequisites

You will need to have node and npm setup and installed on your machine

Installation

1. Clone the sel-puppeteer

git clone https://github.com/johnmurch/sel-puppeteer.git

2. Install NPM packages

npm install

3. There is no step 3 :) should be good to go!

Usage

These scripts were used during the presentation (or generated) to demo Puppeteer and showcase some basic use cases for SEO.

e.g.

• visit a website
• extract page source and DOM
• get Page Title and Meta Description
• view console of a website
• Take screenshots
• View as Mobile
• Keyboard Interaction

├── basic-puppeteer-scripts
│   ├── ss # dump screenshots from screenshot.js
│   │   ├── amazon.jpg
│   │   ├── screenshot-clip.jpg
│   │   ├── screenshot-full.png
│   │   ├── screenshot-top.png
│   ├── basic.js
│   ├── basic-dom-vs-source.js
│   ├── basic-title-meta.js
│   ├── console.js
│   ├── screenshot.js
│   ├── visit.js
│   ├── visit-headless.js
│   ├── visit-mobile.js
│   ├── wysiwyg-amazon.js
├── cli # bash to run chrome headless
│   ├── dump-dom.sh
│   ├── generate-screenshot.sh
│   ├── screenshot.png
├── data # data for bot-a-quick-demo.js, bot-backlinks.js, bot-xtra-gsc.js, bot-redirect.js
│   ├── backlinks.csv
│   ├── demo.csv
│   ├── http___www.johnmurch.com_-Latest links-2020-07-15.csv
│   ├── redirect.txt
├── report # generated from bot-backlinks.js, bot-xtra-gsc.js, bot-redirect.js, dom-vs-source-save.js
│   ├── backlinks.csv
│   ├── dom.html
│   ├── gsc-backlinks.csv
│   ├── redirects.csv
│   ├── source.html
├── bot-a-quick-demo.js
├── bot-backlinks.js
├── bot-redirect.js
├── bot-xtra-gsc.js
├── index.js # 🍻
├── LICENSE
├── logo.png
├── package.json
├── package-lock.json
├── README.md
└── .gitignore

node basic-puppeteer-scripts/console.js # example of how to run

bot-backlinks.js - data/backlinks.csv

node bot-backlinks.js # example of how to run

bot-redirect.js - data/redirect.txt

node bot-redirect.js # example of how to run

bot-xtra-gsc.js - Contains a subset of data data/http*__www.johnmurch.com_-Latest links-2020-07-15.csv

node bot-xtra-gsc.js # example of how to run

License

Distributed under the MIT License. See LICENSE for more information.

Contact

John Murch - @johnmurch