Skip to content
MVP of a OpenWPM-based crawl setup for Webcompat analysis
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
crawl-prep @ 83a90e5

Webcompat Crawls

Configuration and instructions used to crawls top sites using a specially instrumented version of Firefox gathering data for Webcompat analysis.

Generate the seed list via a series of pre-crawls

See ./crawl-prep/

Run an OpenWPM crawl in Google Cloud Platform

See ./crawl-engineering/gcp/

Developer notes

To update the OpenWPM Crawler and crawl-prep submodules to the latest commits in the remotely tracked branches:

git submodule update --remote
You can’t perform that action at this time.