domain-resurrect

Test locally

To test locally, download some WAT files from CommonCrawl into the data directory. For example, look into a snapshot and download the wat.paths.gz.

Then, get some paths from there:

gunzip wat.paths.gz | head

and download them into the data directory:

cd data
wget https://data.commoncrawl.org/${WAT_PATH}

After the data is available, you can more quickly test by running hack/run-local.sh.