CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is 18 commits ahead of Smerity:master.
sebastian-nagel WATSampleOutlinks:
- prefix properties by `wat.outlinks.` (fix-up)
- allow to optionally extract and mark RSS and Atom feeds:
  (if property wat.outlinks.extract.feed is true,
   marker is configured by wat.outlinks.extract.feed.marker)
Latest commit 7f87e6a Mar 28, 2018

Common Crawl Logo

Common Crawl WARC Examples

This repository contains both wrappers for processing WARC files in Hadoop MapReduce jobs and also Hadoop examples to get you started.

There are three examples for Hadoop processing:

  • [WARC files] HTML tag frequency counter using raw HTTP responses
  • [WAT files] Server response analysis using response metadata
  • [WET files] Classic word count example using extracted text

All three assume initially that the files are stored locally in the data subdirectory but can be trivially modified to pull them down from Common Crawl's Amazon S3 bucket. To acquire the files, you can use AWS CLI, S3Cmd or similar.

mkdir data
s3cmd get s3://commoncrawl/crawl-data/CC-MAIN-2013-48/segments/1386163035819/warc/CC-MAIN-20131204131715-00000-ip-10-33-133-15.ec2.internal.warc.gz data/
s3cmd get s3://commoncrawl/crawl-data/CC-MAIN-2013-48/segments/1386163035819/wet/CC-MAIN-20131204131715-00000-ip-10-33-133-15.ec2.internal.warc.wet.gz data/


mkdir data
aws s3 --no-sign-request cp s3://commoncrawl/crawl-data/CC-MAIN-2013-48/segments/1386163035819/warc/CC-MAIN-20131204131715-00000-ip-10-33-133-15.ec2.internal.warc.gz data/
aws s3 --no-sign-request cp s3://commoncrawl/crawl-data/CC-MAIN-2013-48/segments/1386163035819/wet/CC-MAIN-20131204131715-00000-ip-10-33-133-15.ec2.internal.warc.wet.gz data/

To build and run:

mvn package
<path-to-hadoop>/bin/hadoop jar target/cc-warc-examples-0.2-SNAPSHOT-jar-with-dependencies.jar org.commoncrawl.examples.mapreduce.WETWordCount

All three examples place output in the directory /tmp/cc.


MIT License, as per LICENSE