nutch

forked from Aloisius/nutch

CommonCrawl Test version of Nutch

Updated Aug 17, 2016

ia-web-commons

forked from Aloisius/ia-web-commons

Updated Aug 7, 2016

HTML 0 4

cc-index-server

forked from ikreymer/cc-index-server

Deployment of pywb as a CommonCrawl Index Server

Updated Aug 5, 2016

commoncrawl-examples

A library of examples showing how to use the Common Crawl corpus.

Updated Aug 5, 2016

Python 0 3

cc-quick-scripts

forked from Smerity/cc-quick-scripts

Useful scripts for attacking the CommonCrawl dataset and WARC/WET/WAT files

Updated Aug 4, 2016

Python 0 0

cc-crawl-statistics

Updated Jul 28, 2016

Python 0 4

webarchive-indexing

forked from ikreymer/webarchive-indexing

Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

Updated May 17, 2016

Java 0 9

Teneo

forked from Smerity/Teneo

Updated Apr 13, 2016

cc-warc-examples

forked from Smerity/cc-warc-examples

CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

Updated Jan 12, 2016

example-warc-java

Updated Dec 22, 2015

Python 15 14

gzipstream

forked from Smerity/gzipstream

gzipstream allows Python to process multi-part gzip files from a streaming source

Updated Oct 8, 2015

Python 63 48

cc-mrjob

forked from Smerity/cc-mrjob

Demonstration of using Python to process the Common Crawl dataset with the mrjob framework

Updated Jul 30, 2015

Python 1 92

python-hadoop

forked from bityon/python-hadoop

python-hadoop

Updated Jul 28, 2015

hadoop-s3a

forked from Aloisius/hadoop-s3a

An AWS SDK-backed FileSystem driver for Hadoop

Updated Jul 8, 2014

ia-hadoop-tools

forked from Aloisius/ia-hadoop-tools

Updated Jun 27, 2014

aws-sdk-java

forked from Aloisius/aws-sdk-java

Official mirror of the AWS SDK for Java. For more information on the AWS SDK for Java, see our web site:

Updated Mar 11, 2014

commoncrawl-crawler

The CommonCrawl Crawler Engine and Related MapReduce code

Updated Jul 14, 2013

commoncrawl

CommonCrawl Project Repository

Updated Feb 14, 2013

Python 0 7

example-traitor

forked from norvigaward/2012-naward13

Updated Jan 16, 2013

Java 2 3

example-apprankings

forked from norvigaward/2012-naward07

Updated Jan 16, 2013