Skip to content


Switch branches/tags

Latest commit

- avoid multiple extraction of host names from source and base URLs
  - implement method get_links(...) in class ExtractHostLinksJob
  - pass extracted source and base host names to method yield_links(...)
- update IANA TLD list
- consistent naming of source nodes (src_url, src_host instead of
  from_url, from_host)

Git stats


Failed to load latest commit information.

Common Crawl Logo

Common Crawl PySpark Examples

This project provides examples how to process the Common Crawl dataset with Apache Spark and Python:

Further information about the examples and available options is shown via the command-line option --help.

Implementing a Custom Extractor

Extending the CCSparkJob isn't difficult and for many use cases it's sufficient to override a single method (process_record). Have a look at one of the examples, eg. to count HTML tags.


To develop and test locally, you will need to install

pip install -r requirements.txt
  • (optionally, and only if you want to query the columnar index) install S3 support libraries so that Spark can load the columnar index from S3

Compatibility and Requirements

Tested with Spark 2.1.0 – 2.4.6 in combination with Python 2.7 or 3.5, 3.6, 3.7, and with Spark 3.0.0 - 3.2.1 in combination with Python 3.7, 3.8 and 3.9.

Get Sample Data

To develop locally, you'll need at least three data files – one for each format used in at least one of the examples. They can be fetched from the following links:

Alternatively, running downloads the sample data. It also writes input files containing

  • sample input as file:// URLs
  • all input of one monthly crawl as relative paths
    • to use with --input_base_url s3://commoncrawl/ resp. --input_base_url

Note that the sample data is from an older crawl (CC-MAIN-2017-13 run in March 2017). If you want to use more recent data, please visit the Common Crawl site.

Process Common Crawl Data on Spark

Running locally

First, point the environment variable SPARK_HOME to your Spark installation. Then submit a job via

$SPARK_HOME/bin/spark-submit ./ \
	--num_output_partitions 1 --log_level WARN \
	./input/test_warc.txt servernames

This will count web server names sent in HTTP response headers for the sample WARC input and store the resulting counts in the SparkSQL table "servernames" in your warehouse location defined by spark.sql.warehouse.dir (usually in your working directory as ./spark-warehouse/servernames).

The output table can be accessed via SparkSQL, e.g.,

>>> df ="spark-warehouse/servernames")
>>> for row in df.sort(df.val.desc()).take(10): print(row)
Row(key=u'Apache', val=9396)
Row(key=u'nginx', val=4339)
Row(key=u'Microsoft-IIS/7.5', val=3635)
Row(key=u'(no server in HTTP header)', val=3188)
Row(key=u'cloudflare-nginx', val=2743)
Row(key=u'Microsoft-IIS/8.5', val=1459)
Row(key=u'Microsoft-IIS/6.0', val=1324)
Row(key=u'GSE', val=886)
Row(key=u'Apache/2.2.15 (CentOS)', val=827)
Row(key=u'Apache-Coyote/1.1', val=790)

See also

Running in Spark cluster over large amounts of data

As the Common Crawl dataset lives in the Amazon Public Datasets program, you can access and process it on Amazon AWS (in the us-east-1 AWS region) without incurring any transfer costs. The only cost that you incur is the cost of the machines running your Spark cluster.

  1. spinning up the Spark cluster: AWS EMR contains a ready-to-use Spark installation but you'll find multiple descriptions on the web how to deploy Spark on a cheap cluster of AWS spot instances. See also launching Spark on a cluster.

  2. choose appropriate cluster-specific settings when submitting jobs and also check for relevant command-line options (e.g., --num_input_partitions or --num_output_partitions, see below)

  3. don't forget to deploy all dependencies in the cluster, see advanced dependency management

  4. also the the file needs to be deployed or added as argument --py-files to spark-submit. Note: some of the examples require further Python files as dependencies.

Command-line options

All examples show the available command-line options if called with the parameter --help or -h, e.g.

$SPARK_HOME/bin/spark-submit ./ --help

Overwriting Spark configuration properties

There are many Spark configuration properties which allow to tune the job execution or output, see for example see tuning Spark or EMR Spark memory tuning.

It's possible to overwrite Spark properties when submitting the job:

$SPARK_HOME/bin/spark-submit \
    --conf spark.sql.warehouse.dir=myWareHouseDir \
    ... (other Spark options, flags, config properties) \
    ./ \
    ... (program-specific options)

Authenticated S3 Access or Access Via HTTP

Since April 2022 there are two ways to access of Common Crawl data:

  • using HTTP/HTTPS and the base URL or
  • using the S3 API to read the bucket s3://commoncrawl/ requires authentication and makes an Amazon Web Services account mandatory.

This project cc-pyspark uses boto3 to access WARC, WAT or WET files on s3://commoncrawl/. The best way is to ensure that a S3 read-only IAM policy is attached to the the IAM role of the EC2 instances where Common Crawl data is processed, see the IAM user guide. If this is no option (or if the processing is not running on AWS), there are various options to configure credentials in boto3.

Installation of S3 Support Libraries

While WARC/WAT/WET files are read using boto3, accessing the columnar URL index (see option --query of CCIndexSparkJob) is done directly by the SparkSQL engine and requires that S3 support libraries are available. These libs are usually provided when the Spark job is run on a Hadoop cluster running on AWS (eg. EMR). However, they may not be provided for any Spark distribution and are usually absent when running Spark locally (not in a Hadoop cluster). In these situations, the easiest way is to add the libs as required packages by adding --packages org.apache.hadoop:hadoop-aws:3.2.1 to the arguments of spark-submit. This will make Spark manage the dependencies - the hadoop-aws package and transitive dependencies are downloaded as Maven dependencies. Note that the required version of hadoop-aws package depends on the Hadoop version bundled with your Spark installation, e.g., Spark 3.2.1 bundled with Hadoop 3.2 (spark-3.2.1-bin-hadoop3.2.tgz).

Please also note that:

Example call to count words in 10 WARC records host under the .is top-level domain:

$SPARK_HOME/bin/spark-submit \
    --packages org.apache.hadoop:hadoop-aws:3.3.2 \
    ./ \
    --input_base_url s3://commoncrawl/ \
    --query "SELECT url, warc_filename, warc_record_offset, warc_record_length, content_charset FROM ccindex WHERE crawl = 'CC-MAIN-2020-24' AND subset = 'warc' AND url_host_tld = 'is' LIMIT 10" \
    s3a://commoncrawl/cc-index/table/cc-main/warc/ \
    myccindexwordcountoutput \
    --num_output_partitions 1 \
    --output_format json

Columnar index and schema merging

The schema of the columnar URL index has been extended over time by adding new columns. If you want to query one of the new columns (e.g., content_languages), the following Spark configuration option needs to be set:

--conf spark.sql.parquet.mergeSchema=true

However, this option impacts the query performance, so use with care! Please also read cc-index-table about configuration options to improve the performance of Spark SQL queries.

Alternatively, it's possible configure the table schema explicitly:


Examples are originally ported from Stephen Merity's cc-mrjob with the following changes and upgrades:

  • based on Apache Spark (instead of mrjob)
  • boto3 supporting multi-part download of data from S3
  • warcio a Python 2 and Python 3 compatible module to access WARC files

Further inspirations are taken from


MIT License, as per LICENSE