Common Crawl infrastructure updates meta ticket

[kgodey]

The main goal of these updates will be to reduce the number of technologies we're using to process Common Crawl data.

The first step should be to move the AWS Data Pipeline jobs into Apache Airflow.

Next we should experiment with whether or not it's worth trying to get rid of the AWS Glue job.

If these steps are done, we'll have the advantage of bringing all of our data pipelines (at least the ones getting data from the internet into our catalog) into one unified location, with a unified interface so that we can see their status at a glance.

Further, we will have a bit more flexibility and control over dependencies between the jobs. Currently, we have a flow like:

AWS Data Pipeline 1 -> AWS Glue -> AWS Data Pipeline 2

However, if the first of these steps fails, the second two still try to run, costing time and resources. Further, tracking down the source of the failure requires comparing logs across two different AWS services. This would be improved by bringing these steps into the same Apache Airflow DAG, or into a number of DAGs with dependencies defined between them.

Lastly, bringing these pipelines into Airflow will reduce our dependency on AWS-only services, and make our work more approachable and visible to people not on CC Staff.

[mathemancer]

As an intermediate step, it might make sense to create an Airflow DAG with the following steps:

1. Run the PySpark job defined by the first data pipeline
2. Call AWS Glue to run the intermediate transformation job that it handles
3. Run the PySpark job defined by the second data pipeline.

When this is set up, we'll already have many of the visibility advantages of bringing things into Airflow, and we won't have to deal with replacing the AWS Glue transformations with our own code (which might be a pain)