ETL

Description

The ETL module is responsible for extracting data from Amazon and Goodreads data sources, performing transformations and storing it back as JSON files on disk. All relevant modules are stored under thenextbestbook/etl. The ETL module uses pyspark and spark's in built SparkSQL library

Setup

Download the datasets provided in the README file and store them in etl/data folder.

Run the following spark jobs

spark-submit etl/etl_goodreads_script.py

spark-submit etl/etl_amazon_script.py

This will generate books_with_authors.json and ratings_and_reviews.json files in the data folder

Project structure

• etl/etl_goodreads.py and etl/etl_amazon.py are the classes used to perform the necessary ETL jobs. The classes create a spark context upon object creation and have methods to run relevant SQL queries on the dataframe.

• etl/etl_amazon_script.py and etl/etl_goodreads_script.py these are the spark jobs that run the actual ETL processes

• etl/merge_json.py is used to merge the ratings and reviews data from goodreads and amazon.

• etl/tests contains the necessary test cases