For this project we needed to create an automated pipeline that takes in new data, performs the appropriate transformations, and loads the data into existing tables. We refactored code to create one function that takes in three files (Wikipedia data, Kaggle metadata, and the MovieLens ratind data). The function performs the ETL process by adding the data to a PostgreSQL database.
The purpose of this project is to create one function that takes in three files and performs the ETL process using a PostgreSQL database.