Install mysql version: 8.0.34 Recommended installer download link: https://dev.mysql.com/downloads/
Tools used:
- Python
- MySQL
- Spark
Front End: Created a CLI based application. Based on the input provided, the user can perform various CRUD operations on the connected MySQL database.
Back End: Extracted the data stored in different JSON files as well as from a REST API endpoint. Utilized pySpark for efficient handling the manipulation and querying of large amounts of data. Ensured data quality through standardization, and cleaning. Created a MySQL database, and loaded the data into the database.
Reliability and Integrity: Focused on error handling