This project demonstrates an end-to-end ELT pipeline utilizing modern data engineering tools such as Python, Docker, and Apache Airflow. It incorporates industry best practices, including automated unit testing, data quality validation, and CI/CD workflows, ensuring a robust and reliable data integration process.
Data is sourced via the YouTube Data API v3. While this implementation defaults to fetching data from the 'MrBeast' channel, the pipeline is fully configurable to support any YouTube channel by updating the relevant handle in the configuration settings.
The project leverages Apache Airflow for workflow orchestration, deployed via Docker containers for consistency and scalability. The ELT workflow consists of the following stages:
- Extraction: YouTube API data is retrieved using modular Python scripts.
- Loading: Raw data is initially persisted into a
staging schemawithin a containerized PostgreSQL database. - Transformation: Data undergoes processing and validation before being loaded into the
core schemafor analytical use.
The pipeline supports both initial full loads and incremental upserts to maintain data freshness. Once the data reaches the core schema and passes automated quality checks, it is optimized and ready for analysis.
The following metrics are extracted:
- Video ID
- Video Title
- Upload Date
- Duration
- Video Views
- Likes Count
- Comments Count
- Containerization - Docker, Docker-Compose
- Orchestration - Airflow
- Data Storage - Postgres
- Languages - Python, SQL
- Testing - SODA, pytest
- CI-CD - Github Actions