DE Zoomcamp 2024 by DataTalksClub

This repository contains files that I made or edit to fulfill my answer for tasks along my participation on Data Engineering Zoomcamp 2024.

Week 1: Introduction and Prerequisites

• Contents: GCP, Docker, docker-compose, Postgres run with Docker, Terraform
• Homework: Homework
• Answer: Answer

Week 2: Workflow Orchestration

• Contents: Data Lake, Workflow orchestration, Mage, ETL
• Homework: Homework
• Answer: Answer

Week 3: Data Warehouse

• Contents: Data Warehouse, Google BigQuery, Partitioning and Clustering, Internals of BigQuery, BigQuery Machine Learning
• Homework: Homework
• Answer: Answer

Week 4: Analytics Engineering

• Contents: Analytics Engineering, dbt (data build tool), dbt models, Testing and documenting, Deployment to the cloud and locally, Visualizing the data
• Homework:
• Answer:

Week 5: Batch Processing

• Contents: Batch Processing, Spark, Spark Dataframes, Spark SQL, Spark GroupBy and Joins
• Homework:
• Answer:

Week 6: Streaming Processing

• Contents: Kafka, Kafka Streams, Kafka Connect, and KSQL
• Homework:
• Answer:

Week 7: Final Project

• Guidelines: Guidelines
• Answer: