Syllabus
- Module 1: Containerization and Infrastructure as Code
- Module 2: Workflow Orchestration
- Workshop 1: Data Ingestion
- Module 3: Data Warehouse
- Module 4: Analytics Engineering
- Module 5: Batch processing
- Module 6: Streaming
- Workshop 2: Stream Processing with SQL
- Project
- Start: 15 January 2024 (Monday) at 17:00 CET
- Registration link: https://airtable.com/shr6oVXeQvSI5HuWD
- Cohort folder with homeworks and deadlines
- Check FAQ if you have problems
Note: NYC TLC changed the format of the data we use to parquet. In the course we still use the CSV files accessible here.
- Course overview
- Introduction to GCP
- Docker and docker-compose
- Running Postgres locally with Docker
- Setting up infrastructure on GCP with Terraform
- Preparing the environment for the course
- Homework
- Data Lake
- Workflow orchestration
- Workflow orchestration with Mage
- Homework
- Data Warehouse
- BigQuery
- Partitioning and clustering
- BigQuery best practices
- Internals of BigQuery
- BigQuery Machine Learning
- Basics of analytics engineering
- dbt (data build tool)
- BigQuery and dbt
- Postgres and dbt
- dbt models
- Testing and documenting
- Deployment to the cloud and locally
- Visualizing the data with google data studio and metabase
- Batch processing
- What is Spark
- Spark Dataframes
- Spark SQL
- Internals: GroupBy and joins
- Introduction to Kafka
- Schemas (avro)
- Kafka Streams
- Kafka Connect and KSQL
Putting everything we learned to practice
- Week 1 and 2: working on your project
- Week 3: reviewing your peers