Data Engineering Foundations

Hands-on repo bridging applied analytics (SQL, BI, warehousing) and systems internals (Spark, SQL tuning, streaming, lakehouse patterns). Assessment is prove-it: runnable jobs, captured plans, and passing pytest.

Start here

• Setup — WSL, Windows, uv, Docker (infra/), and make commands
• Progression — the beginner→cap onboarding ladder on one page
• Curriculum — tiered reading lists and sequencing (committed in docs/)
• Modules & labs — which folder, which services, which commands
• Module folders — direct links to exercises and READMEs

Optional local textbook extracts only: references/ (gitignored). Do not commit publisher material there.

Learner path

1. README.md — you are here
2. docs/progression.md — the absolute-beginner → cap ladder on one page
3. docs/setup.md — WSL, Docker, uv, make
4. docs/modules.md — which service and commands per module
5. modules/NN_*/README.md — exercises for the module you are on (e.g. 02 SQL)
6. docs/curriculum.md — reading lists when you need depth

Quickstart

Run from WSL at the repo root. See setup for Windows without WSL.

cp .env.example .env
make setup
make up
make seed
make load-sql
make test
make spark-submit JOB=modules/04_spark_internals/join_aggregate_job.py
make check

Native Windows (no WSL) — same steps via the PowerShell task runner:

Copy-Item .env.example .env
.\tasks.ps1 setup
.\tasks.ps1 up
.\tasks.ps1 seed
.\tasks.ps1 test

Cloud module (09)

uv sync --extra cloud        # install fsspec backends (s3fs/gcsfs/adlfs)
make up-cloud                # LocalStack + fake-gcs + Azurite
uv run python modules/09_cloud_portability/object_store_roundtrip.py --cloud aws
make test-cloud
make down-cloud

dbt / orchestration / catalog (10)

uv sync --extra dbt          # install dbt-duckdb
make dbt-run                 # build models, run data tests, generate the catalog
make test-dbt

Modules (01–10)

• 01 Python · light (by design — least role-relevant)
• 02 SQL · built
• 03 BI · built (concepts)
• 04 PySpark · built
• 05 Warehousing · built
• 06 Kafka · built (roundtrip + concepts)
• 07 AI-assisted · agent files
• 08 Lakehouse · built
• 09 Cloud portability (AWS/GCP/Azure) · built
• 10 dbt / orchestration / data catalog · built

Scale knob

make seed-large
# or: uv run python -m def_.datagen.cli --scale-gb 2.0

Agent context

• CLAUDE.md
• .cursor/rules/

License

Learning use. Keep long textbook paste in local references/ only.