dbt-spark-demo

Use GitHub Codespaces / VS Code Remote Dev Toolkit

Or

Local setup steps:

1. Install requirements (Linux)
2. Deploy spark server
3. Run dagster web ui

Requirement

Debian

apt-get update
# For thrift
apt-get -y install gcc libsasl2-modules libsasl2-dev libsasl2-modules-gssapi-heimdal

Spark Server

services

• minio
• spark thrift server (STS)

docker-compose up -d

Dagster + DBT

cd my-dagster-project
pip install -e ".[dev]"
dagster dev

Open Web UI: http://127.0.0.1:3000

Misc

Disable auth

Disable auth by removing hive-site.xml mount of sts services.