Skip to content

Commit

Permalink
Adding Dagster-related code
Browse files Browse the repository at this point in the history
  • Loading branch information
zoltanctoth committed Jan 15, 2024
1 parent 836b407 commit 8feab5f
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 11 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ dbtlearn/package-lock.yml
dbtlearn/target/
dbtlearn/dbt_packages/
archive
dbt_dagster_project

logs/
venv/
Expand Down
20 changes: 18 additions & 2 deletions DAGSTER.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,26 @@
# Installation
# Orchestrating dbt with Dagster

## Set up your environment
Let's create a virtualenv and install dbt and dagster. These packages are located in [requirements.txt](requirements.txt).
```
virutalenv venv -p python3.11
pip install -r requirements.txt
```

## Create a dagster project
Dagster has a command for creating a dagster project from an existing dbt project:
```
dagster-dbt project scaffold --project-name dbt_dagster_project --dbt-project-dir=dbtlearn
```

# Start dev server
_At this point in the course, open [schedules.py](dbt_dagster_project/dbt_dagster_project/schedules.py) and uncomment the schedule logic._

# Start dagster
Now that our project is created, start the Dagster server:

```
cd dbt_dagster_project
DAGSTER_DBT_PARSE_PROJECT_ON_LOAD=1 dagster dev
```

We will continue our work on the dagster UI at [http://localhost:3000/](http://localhost:3000)
12 changes: 3 additions & 9 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
dagster==1.5.9
dagster-dbt==0.21.9
dagster-graphql==1.5.9
dagster-pipes==1.5.9
dagster-webserver==1.5.9
dbt-core==1.7.2
dbt-extractor==0.5.0
dbt-semantic-interfaces==0.4.1
dbt-snowflake==1.7.0
dbt-snowflake==1.7.1
dagster-dbt==0.22.0
dagster-webserver==1.6.0

0 comments on commit 8feab5f

Please sign in to comment.