Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dbt docs with DbtProject #21568

Draft
wants to merge 21 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/content/_navigation.json
Original file line number Diff line number Diff line change
Expand Up @@ -859,6 +859,10 @@
"title": "dbt",
"path": "/integrations/dbt",
"children": [
{
"title": "Quickstart",
"path": "/integrations/dbt/quickstart"
},
{
"title": "dbt & Dagster tutorial",
"path": "/integrations/dbt/using-dbt-with-dagster",
Expand All @@ -881,6 +885,10 @@
}
]
},
{
"title": "dbt & Dagster+ guide",
"path": "/integrations/dbt/using-dbt-with-dagster-plus"
},
{
"title": "Reference",
"path": "/integrations/dbt/reference"
Expand Down
141 changes: 141 additions & 0 deletions docs/content/integrations/dbt/quickstart.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
---
title: "Quickstart: Run your dbt project with Dagster"
description: Get started swiftly with this simple dbt + Dagster example.
---

# Quickstart: Run your dbt project with Dagster

<Note>
<strong>Note</strong>: This guide uses the <code>DbtProject</code> class,
which is an experimental feature. Visit the{" "}
<a href="/\_apidocs/libraries/dagster-dbt">
dagster-dbt library API reference
</a>{" "}
for more info.
</Note>

If you just want to see the code of the example, skip to the end on this guide.

<div className="w-full inline-flex flex-col space-y-2 md:space-y-0 md:flex-row md:space-x-4">
<Button link="#using-dbt-with-dagster-in-a-single-file">See the code</Button>
</div>

---

## Setup your environment

<Note>
<strong>Note</strong>: We strongly recommend installing Dagster inside a
Python virtualenv. Visit the{" "}
<a href="/getting-started/install">Dagster installation docs</a> for more
info.
</Note>

Install dbt, Dagster, and the Dagster webserver/UI\*\*. Run the following to install everything using pip:

```shell
pip install dagster-dbt dagster-webserver
```

The `dagster-dbt` library installs both `dbt-core` and `dagster` as dependencies. Refer to the [dbt](https://docs.getdbt.com/dbt-cli/install/overview) and [Dagster](/getting-started/install) installation docs for more info.

---

## Run your dbt project in Dagster's UI

Now that your code is ready, you can run Dagster's UI to take a look at your dbt project.

1. Change directories to the Dagster project directory:

```shell
cd my_dagster_project/
```

2. To start Dagster's UI, run the following:

```shell
dagster dev
```

Which will result in output similar to:

```shell
Serving dagster-webserver on http://127.0.0.1:3000 in process 70635
```

3. In your browser, navigate to <http://127.0.0.1:3000>. The page will display the asset graph for the job created by the schedule definition:

<!--
![Asset graph for your job in Dagster's UI, containing dbt models loaded as Dagster assets](/images/integrations/dbt/quickstart/asset_graph.png)
-->

<Image
alt="Asset graph for your job in Dagster's UI, containing dbt models loaded as Dagster assets"
src="/images/integrations/dbt/quickstart/asset_graph.png"
width={2688}
height={1680}
/>

3. In Dagster, running a dbt model corresponds to _materializing_ an asset. Because a schedule definition has been added to your code location, your assets will be materialized at the next cron tick. The assets can also be materialized manually by click the **Materialize all** button near the top right corner of the page. This will launch a run to materialize the assets.

---

## Using dbt with Dagster in a single file

Running your dbt project with Dagster can be easily done after creating a single file. For this example, let's consider a basic use case - say you want to represent your dbt models as Dagster assets and run them daily at midnight.

With your text editor of choice, create a Python file in the same directory as your dbt project directory and add the following code. Note that since this file contains [all Dagster definitions required for your code location](/concepts/code-locations), it is recommended to name it `definitions.py`.

The following code assumes that your Python file and dbt project directory are adjacent in the same directory. If that's not the case, make sure to update the `RELATIVE_PATH_TO_YOUR_DBT_PROJECT` constant so that it points to your dbt project.

```python file=/integrations/dbt/quickstart.py startafter=start_example endbefore=end_example
from pathlib import Path

from dagster import AssetExecutionContext, Definitions
from dagster_dbt import (
DbtCliResource,
DbtProject,
build_schedule_from_dbt_selection,
dbt_assets,
)

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"

my_project = DbtProject(
project_dir=Path(__file__)
.joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
.resolve(),
)


@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()


my_schedule = build_schedule_from_dbt_selection(
[my_dbt_assets],
job_name="materialize_dbt_models",
cron_schedule="0 0 * * *",
dbt_select="fqn:*",
)

defs = Definitions(
assets=[my_dbt_assets],
schedules=[my_schedule],
resources={
"dbt": DbtCliResource(project_dir=my_project),
},
)
```

---

## What's next?

Congratulations on successfully running your dbt project in Dagster!

In this example, we created a single file to handle a simple use case and run your dbt project. To learn more about our dbt integrations and how to handle more complex use cases, consider the following options:

- To find out more about integrating dbt with Dagster, begin the official dbt scaffold tutorial, like the [dbt & Dagster project](/integrations/dbt/using-dbt-with-dagster).
- For an end-to-end example, from the project creation to the deployment to Dagster+, checkout out the Dagster & dbt course in [Dagster University](https://courses.dagster.io).
1 change: 1 addition & 0 deletions docs/content/integrations/dbt/reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ For a step-by-step implementation walkthrough, refer to the [Using dbt with Dags
| <PyObject module="dagster_dbt" object="dbt_assets" decorator /> | A decorator used to define Dagster assets for dbt models defined in a dbt manifest. |
| <PyObject module="dagster_dbt" object="DbtCliResource" /> | A class that defines a Dagster resource used to execute dbt CLI commands. |
| <PyObject module="dagster_dbt" object="DbtCliInvocation" /> | A class that defines the representation of an invoked dbt command. |
| <PyObject module="dagster_dbt" object="DbtProject" /> | A class that defines the representation of a dbt project and related settings that assist with managing manifest.json preparation. |
| <PyObject module="dagster_dbt" object="DagsterDbtTranslator" /> | A class that can be overridden to customize how Dagster asset metadata is derived from a dbt manifest. |
| <PyObject module="dagster_dbt" object="DagsterDbtTranslatorSettings" /> | A class with settings to enable Dagster features for a dbt project. |
| <PyObject module="dagster_dbt" object="DbtManifestAssetSelection" /> | A class that defines a selection of assets from a dbt manifest and a dbt selection string. |
Expand Down
42 changes: 42 additions & 0 deletions docs/content/integrations/dbt/using-dbt-with-dagster-plus.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: "Guide: Using dbt with Dagster+"
description: Deploy your dbt + Dagster project in Dagster+.
---

# Guide: Using dbt with Dagster+

## Import your dbt project to Dagster+

[//]: <> "TODO: Add screenshot of Dagster+"

### Example: import a dbt project from a GitHub repo

[//]: <> "TODO: Add screenshot of Dagster+"

#### Select and deploy the repo

#### Review the state of the repo

#### Understanding the file structure

## Import your dbt + Dagster project to Dagster+

[//]: <> "TODO: Reference the using dbt with dagster tutorial"

[//]: <> "TODO: Add screenshot of Dagster+"

### Example: import a dbt + Dagster project from a GitHub repo

[//]: <> "TODO: Add screenshot of Dagster+"

#### Select and deploy the repo

#### Review the state of the repo

## Understanding the cloud action files

### With GitHub

## Ready to get started?

[//]: <> "TODO: Include reference to dbt course in Dagster University"
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# ruff: isort: skip_file

# start_example
from pathlib import Path

from dagster import AssetExecutionContext, Definitions
from dagster_dbt import (
DbtCliResource,
DbtProject,
build_schedule_from_dbt_selection,
dbt_assets,
)

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"

my_project = DbtProject(
project_dir=Path(__file__)
.joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
.resolve(),
)


@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
yield from dbt.cli(["build"], context=context).stream()


my_schedule = build_schedule_from_dbt_selection(
[my_dbt_assets],
job_name="materialize_dbt_models",
cron_schedule="0 0 * * *",
dbt_select="fqn:*",
)

defs = Definitions(
assets=[my_dbt_assets],
schedules=[my_schedule],
resources={
"dbt": DbtCliResource(project_dir=my_project),
},
)
# end_example