Skip to content

Commit

Permalink
Update prefect-dbt index page (#13187)
Browse files Browse the repository at this point in the history
  • Loading branch information
@discdiver
discdiver committed May 2, 2024
1 parent 1d2fa2d commit 18afa50
Showing 1 changed file with 58 additions and 83 deletions.
141 changes: 58 additions & 83 deletions docs/integrations/prefect-dbt/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,27 +1,22 @@
# prefect-dbt

<p align="center">
<a href="https://pypi.python.org/pypi/prefect-dbt/" alt="PyPI version">
<img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-dbt?color=26272B&labelColor=090422"></a>0
<a href="https://pepy.tech/badge/prefect-dbt/" alt="Downloads">
<img src="https://img.shields.io/pypi/dm/prefect-dbt?color=26272B&labelColor=090422" /></a>
</p>
With prefect-dbt, you can trigger and observe dbt Cloud jobs, execute dbt Core CLI commands, and incorporate other tools, such as Snowflake, into your dbt runs.
Prefect provides a global view of the state of your workflows and allows you to take action based on state changes.

With prefect-dbt, you can easily trigger and monitor dbt Cloud jobs, execute dbt Core CLI commands, and incorporate other services, like Snowflake, into your dbt runs!
## Getting started

Check out the examples below to get started!
1. [Install prefect-dbt](#installation)
1. [Register newly installed blocks types](#registering-block-types)

## Getting Started

Be sure to install [prefect-dbt](#installation) and [save a block](#saving-credentials-to-block) to run the examples below!
Explore the examples below to learn how to use Prefect with dbt.

### Integrate dbt Cloud jobs with Prefect flows

If you have an existing dbt Cloud job, take advantage of the flow, `run_dbt_cloud_job`.
If you have an existing dbt Cloud job, you can use the pre-built flow `run_dbt_cloud_job` to trigger a job run and wait until the job run is finished.

This flow triggers the job and waits until the job run is finished.
If some nodes fail, `run_dbt_cloud_job` efficiently retries the unsuccessful nodes.

If certain nodes fail, `run_dbt_cloud_job` efficiently retries the specific, unsuccessful nodes.
Prior to running this flow, [save your dbt Cloud credentials to a DbtCloudCredentials block](#saving-credentials-to-a-block)

```python
from prefect import flow
Expand All @@ -42,19 +37,18 @@ run_dbt_job_flow()

### Integrate dbt Core CLI commands with Prefect flows

`prefect-dbt` also supports execution of dbt Core CLI commands.

To get started, if you don't have a `DbtCoreOperation` block already saved,
set the commands that you want to run; it can include a mix of dbt and non-dbt commands.
`prefect-dbt` supports execution of dbt Core CLI commands.

Then, optionally specify the `project_dir`.
If you don't have a `DbtCoreOperation` block saved, create one and set the commands that you want to run.

If `profiles_dir` is unset, it will try to use the `DBT_PROFILES_DIR` environment variable.
If that's also not set, it will use the default directory `$HOME/.dbt/`.
Optionally, specify the `project_dir`.
If `profiles_dir` is not set, the `DBT_PROFILES_DIR` environment variable will be used.
If `DBT_PROFILES_DIR` is not set, the default directory will be used `$HOME/.dbt/`.

#### Using an existing profile

If you already have an existing dbt profile, specify the `profiles_dir` where `profiles.yml` is located.
If you have an existing dbt profile, specify the `profiles_dir` where `profiles.yml` is located.
You can use it in code like this:

```python
from prefect import flow
Expand All @@ -76,7 +70,8 @@ trigger_dbt_flow()

To setup a new profile, first [save and load a DbtCliProfile block](#saving-credentials-to-block) and use it in `DbtCoreOperation`.

Then, specify `profiles_dir` where `profiles.yml` will be written.
Then, specify`profiles_dir` where `profiles.yml` will be written.
Here's example code with placeholders:

```python
from prefect import flow
Expand All @@ -102,81 +97,91 @@ trigger_dbt_flow()

## Resources

If you need help getting started with or using dbt, please consult the [dbt documentation](https://docs.getdbt.com/docs/building-a-dbt-project/documentation).
If you need help using dbt, consult the [dbt documentation](https://docs.getdbt.com/docs/building-a-dbt-project/documentation).

### Installation

To use `prefect-dbt` with dbt Cloud:
To install `prefect-dbt` for use with dbt Cloud:

<div class = "terminal">
```bash
pip install prefect-dbt
```
</div>

To use dbt Core (CLI):
To install with additional functionality for dbt Core (CLI):

<div class = "terminal">
```bash
pip install "prefect-dbt[cli]"
```
</div>

To use dbt Core with Snowflake profiles:
To install with additional functionality for dbt Core and Snowflake profiles:

<div class = "terminal">
```bash
pip install "prefect-dbt[snowflake]"
```
</div>

To use dbt Core with BigQuery profiles:
To install with additional functionality for dbt Core and BigQuery profiles:

<div class = "terminal">
```bash
pip install "prefect-dbt[bigquery]"
```
</div>

To use dbt Core with Postgres profiles:
To install with additional functionality for dbt Core and Postgres profiles:

<div class = "terminal">
```bash
pip install "prefect-dbt[postgres]"
```
</div>

!!! warning "Some dbt Core profiles require additional installation"

According to dbt's [Databricks setup page](https://docs.getdbt.com/reference/warehouse-setups/databricks-setup), users must first install the adapter:

<div class = "terminal">
```bash
pip install dbt-databricks
```
</div>

Check out the [desired profile setup page](https://docs.getdbt.com/reference/profiles.yml) on the sidebar for others.

Requires an installation of Python 3.8+.

We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.
prefect-dbt requires Python 3.8 or newer.

These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).
We recommend using a Python virtual environment manager such as conda, venv, or pipenv.

### Saving credentials to block
### Registering block types

Note, to use the `load` method on Blocks, you must already have a block document [saved through code](https://docs.prefect.io/concepts/blocks/#saving-blocks) or [saved through the UI](https://docs.prefect.io/ui/blocks/).
Register the block types in the prefect-dbt module to make them available for use.

!!! info "Registering blocks"
<div class = "terminal">
```bash
prefect block register -m prefect_dbt
```
</div>

Register blocks in this module to
[view and edit them](https://docs.prefect.io/ui/blocks/)
on Prefect Cloud:
### Saving credentials to a block

```bash
prefect block register -m prefect_dbt
```
Blocks can be [created through code](/concepts/blocks/#saving-blocks) or through the UI.

#### dbt Cloud

To create a dbt Cloud credentials block:
To create a dbt Cloud Credentials block do the following:

1. Head over to your [dbt Cloud profile](https://cloud.getdbt.com/settings/profile).
2. Login to your dbt Cloud account.
3. Scroll down to "API" or click "API Access" on the sidebar.
1. Go to your [dbt Cloud profile](https://cloud.getdbt.com/settings/profile).
2. Log in to your dbt Cloud account.
3. Scroll to **API** or click **API Access** on the sidebar.
4. Copy the API Key.
5. Click Projects on the sidebar.
5. Click **Projects** on the sidebar.
6. Copy the account ID from the URL: `https://cloud.getdbt.com/settings/accounts/<ACCOUNT_ID>`.
7. Create a short script, replacing the placeholders.
7. Create and run the following script, replacing the placeholders.

```python
from prefect_dbt.cloud import DbtCloudCredentials
Expand All @@ -187,13 +192,13 @@ DbtCloudCredentials(
).save("CREDENTIALS-BLOCK-NAME-PLACEHOLDER")
```

Then, to create a dbt Cloud job block:
Then, to create a dbt Cloud job block do the following:

1. Head over to your [dbt home page](https://cloud.getdbt.com/).
2. On the top nav bar, click on Deploy -> Jobs.
2. On the top nav bar, click on **Deploy** -> **Jobs**.
3. Select a job.
4. Copy the job ID from the URL: `https://cloud.getdbt.com/deploy/<ACCOUNT_ID>/projects/<PROJECT_ID>/jobs/<JOB_ID>`
5. Create a short script, replacing the placeholders.
5. Create and run the following script, replacing the placeholders.

```python
from prefect_dbt.cloud import DbtCloudCredentials, DbtCloudJob
Expand All @@ -205,7 +210,7 @@ dbt_cloud_job = DbtCloudJob(
).save("JOB-BLOCK-NAME-PLACEHOLDER")
```

Congrats! You can now easily load the saved block, which holds your credentials:
You can now load the saved block, which can access your credentials:

```python
from prefect_dbt.cloud import DbtCloudJob
Expand All @@ -217,13 +222,9 @@ DbtCloudJob.load("JOB-BLOCK-NAME-PLACEHOLDER")

!!! info "Available `TargetConfigs` blocks"

The following may vary slightly depending on the service you want to incorporate.

Visit the [API Reference](cli/configs/base) to see other built-in `TargetConfigs` blocks.

If the desired service profile is not available, check out the
[Examples Catalog](examples_catalog/#clicredentials-module) to see how you can
build one from the generic `TargetConfigs` class.
If the desired service profile is not available, check out the [Examples Catalog](examples_catalog/#clicredentials-module) to see how you can build one from the generic `TargetConfigs` class.

To create dbt Core target config and profile blocks for BigQuery:

Expand Down Expand Up @@ -274,29 +275,3 @@ from prefect_dbt.cloud import DbtCoreOperation

DbtCoreOperation.load("DBT-CORE-OPERATION-BLOCK-NAME-PLACEHOLDER")
```

### Feedback

If you encounter any bugs while using `prefect-dbt`, feel free to open an issue in the [prefect](https://github.com/PrefectHQ/prefect) repository.

If you have any questions or issues while using `prefect-dbt`, you can find help in the [Prefect Slack community](https://prefect.io/slack).

### Contributing

If you'd like to help contribute to fix an issue or add a feature to `prefect-dbt`, please [propose changes through a pull request from a fork of the repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork).

Here are the steps:

1. [Fork the repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#forking-a-repository)
2. [Clone the forked repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo#cloning-your-forked-repository)
3. Install the repository and its dependencies:
```
pip install -e ".[dev]"
```
4. Make desired changes
5. Add tests
6. Install `pre-commit` to perform quality checks prior to commit:
```
pre-commit install
```
7. `git commit`, `git push`, and create a pull request

0 comments on commit 18afa50

Please sign in to comment.