DBT AI

An application that allows AI powered DBT development and recommendations for your DBT models.

Features

• Scans all dbt models and generates a report containing recommendations for each model
  • By default only basic recommendations are provided
  • Ability to request for more advanced recommendations
• Create DBT models by using AI prompts
• Lists dbt models that are missing associated metadata e.g. in a schema.yml file or equivalent
• DBT model lineage description is listed in the terminal (more features coming soon)

Installation

First time installation:

pip install dbt-ai

To upgrade to the latest version:

pip install dbt-ai --upgrade

To install a specific version:

pip install dbt-ai==<version>

Replace <version> with your desired version e.g. 0.2.0. You can view available versions in the Releases section of this repo or on our Pypi page. WARNING: This is an early phase application that may still contain bugs

Prerequisites

• In order to benefit from AI features, you need your own OpenAI API Key with the initial version of this application
  • Once you sign up to OpenAI you can create an API key.
  • Trial version gives you a certain amount of credits allowing you to make many API calls
  • Usage beyond the trial credits require billing details. API usage pricing provides more info
• Ideally you already have dbt project to test this out on
• Python 3.10 or greater is required

Usage

Setting up your API key is needed for all the AI features

• Set up your OpenAI API key as an environment variable:

export OPENAI_API_KEY="your_openai_api_key"

Generate Recommendations (AI)

This feature will generate the recommendation report and also inform you about any models missing metadata

Basic Usage

The default setting is to provide basic recommendations

1. Run the application passing in the path to your dbt project:

dbt-ai -f path/to/dbt/project

For example, if you are already inside your dbt project directory, you can run:

dbt-ai -f .

Available Settings

• Database: specify which database system your dbt models are written for. Default: snowflake
  • -d / --database
  • Available values: snowflake, postgres, redshift, bigquery
  • Usage example:

  dbt ai -f . -d snowflake

• Advanced Recommendations: Request for advanced recommendations. Default: No setting defaults to basic recommendations
  • -a / --advanced-rec
  • Available values: Only flag required
  • Usage example:

  dbt ai -f . -a 

  Or

  dbt ai -f . --advanced-req

Please allow some time for the AI model to process your dbt models. The application will process all dbt model files in your project and generate an HTML report with suggestions for each model. The report will be saved as dbt_model_suggestions.html within the dbt project directory. Upon generation of the report, it will be opened in a new browser tab.

Create DBT Models from prompt (AI)

This feature lets you specify a prompt, which creates AI generated DBT model files in the models/ directory of the specified dbt project. The AI model has access to your sources.yml file, if you wish to refer to any sources in your prompt. Being specific will provide better results.

1. Run the application with the --create-models flag to specify the prompt you wish to use to create your DBT models

dbt-ai -f path/to/dbt/project --create-models 'your prompt goes here'

Here is an example:

dbt-ai -f . --create-models 'Write me a model that uses all the sources available in sources.yml and joins them together using the id column'

Generated Report

This shows an example of a report generated from a DBT project containing 3 models

Contributing

We welcome contributions to the project! Please feel free to open issues or submit pull requests with your improvements and suggestions.

See CONTRIBUTING.md to get started and develop in this repo.