dlt_dbt_cloud

Repository with demos of using DLT and DBT Cloud

Installation

pip install dlt[bigquery]

Set up the pokemon pipeline

To get started with this data pipeline, follow these steps:

Init the pipeline

Enter the following command:

dlt init pokemon bigquery

For more information, read the Add a verified source.

Add credentials

1. In the .dlt folder, there's a file called secrets.toml. It's where you store sensitive information securely, like access tokens. Keep this file safe.

   Use the following format for service account authentication:

   [sources.source_name]
   secret = "Please set me up!"

   Pokemon verified source doesn't require authentication, so we don't need to provide credentials.

2. Enter credentials for the BigQuery destination as per the docs:

   [destination.bigquery]
   location = "US"
   
   [destination.bigquery.credentials]
   project_id = "project_id" # please set me up!
   private_key = "private_key" # please set me up!
   client_email = "client_email" # please set me up!

For more information, read the General Usage: Credentials.

Set up the dbt Cloud

Sign in dbt Cloud

Go through this Quickstart for dbt Cloud and BigQuery.

Create the dbt model

Create the model for your data with the tutorial: How to build SQL models.

Update pipeline script

Add the following code into your pipeline script (pipelines/pokemon_pipeline.py):

from dlt.helpers.dbt_cloud import run_dbt_cloud_job

run_info = run_dbt_cloud_job()
print(f"Job run status: {run_info['status_humanized']}")

Credentials

Use the following format for dbt Cloud API authentication in .dlt/secrets.toml:

[dbt_cloud]
api_token = "set me up!" # required for authentication
account_id = "set me up!" # required for both helpers function
job_id = "set me up!" # optional only for run_dbt_cloud_job function (you can pass this explicitly as an argument to the function)

More information about dbt cloud helpers in DBT Cloud Client and Helper Functions.

Run the pipeline

Now you are ready to run the pipeline! To get started, run the following command:

python pokemon_pipeline.py