dbt-bigquery-gitpod-config

A gitpod config for using dbt-bigquery in vscode

Introduction

This repository documents how you can use dbt with BigQuery in Gitpod's web-based VS Code environment. It leverages the vscode-dbt-bigquery-power-user VS Code extension to provide a powerful online coding environment using dbt that almost matches a desktop VS Code experience.

Quickstart

If you want to try using dbt on gitpod using VS Code DBT BigQuery Power User extension, you can follow the instructions here.

Requirements

• a dbt project using BigQuery that is version controlled in any repo supported by gitpod (Github, Gitlab, Bitbucket)
• a service account file for the bigquery project. To create a service account file, see the Creating and Managing service account keys page.
• a Gitpod account

This assumes you already have a git repository account as well as a BigQuery GCP project as well as an initial dbt project provisioned (dbtCloud account optional). For instructions to setup dbt and BigQuery, please see the dbt setup documentation

Usage

• To start configuring your dbt-bigquery project to run on gitpod, you can install a Chrome/Brave/Firefox extension called Gitpod that displays a Gitpod button on your repo's web page. As an alternative, you can just open a link https://gitpod.io/#<your-repo-url>.

• Then, in the terminal of the vscode editor, run the following command:

curl -s https://raw.githubusercontent.com/butchland/dbt-gitpod-config/main/scripts/loadconfig | bash

This will copy over the following files from the config directory:

• .gitpod.yml
• gitpod-dbt-profiles.yml
• gitpod-vscode-settings.json
• requirements.txt
• .gitignore - (optional, will only copy if it doesn't exist yet)

This assumes of course, that you haven't configured your gitpod configuration file .gitpod.yml in your project yet.

If you have an existing .gitpod.yml file, rename it temporarily to something else before running the previous curl command (make sure to merge the contents of the original .gitpod.yml you renamed with the .gitpod.yml that was copied over by the curl command afterwards).

Then in the Gitpod Variables page gitpod.io/variables, create a new variable named CREDENTIALS_JSON with the scope for your username/<project-name>. Copy the entire contents of the service account json file into the value field for the variable.

If you already have an existing .gitignore file, make sure the following entries are added to it:

• .credentials
• .vscode
• venv
• .extensions

After running script, you should now commit these new files to your version control and push the changes to the remote repository. You can then delete the gitpod workspace and create a new workspace by either clicking the Gitpod button or opening the link https://gitpod.io/#<your-repo-url>.

The new gitpod.yml configuration will then update the .vscode/settings.ini for vscode's dbt integration and also setup dbt_profiles.yml for your bigquery project in the $HOME/.dbt folder (which are not persisted across sessions in the gitpod environment).

You can edit these configuration files to further customize your gitpod environment.

IMPORTANT: Copy the section (especially profile_name and dataset) for your project from your local dbt_profiles.yml to the gitpod-dbt-profiles.yml as the gitpod config will refresh this each time it starts your gitpod workspace. The same thing applies to your .vscode/settings.json which is copied over from the gitpod-vscode-settings.json if it doesn't exist yet.

Adding Gitpod VS Code extensions

If you would like to add VS Code extensions that are already available in the Open VSX Registry, just add them in the relevant section (vscode/extensions) of your .gitpod.yml

For other extensions that are not yet available in the Open VSX Registry, you can download the extension and push it into any internet accessible location (e.g. github) and update your .gitpod.yml to download them into your gitpod workspace and install into your gitpod VSCode editor. The current .gitpod.yml already installs the vscode-dbt and sqlfluff VS Code extensions by default.

If you have issues and suggestions please file an issue in the github issue tab.