You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -4,271 +4,103 @@ description: "You can easily make changes in dbt and see them updated in your Li
4
4
---
5
5
6
6
<Info>
7
-
8
-
Lightdash supports dbt v1.4.0 and above. If you are using an older version of dbt, you will need to upgrade to sync your project to Lightdash
9
-
7
+
Lightdash supports dbt v1.4.0 and above. If you are using an older version of dbt, you will need to upgrade to sync your project to Lightdash
10
8
</Info>
11
9
12
-
## 1. Automatically: deploy your changes to Lightdash using a GitHub action
13
-
14
-
If you've connected Lightdash to GitHub, you can setup a `github action` and get Lightdash to deploy your project automatically. This is the easiest way to keep Lightdash in sync with your changes in dbt.
15
-
16
-
### Step 1: add the credentials to Github secrets
17
10
18
-
We are going to add some secrets and config to GitHub actions, but you don't want those to be public, so the best way to do this is to add them as secrets on Github.
11
+
## Syncing your dbt project to Lightdash
19
12
20
-
<Info>
21
-
22
-
If you already have a GitHub action for Lightdash, then you can use the same Lightdash secrets you created for your other action.
23
-
</Info>
13
+
You can sync your dbt project code with Lightdash in a few different ways. We recommend everyone set up continuous deployment, but you can also refresh in the Lightdash app or deploy from the CLI.
24
14
25
-
Go to your repo, click on `Settings` , on the left sidebar, click on `Secrets` under `Security`. Now click on the `New repository secret`
[Read how to do that and check out our example workflow files](/guides/cli/how-to-use-lightdash-deploy#automatically-deploy-your-changes-to-lightdash-using-a-github-action).
31
19
32
-
We need to add the following secrets:
33
20
34
-
##### `LIGHTDASH_API_KEY`
21
+
###2. Click "Refresh dbt" in Lightdash
35
22
36
-
Create a new personal access token, by going to `Settings`\>`Personal Access Tokens`. This is the token you'll put in for `LIGHTDASH_API_KEY`.
23
+
The button can be found on the Query from tables page.
The UUID for your project. For example, if your URL looks like `https://eu1.lightdash.cloud/projects/3538ab33-dc90-aabb-bc00-e50bba3a5f69/tables`, then `3538ab33-dc90-45f0-aabb-e50bba3a5f69` is your `LIGHTDASH_PROJECT`
29
+
_If you're using a git connection (like GitHub, Gitlab or Bitbucket), you'll need to push + merge your changes to the branch that your Lightdash project is connected to before you press `Refresh dbt`._
45
30
46
-
##### `LIGHTDASH_URL`
31
+
<Info>
32
+
If you've made any changes to the underlying data (for example, adding a new column in your `model.sql` file or changing the SQL logic of a dimension), then you need to run: `dbt run -m yourmodel` before you click `Refresh dbt` in Lightdash.
33
+
</Info>
47
34
48
-
This is either `https://eu1.lightdash.cloud` or `https://app.lightdash.cloud` for Lightdash Cloud users (check the URL to your Lightdash project). If you self-host, this should be your own custom domain.
49
35
50
-
##### `DBT_PROFILES`
36
+
###3. Push code from the CLI
51
37
52
-
Some tips for this bit:
38
+
If you're using the [Lightdash CLI](/guides/cli/how-to-install-the-lightdash-cli), you can use `lightdash deploy` to deploy your changes to Lightdash. [Read more about how to use `lightdash deploy`](/guides/cli/how-to-use-lightdash-deploy).
53
39
54
-
* You might be able to copy a bunch of the information from your local `profiles.yml` file. You can see what's in there by typing `cat ~/.dbt/profiles.yml` in your terminal.
55
-
* If you have a separate `prod` and `dev` profile, you probably want to use the information from your `prod` profile for your GitHub action.
56
-
* If you want to have different connection settings depending on the user that opened the pull request (dev profiles), then [check out this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
40
+
<Warning>
41
+
We don't recommend using `lightdash deploy` from your local environment as the primary way you update Lightdash since small mistakes can lead to production issues.
42
+
</Warning>
57
43
58
-
Find your data warehouse from the list below to get a profiles.yml file template. Fill out this template, and this is your `DBT_PROFILES` secret.
59
44
60
-
<AccordionGroup>
61
-
<Accordiontitle="BigQuery">
62
-
BigQuery OAuth:
45
+
## dbt project settings
63
46
64
-
Step 1: create a secret called `GOOGLE_APPLICATION_CREDENTIALS`
47
+
For more information about dbt connection types (Github, Gitlab, Bitbucket, etc.) and the fields required for each type, [read the dbt project section in our connection guide](/get-started/setup-lightdash/connect-project#2-import-a-dbt-project).
65
48
66
-
Add the service account credentials (the JSON file) that you want to use for your GitHub action. It should look something like this:
49
+
Below are details about the univeral fields for all connected dbt projects.
Step 2: create another secret called `DBT_PROFILES`
54
+
**Target** contains information about your dbt connection to your warehouse.
85
55
86
-
Copy-paste this template into the secret and fill out the details.
56
+
It's the dataset or schema in your data warehouse that Lightdash will look for your dbt models. By default, we set this to be the same value as you have as the default in your `profiles.yml` file when you run `lightdash deploy` (if that's how you created or recently deployed your project).
87
57
88
-
This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
58
+
If you want to update this, you can enter the target of your choice in the project settings (for example `prod` or `analytics`.)
89
59
90
-
```yaml
91
-
[my-bigquery-db]: # this is the name of your project
92
-
target: dev
93
-
outputs:
94
-
dev:
95
-
type: bigquery
96
-
method: oauth
97
-
keyfile: keyfile.json # no need to change this! We'll automatically use the keyfile you created in the last step.
98
-
project: [GCP project id]
99
-
dataset: [the name of your dbt dataset]
60
+
[Read more about dbt targets in the dbt docs.](https://docs.getdbt.com/reference/dbt-jinja-functions/target)
100
61
101
-
```
102
62
103
-
More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-file](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-file)
63
+
### dbt selector
104
64
105
-
</Accordion>
65
+
You can filter out models in your dbt project that you don't want to see in Lightdash. This is useful if you have a large
66
+
dbt project and you want to speed up the sync process. Unlike [table selection](/guides/adding-tables-to-lightdash#limiting-the-tables-in-lightdash-using-dbt-tags), this selector is applied to the dbt models, so it will skip the entire compilation process for the models that you don't want to see in Lightdash.
106
67
107
-
<Accordiontitle="Postgres">
108
-
Postgres profile configuration:
68
+
To do this, you can add a **dbt selector** to your project settings. This is a JSON object that contains the models you want to include in Lightdash.
109
69
110
-
```yaml
111
-
company-name:
112
-
target: dev
113
-
outputs:
114
-
dev:
115
-
type: postgres
116
-
host: [hostname]
117
-
user: [username]
118
-
password: [password]
119
-
port: [port]
120
-
dbname: [database name]
121
-
schema: [dbt schema]
122
-
threads: [1 or more]
123
-
keepalives_idle: 0
124
-
connect_timeout: 10
125
-
retries: 1
70
+
For example, if you only want to include the `my_model` and all models with the `lightdash` tag in Lightdash, you can add the following to your dbt project settings:
126
71
72
+
```console
73
+
my_model tag:lightdash
127
74
```
128
75
129
-
More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/postgres-profile#profile-configuration](https://docs.getdbt.com/reference/warehouse-profiles/postgres-profile#profile-configuration)
130
-
131
-
This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
132
-
</Accordion>
76
+
We support all dbt selectors. [Read more about selectors in the dbt docs](https://docs.getdbt.com/reference/node-selection/syntax#combining-state-and-result-selectors).
133
77
134
78
135
-
<Accordiontitle="Redshift">
136
-
Redshift password-based authentication:
79
+
### Environment variables
137
80
138
-
```yaml
139
-
company-name:
140
-
target: dev
141
-
outputs:
142
-
dev:
143
-
type: redshift
144
-
host: [hostname.region.redshift.amazonaws.com]
145
-
user: [username]
146
-
password: [password]
147
-
port: 5439
148
-
dbname: analytics
149
-
schema: analytics
150
-
threads: 4
151
-
keepalives_idle: 240
152
-
connect_timeout: 10
153
-
ra3_node: true # enables cross-database sources
154
-
155
-
```
81
+
If you've used [environment variables in your dbt `profiles.yml` file](https://docs.getdbt.com/reference/dbt-jinja-functions/env%5Fvar), you can add these to Lightdash here.
156
82
157
-
More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/redshift-profile#password-based-authentication](https://docs.getdbt.com/reference/warehouse-profiles/redshift-profile#password-based-authentication)
83
+
For each environment variable, you'll need to add the `key` + `value` pair for the item.
158
84
159
-
This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
160
-
</Accordion>
85
+
You'll normally find these values in a file called `.env` in your dbt project directory.
161
86
162
-
<Accordiontitle="Snowflake">
163
-
User / Private Key authentication:
87
+
For example, I might have something like:
164
88
165
89
```yaml
166
-
my-snowflake-db:
167
-
target: dev
90
+
profile:
91
+
target: prod
168
92
outputs:
169
-
dev:
170
-
type: snowflake
171
-
account: [account id]
172
-
173
-
# User/private_key auth
174
-
private_key_path: [path/to/private.key]
175
-
private_key_passphrase: [passphrase for the private key, if key is encrypted]
176
-
177
-
role: [user role]
178
-
database: [database name]
179
-
warehouse: [warehouse name]
180
-
schema: [dbt schema]
181
-
threads: [1 or more]
182
-
client_session_keep_alive: False
183
-
query_tag: [anything]
184
-
93
+
prod:
94
+
type: postgres
95
+
host: 127.0.0.1
96
+
user: "{{ env_var('DBT_USER') }}"
97
+
....
185
98
```
186
99
187
-
More info in dbt's profiles docs: [https://docs.getdbt.com/docs/core/connect-data-platform/snowflake-setup#key-pair-authentication](https://docs.getdbt.com/docs/core/connect-data-platform/snowflake-setup#key-pair-authentication)
188
-
189
-
This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
190
-
</Accordion>
191
-
192
-
193
-
<Accordiontitle="DataBricks">
194
-
Set up a DataBricks target:
100
+
Then a `.env` file like:
195
101
196
102
```yaml
197
-
your_profile_name:
198
-
target: dev
199
-
outputs:
200
-
dev:
201
-
type: databricks
202
-
catalog:
203
-
[
204
-
optional catalog name,
205
-
if you are using Unity Catalog,
206
-
only available in dbt-databricks>=1.1.1,
207
-
]
208
-
schema: [schema name]
209
-
host: [yourorg.databrickshost.com]
210
-
http_path: [/sql/your/http/path]
211
-
token: [dapiXXXXXXXXXXXXXXXXXXXXXXX] # Personal Access Token (PAT)
212
-
threads: [1 or more]
213
-
103
+
export DBT_USER="myspecialuserkey123"
214
104
```
215
105
216
-
More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-json](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-json)
217
-
218
-
This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
219
-
220
-
</Accordion>
221
-
</AccordionGroup>
222
-
223
-
224
-
### Step 2: Create deploy.yml workflow in Github
225
-
226
-
Go to your repo, click on `Actions` menu.
227
-
228
-
If you don't have any GitHub actions, you'll just need to click on `Configure`
Now copy [this file](https://github.com/lightdash/cli-actions/blob/main/deploy.yml) from our [cli-actions](https://github.com/lightdash/cli-actions) repo.
241
-
242
-
Give it a nice name like `deploy-lightdash.yml`
243
-
244
-
And commit this to your repo by clicking on `Start commit`.
245
-
246
-
### You're done!
247
-
248
-
Everytime you make a change to your repo, on the `main` branch, it will automatically deploy your new config into your Lightdash projects
## 2. In the UI: Syncing your dbt changes using `refresh dbt`
257
-
258
-
Whenever you make changes to your YAML files, you can sync Lightdash and see these changes by clicking the `refresh dbt` button in the Explore view of the app.
If you're using a git connection (like GitHub, Gitlab or Bitbucket), you'll need to push + merge your changes to the branch that your Lightdash project is connected to before you run `refresh dbt`.
265
-
266
-
## 3. From the command line: Syncing your dbt changes using `lightdash deploy`
267
-
268
-
If you're using the [Lightdash CLI](/guides/cli/how-to-install-the-lightdash-cli), you can use the `lightdash deploy` command to deploy your changes to your Lightdash project.
269
-
270
-
To read more about how to use `lightdash deploy`, [check out our docs](/guides/cli/how-to-use-lightdash-deploy).
271
-
272
-
## Note: If you've made any changes to the underlying data, you need to run dbt first
273
-
274
-
If you've made any changes to the underlying data (for example, adding a new column in your `model.sql` file or changing the SQL logic of an existing dimension), then you need to run: `dbt run -m yourmodel` before you click `refresh dbt` in Lightdash.
106
+
So, in Lightdash, I'd add a new environment variable and put `key` as `DBT_USER` and `value` as `myspecialuserkey123`.
0 commit comments