update connection page and dbt project reference

docs.json

@@ -68,7 +68,7 @@
               {
                 "group": "Exploring Data in Lightdash",
                 "pages": [
-                  "guides/ai-analyst",
+                  "guides/ai-agents",
                   "guides/metrics-catalog",
                   "guides/limiting-data-using-filters",                  
                   "guides/interactive-dashboards",
@@ -170,7 +170,7 @@
                   "references/filters",
                   "references/table-calculations",
                   "references/custom-fields",
-                  "references/custom-tooltips",
+                  "references/custom-tooltip",
                   "references/custom-charts"
                 ]
               },

references/dbt-projects.mdx

@@ -4,271 +4,103 @@ description: "You can easily make changes in dbt and see them updated in your Li
 ---
 
 <Info>
-
-Lightdash supports dbt v1.4.0 and above. If you are using an older version of dbt, you will need to upgrade to sync your project to Lightdash
-
+  Lightdash supports dbt v1.4.0 and above. If you are using an older version of dbt, you will need to upgrade to sync your project to Lightdash
 </Info>
 
-## 1. Automatically: deploy your changes to Lightdash using a GitHub action
-
-If you've connected Lightdash to GitHub, you can setup a `github action` and get Lightdash to deploy your project automatically. This is the easiest way to keep Lightdash in sync with your changes in dbt.
-
-### Step 1: add the credentials to Github secrets
 
-We are going to add some secrets and config to GitHub actions, but you don't want those to be public, so the best way to do this is to add them as secrets on Github.
+## Syncing your dbt project to Lightdash 
 
-<Info>
-
-If you already have a GitHub action for Lightdash, then you can use the same Lightdash secrets you created for your other action.
-</Info>
+You can sync your dbt project code with Lightdash in a few different ways. We recommend everyone set up continuous deployment, but you can also refresh in the Lightdash app or deploy from the CLI.
 
-Go to your repo, click on `Settings` , on the left sidebar, click on `Secrets` under `Security`. Now click on the `New repository secret`
 
-<Frame>
- ![](/images/references/github-secrets-fc35152c7ebb768974b385a0e2601dc5.jpg)
-</Frame>
+### 1. Set up continous deployment
 
+[Read how to do that and check out our example workflow files](/guides/cli/how-to-use-lightdash-deploy#automatically-deploy-your-changes-to-lightdash-using-a-github-action). 
 
-We need to add the following secrets:
 
-##### `LIGHTDASH_API_KEY`
+### 2. Click "Refresh dbt" in Lightdash
 
-Create a new personal access token, by going to `Settings` \> `Personal Access Tokens`. This is the token you'll put in for `LIGHTDASH_API_KEY`.
+The button can be found on the Query from tables page. 
 
 <Frame>
-  <img src="/images/references/personal-access-token-9ed8a8c2b16edc5ee745c4909d4093d3.png"/>
+  ![screenshot-refresh-dbt](/images/references/dbt-refresh.png)
 </Frame>
 
-##### `LIGHTDASH_PROJECT`
-
-The UUID for your project. For example, if your URL looks like `https://eu1.lightdash.cloud/projects/3538ab33-dc90-aabb-bc00-e50bba3a5f69/tables`, then `3538ab33-dc90-45f0-aabb-e50bba3a5f69` is your `LIGHTDASH_PROJECT`
+_If you're using a git connection (like GitHub, Gitlab or Bitbucket), you'll need to push + merge your changes to the branch that your Lightdash project is connected to before you press `Refresh dbt`._
 
-##### `LIGHTDASH_URL`
+<Info>
+If you've made any changes to the underlying data (for example, adding a new column in your `model.sql` file or changing the SQL logic of a dimension), then you need to run: `dbt run -m yourmodel` before you click `Refresh dbt` in Lightdash.
+</Info>
 
-This is either `https://eu1.lightdash.cloud` or `https://app.lightdash.cloud` for Lightdash Cloud users (check the URL to your Lightdash project). If you self-host, this should be your own custom domain.
 
-##### `DBT_PROFILES`
+### 3. Push code from the CLI
 
-Some tips for this bit:
+If you're using the [Lightdash CLI](/guides/cli/how-to-install-the-lightdash-cli), you can use `lightdash deploy` to deploy your changes to Lightdash. [Read more about how to use `lightdash deploy`](/guides/cli/how-to-use-lightdash-deploy).
 
-* You might be able to copy a bunch of the information from your local `profiles.yml` file. You can see what's in there by typing `cat ~/.dbt/profiles.yml` in your terminal.
-* If you have a separate `prod` and `dev` profile, you probably want to use the information from your `prod` profile for your GitHub action.
-* If you want to have different connection settings depending on the user that opened the pull request (dev profiles), then [check out this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
+<Warning>
+  We don't recommend using `lightdash deploy` from your local environment as the primary way you update Lightdash since small mistakes can lead to production issues.
+</Warning>
 
-Find your data warehouse from the list below to get a profiles.yml file template. Fill out this template, and this is your `DBT_PROFILES` secret.
 
-<AccordionGroup>
-<Accordion title="BigQuery">
-BigQuery OAuth:
+## dbt project settings
 
-Step 1: create a secret called `GOOGLE_APPLICATION_CREDENTIALS`
+For more information about dbt connection types (Github, Gitlab, Bitbucket, etc.) and the fields required for each type, [read the dbt project section in our connection guide](/get-started/setup-lightdash/connect-project#2-import-a-dbt-project).
 
-Add the service account credentials (the JSON file) that you want to use for your GitHub action. It should look something like this:
+Below are details about the univeral fields for all connected dbt projects.
 
-```json
-{
-  "type": "service_account",
-  "project_id": "jaffle_shop",
-  "private_key_id": "12345",
-  "private_key": "-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----\n",
-  "client_email": "jaffle_shop@jaffle_shop.iam.gserviceaccount.com",
-  "client_id": "12345",
-  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
-  "token_uri": "https://oauth2.googleapis.com/token",
-  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
-  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/jaffle_shop"
-}
 
-```
+### Target name
 
-Step 2: create another secret called `DBT_PROFILES`
+**Target** contains information about your dbt connection to your warehouse.
 
-Copy-paste this template into the secret and fill out the details.
+It's the dataset or schema in your data warehouse that Lightdash will look for your dbt models. By default, we set this to be the same value as you have as the default in your `profiles.yml` file when you run `lightdash deploy` (if that's how you created or recently deployed your project).
 
-This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
+If you want to update this, you can enter the target of your choice in the project settings (for example `prod` or `analytics`.)
 
-```yaml
-[my-bigquery-db]: # this is the name of your project
-  target: dev
-  outputs:
-    dev:
-      type: bigquery
-      method: oauth
-      keyfile: keyfile.json # no need to change this! We'll automatically use the keyfile you created in the last step.
-      project: [GCP project id]
-      dataset: [the name of your dbt dataset]
+[Read more about dbt targets in the dbt docs.](https://docs.getdbt.com/reference/dbt-jinja-functions/target)
 
-```
 
-More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-file](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-file)
+### dbt selector
 
-</Accordion>
+You can filter out models in your dbt project that you don't want to see in Lightdash. This is useful if you have a large
+dbt project and you want to speed up the sync process. Unlike [table selection](/guides/adding-tables-to-lightdash#limiting-the-tables-in-lightdash-using-dbt-tags), this selector is applied to the dbt models, so it will skip the entire compilation process for the models that you don't want to see in Lightdash.
 
-<Accordion title="Postgres">
-Postgres profile configuration:
+To do this, you can add a **dbt selector** to your project settings. This is a JSON object that contains the models you want to include in Lightdash.
 
-```yaml
-company-name:
-  target: dev
-  outputs:
-    dev:
-      type: postgres
-      host: [hostname]
-      user: [username]
-      password: [password]
-      port: [port]
-      dbname: [database name]
-      schema: [dbt schema]
-      threads: [1 or more]
-      keepalives_idle: 0
-      connect_timeout: 10
-      retries: 1
+For example, if you only want to include the `my_model` and all models with the `lightdash` tag in Lightdash, you can add the following to your dbt project settings:
 
+```console
+my_model tag:lightdash
 ```
 
-More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/postgres-profile#profile-configuration](https://docs.getdbt.com/reference/warehouse-profiles/postgres-profile#profile-configuration)
-
-This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
-</Accordion>
+We support all dbt selectors. [Read more about selectors in the dbt docs](https://docs.getdbt.com/reference/node-selection/syntax#combining-state-and-result-selectors).
 
 
-<Accordion title="Redshift">
-Redshift password-based authentication:
+### Environment variables
 
-```yaml
-company-name:
-  target: dev
-  outputs:
-    dev:
-      type: redshift
-      host: [hostname.region.redshift.amazonaws.com]
-      user: [username]
-      password: [password]
-      port: 5439
-      dbname: analytics
-      schema: analytics
-      threads: 4
-      keepalives_idle: 240
-      connect_timeout: 10
-      ra3_node: true # enables cross-database sources
-
-```
+If you've used [environment variables in your dbt `profiles.yml` file](https://docs.getdbt.com/reference/dbt-jinja-functions/env%5Fvar), you can add these to Lightdash here.
 
-More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/redshift-profile#password-based-authentication](https://docs.getdbt.com/reference/warehouse-profiles/redshift-profile#password-based-authentication)
+For each environment variable, you'll need to add the `key` + `value` pair for the item.
 
-This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
-</Accordion>
+You'll normally find these values in a file called `.env` in your dbt project directory.
 
-<Accordion title="Snowflake">
-User / Private Key authentication:
+For example, I might have something like:
 
 ```yaml
-my-snowflake-db:
-  target: dev
+profile:
+  target: prod
   outputs:
-    dev:
-      type: snowflake
-      account: [account id]
-
-      # User/private_key auth
-      private_key_path: [path/to/private.key]
-      private_key_passphrase: [passphrase for the private key, if key is encrypted]
-
-      role: [user role]
-      database: [database name]
-      warehouse: [warehouse name]
-      schema: [dbt schema]
-      threads: [1 or more]
-      client_session_keep_alive: False
-      query_tag: [anything]
-
+    prod:
+      type: postgres
+      host: 127.0.0.1
+      user: "{{ env_var('DBT_USER') }}"
+      ....
 ```
 
-More info in dbt's profiles docs: [https://docs.getdbt.com/docs/core/connect-data-platform/snowflake-setup#key-pair-authentication](https://docs.getdbt.com/docs/core/connect-data-platform/snowflake-setup#key-pair-authentication)
-
-This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
-</Accordion>
-
-
-<Accordion title="DataBricks">
-Set up a DataBricks target:
+Then a `.env` file like:
 
 ```yaml
-your_profile_name:
-  target: dev
-  outputs:
-    dev:
-      type: databricks
-      catalog:
-        [
-          optional catalog name,
-          if you are using Unity Catalog,
-          only available in dbt-databricks>=1.1.1,
-        ]
-      schema: [schema name]
-      host: [yourorg.databrickshost.com]
-      http_path: [/sql/your/http/path]
-      token: [dapiXXXXXXXXXXXXXXXXXXXXXXX] # Personal Access Token (PAT)
-      threads: [1 or more]
-
+export DBT_USER="myspecialuserkey123"
 ```
 
-More info in dbt's profiles docs: [https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-json](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-json)
-
-This will always use this project connection in your GitHub actions. If you want your preview projects to have different connection settings depending on the user that opened the pull request (dev profiles), then see what you need to add to your secret [in this guide](/guides/cli/how-to-use-lightdash-preview#how-to-use-the-developer-credentials-in-your-preview-project).
-
-</Accordion>
-</AccordionGroup>
-
-
-### Step 2: Create deploy.yml workflow in Github
-
-Go to your repo, click on `Actions` menu.
-
-If you don't have any GitHub actions, you'll just need to click on `Configure`
-
-<Frame>
-![Github actions page](/images/references/github-actions-f74f88aa1b1413a111be7dbf16d65d53.png)
-</Frame>
-
-If you have some GitHub actions in your repo already, click on `New workflow`, then select `setup a workflow yourself`.
-
-<Frame>
-  ![](/images/references/new-workflow-existing-actions-e321bd117d93bf139a01424d3771d0e6.jpg)
-</Frame>
-
-Now copy [this file](https://github.com/lightdash/cli-actions/blob/main/deploy.yml) from our [cli-actions](https://github.com/lightdash/cli-actions) repo.
-
-Give it a nice name like `deploy-lightdash.yml`
-
-And commit this to your repo by clicking on `Start commit`.
-
-### You're done!
-
-Everytime you make a change to your repo, on the `main` branch, it will automatically deploy your new config into your Lightdash projects
-
-You can see the log on the `Github actions` page
-
-<Frame>
-  ![](/images/references/github-run-e24152442d7002253f6fd6dc66a9a24e.png)
-</Frame>
-
-## 2. In the UI: Syncing your dbt changes using `refresh dbt`
-
-Whenever you make changes to your YAML files, you can sync Lightdash and see these changes by clicking the `refresh dbt` button in the Explore view of the app.
-
-<Frame>
-  ![screenshot-refresh-dbt](/images/references/screenshot-refresh-dbt-3b6ee58ee347b0b10cd89aacf09e3abf.png)
-</Frame>
-
-If you're using a git connection (like GitHub, Gitlab or Bitbucket), you'll need to push + merge your changes to the branch that your Lightdash project is connected to before you run `refresh dbt`.
-
-## 3. From the command line: Syncing your dbt changes using `lightdash deploy`
-
-If you're using the [Lightdash CLI](/guides/cli/how-to-install-the-lightdash-cli), you can use the `lightdash deploy` command to deploy your changes to your Lightdash project.
-
-To read more about how to use `lightdash deploy`, [check out our docs](/guides/cli/how-to-use-lightdash-deploy).
-
-## Note: If you've made any changes to the underlying data, you need to run dbt first
-
-If you've made any changes to the underlying data (for example, adding a new column in your `model.sql` file or changing the SQL logic of an existing dimension), then you need to run: `dbt run -m yourmodel` before you click `refresh dbt` in Lightdash.
+So, in Lightdash, I'd add a new environment variable and put `key` as `DBT_USER` and `value` as `myspecialuserkey123`.