diff --git a/docs/2-getting-started/cloud-5min-tutorial.md b/docs/2-getting-started/cloud-5min-tutorial.md deleted file mode 100644 index b7e086fb..00000000 --- a/docs/2-getting-started/cloud-5min-tutorial.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -title: 5 min tutorial ---- - -# 5-min Toturial - -Coming soon. \ No newline at end of file diff --git a/docs/2-getting-started/gitlab-pat-guide.md b/docs/2-getting-started/gitlab-pat-guide.md index 5f612558..d707744a 100644 --- a/docs/2-getting-started/gitlab-pat-guide.md +++ b/docs/2-getting-started/gitlab-pat-guide.md @@ -80,5 +80,5 @@ Once you have your Personal Access Token: ## Related Documentation -- [Cloud-based Git Platform Integration](./start-free-with-cloud.md#git-integration) +- [Connect Git Provider](./start-free-with-cloud.md#2-connect-git-provider) - [CI/CD Getting Started](../7-cicd/ci-cd-getting-started.md) diff --git a/docs/2-getting-started/start-free-with-cloud.md b/docs/2-getting-started/start-free-with-cloud.md index e637914e..5c72a1be 100644 --- a/docs/2-getting-started/start-free-with-cloud.md +++ b/docs/2-getting-started/start-free-with-cloud.md @@ -1,182 +1,332 @@ --- -title: Start with Recce Cloud +title: Get Started with Recce Cloud --- -# Start with Recce Cloud +# Get Started with Recce Cloud -Validate data changes and collaborate with your team with automation. +This tutorial helps analytics engineers and data engineers set up Recce Cloud to automate data review on pull requests. -πŸ‘‰ **[Start with Recce Cloud](https://cloud.reccehq.com){target="_blank"}** +## Goal -Setup steps: +Reviewing data changes in PRs is error-prone without visibility into downstream impact. After setup, the Recce agent reviews your data changes on every PRβ€”showing what changed and what it affects. -1. [Git Platform Integration](#git-integration) -2. [Data Warehouse Diffing](#data-diffing) -3. [CI/CD Automation](#cicd-automation) +To validate changes, Recce compares **Base** vs **Current** environments: -Fall back to [manual](#manual) if you unable to finish the setup. +- **Base**: models in the main branch (production) +- **Current**: models in the PR branch +Recce requires dbt artifacts from both environments. This guide covers: -## Prerequisites -1. Admin access for git platform -2. Data warehouse credentials with read access -3. - - -## Git Platform Integration {#git-integration} - -Connect your GitHub or GitLab repository to see all PRs/MRs in one place and validate changes before they hit -production. - -### Setup Requirements - -| GitHub | GitLab | -|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| -| β€’ GitHub repository with dbt project
β€’ Repository admin access for initial setup
β€’ Active PRs with model changes | β€’ GitLab project with dbt project
β€’ Project maintainer or owner access for initial setup
β€’ Active merge requests with model changes | - -!!!Note - You'll need administrative access to the Organization/Group you want to connect. Please ensure you have the necessary - permissions for **App installations** (GitHub) or **Providing a Personal Access Token** (GitLab). - -### Connection - -Connect your repository to track pull requests/merge requests and validate changes. - -!!!Note - Keep **Connection setup** note the same as before if there was one specific to this section. +- dbt profile configuration for Base and Current +- CI/CD workflow setup -**Connection setup:** +For accurate comparisons, both environments should use consistent data ranges. See [Best Practices for Preparing Environments](../7-cicd/best-practices-prep-env.md) for environment strategies. -| GitHub | GitLab | -|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 1. Navigate to settings
2. Connect GitHub repository
3. Authorize Recce access
4. Select repository | 1. Navigate to settings
2. Connect GitLab by providing a Personal Access Token ([see our directions here](./gitlab-pat-guide.md))
3. Connect a project by adding a GitLab Project or URL to a Recce Project | +This guide uses Snowflake, GitHub, and GitHub Actions examples, but can be adapted to your configuration. The setup assumes: +- Production runs a full daily refresh +- No pre-configured per-PR environments exist yet +- Each developer has their own dev environment for local work -### How to Use PR/MR Tracking +We'll configure CI to create isolated, per-PR schemas automatically. -Once connected, Recce displays all open and draft PRs/MRs in your Recce Cloud project. +## Prerequisites -### PR/MR Validation Workflow +- [x] **Recce Cloud account**: free trial at [cloud.reccehq.com](https://cloud.reccehq.com) +- [x] **dbt project in a git repository that runs successfully:** your environment can execute `dbt build` and `dbt docs generate` +- [x] **Repository admin access for setup**: required to add workflows and secrets +- [x] **Data warehouse**: read access to your warehouse for data diffing -- View open/draft PRs/MRs in dashboard -- Select PR/MR to validate -- Upload PR/MR metadata (until CI/CD is configured) -- Launch Recce to analyze changes +## Onboarding Process Overview -## Data Warehouse Diffing {#data-diffing} +After signing up, you'll enter the onboarding flow: -Do data diffing to see how changes affect your actual data. Configure your data warehouse connection to compare query -results and catch issues before they impact production. +1. Connect data warehouse +2. Connect Git provider +3. Add Recce to CI/CD +4. Merge the CI/CD change -### Setup Requirements +## Recce Web Agent Setup [Experimental] -- Data warehouse credentials with read access -- Network connectivity to your warehouse +You can use the Recce Web Agent to help automate your setup. Currently it handles **step 3** (Add Recce to CI/CD): -### Supported Warehouses +1. The agent analyzes your repository and CI/CD setup +2. You answer clarifying questions the agent asks about your environment strategy +3. The agent creates a PR with customized workflow files -- Snowflake -- Databricks -- Others coming in future releases +The agent covers common setups and continues to expand coverage. If your setup isn't supported yet, the agent directs you to the Setup Guide below for manual configuration. Need help? Contact us at support@reccehq.com. -### Warehouse Connection +**Coming soon**: The agent will guide you through steps 1–3, including warehouse connection, Git connection, and CI/CD configuration. -Configure connection to your data warehouse to enable query result comparisons. +--- -**Quick setup:** +## Setup Guide -1. In your [project home](https://cloud.reccehq.com/), click the gear icon next to **Warehouse Connection** -2. Create a new connection or select an existing one from the dropdown -3. Your connection is linked and ready to use +This guide explains each onboarding step in detail. -For detailed connection settings, see [Connect to Warehouse](../5-data-diffing/connect-to-warehouse.md). Connection -credentials are encrypted and secure, see our [security practices](https://reccehq.com/security/). +First, go to [cloud.reccehq.com](https://cloud.reccehq.com) and create your free account. +### 1. Connect Data Warehouse -## CI/CD Automation {#cicd-automation} +1. Select your data warehouse (e.g. Snowflake) +2. Provide your read-only warehouse credentials -Set up CI/CD to automatically upload metadata and run validation checks on every PR. +> **Note**: This guide uses Snowflake. For supported warehouses, see [Connect to Warehouse](../5-data-diffing/connect-to-warehouse.md). -!!!Note - Available with Team plan (free trial included). +### 2. Connect Git Provider -### Setup Requirements +1. Click **Connect GitHub** +2. Authorize the Recce app installation +3. Select the repositories you want to connect -See the CI/CD sections for complete setup guides: +> **Note**: This guide uses GitHub. For GitLab setup, see [GitLab Personal Access Token](gitlab-pat-guide.md). -- [Getting Started with CI/CD](../7-cicd/ci-cd-getting-started.md) -- GitHub CI/CD - - [Setup CI for GitHub](../7-cicd/github/setup-ci.md) - - [Setup CD for GitHub](../7-cicd/github/setup-cd.md) -- GitLab CI/CD - - [Setup CI for GitLab](../7-cicd/gitlab/setup-ci.md) - - [Setup CD for GitLab](../7-cicd/gitlab/setup-cd.md) +### 3. Add Recce to CI/CD -### Automation Benefits +This step adds CI/CD workflow files to your repository. The agent creates these automatically. For manual setup, create and merge a PR with the templates below. -- Automatic metadata upload on every PR -- Consistent validation across all PRs -- Reduced manual setup steps -- Integrated PR status checks -- Validation results directly in PR +> **Note**: This guide uses GitHub Actions. For other CI/CD platforms, see [Setup CD](../7-cicd/setup-cd.md) and [Setup CI](../7-cicd/setup-ci.md). +#### Set Up Profile.yml -## Manual Upload Metadata {#manual} +The profile.yml file tells your system where to look for the "base" and "current" builds. We have a sample `profile.yml` file: -Recce shows what changed between base and current environments and helps assess potential impact. The most common case is comparing your development branch against your production or main branch to see what your changes will impact. +```yaml +: + target: dev + outputs: + dev: + type: snowflake + account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}" + user: "{{ env_var('SNOWFLAKE_USER') | as_text }}" + password: "{{ env_var('SNOWFLAKE_PASSWORD') | as_text }}" + role: DEVELOPER + database: cloud_database + warehouse: LOAD_WH + schema: "{{ env_var('SNOWFLAKE_SCHEMA') | as_text }}" + threads: 4 -If you don’t use the GitHub/GitLab or havn’t setup CI/CD yet, you can manual upload + ## Add a new target for CI + ci: + type: snowflake + account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}" + user: "{{ env_var('SNOWFLAKE_USER') | as_text }}" + password: "{{ env_var('SNOWFLAKE_PASSWORD') | as_text }}" + role: DEVELOPER + database: cloud_database + warehouse: LOAD_WH + schema: "{{ env_var('SNOWFLAKE_SCHEMA') | as_text }}" + threads: 4 -- Web interface: Click "Update" on the session you want to update in Recce Cloud. - 1. Click "Update" in base session to upload baseline metadata - 2. Click "Update" in current session to upload comparison metadata - 3. Click "Launch" to compare current against base -- CLI command: + prod: + type: snowflake + account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}" + user: "{{ env_var('SNOWFLAKE_USER') | as_text }}" + password: "{{ env_var('SNOWFLAKE_PASSWORD') | as_text }}" + role: DEVELOPER + database: cloud_database + warehouse: LOAD_WH + schema: PUBLIC + threads: 4 +``` +In this sample: + +1. **Base** uses the `prod` target pointing to the `PUBLIC` schema (your production data) +2. **Current** uses the `ci` target with a dynamic schema via `env_var('SNOWFLAKE_SCHEMA')` + +The `ci` target uses an environment variable for the schema name. In `pr-workflow.yml` below, we set `SNOWFLAKE_SCHEMA: "PR_${{ github.event.pull_request.number }}"` to create isolated environments per PR (e.g., `PR_123`, `PR_456`). This isolates each PR's data so multiple PRs can run without conflicts. + +> NOTE: Ensure your data warehouse allows creating schemas dynamically. The CI runner needs write permissions to create PR-specific schemas (e.g., `PR_123`). + +#### About Secrets + +The workflows use two types of secrets: + +- **`GITHUB_TOKEN`**: automatically provided by GitHub Actions, no configuration needed. This is used by the GitHub integration you just set up to connect the results of the call to Recce. +- **Warehouse credentials**: your existing secrets for dbt (e.g., `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_USER`, `SNOWFLAKE_PASSWORD`). If your dbt project already runs in CI, you have these configured. + +#### Set Up Base Metadata Updates + +The Base environment should reflect the dbt configuration in the main branch. Example workflow file: `base-workflow.yml` + +```yaml +name: Update Base Metadata +on: + push: + branches: ["main"] + schedule: + - cron: "0 2 * * *" + workflow_dispatch: + +concurrency: + group: ${{ github.workflow }} + cancel-in-progress: true + +jobs: + update-base-session: + runs-on: ubuntu-latest + timeout-minutes: 30 + permissions: + contents: read + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: "pip" + + - name: Install dependencies + run: pip install -r requirements.txt + + - name: Prepare dbt artifacts + run: | + dbt deps + dbt build --target prod + dbt docs generate --target prod + env: + DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }} + SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} + SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }} + SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} + SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }} + SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} + + ## Add this part + - name: Upload to Recce Cloud + run: | + pip install recce-cloud + recce-cloud upload --type prod + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` -recce upload-session --session-id + +This sample workflow: + +- Runs once a day +- Installs Python 3.11 and the contents of `requirements.txt`, and recce-cloud +- **Calls `dbt docs generate`** to generate artifacts +- **Calls `recce-cloud upload --type prod`** to upload the Base metadata, using `GITHUB_TOKEN` for authentication + +To integrate into your own configuration, ensure your workflow includes the bolded steps. + +#### Set Up Current Metadata Updates + +The Current environment should reflect the dbt configuration in the PR branch. Recce provides an example workflow file: `pr-workflow.yml` + +```yaml +name: Validate PR Changes +on: + pull_request: + branches: ["main"] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + validate-changes: + runs-on: ubuntu-latest + timeout-minutes: 45 + permissions: + contents: read + pull-requests: write + steps: + - name: Checkout PR branch + uses: actions/checkout@v4 + with: + fetch-depth: 2 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + cache: "pip" + + - name: Install dependencies + run: pip install -r requirements.txt + + - name: Build current branch artifacts + run: | + dbt deps + dbt build --target ci + dbt docs generate --target ci + env: + DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }} + SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} + SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }} + SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} + SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }} + SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} + SNOWFLAKE_SCHEMA: "PR_${{ github.event.pull_request.number }}" + + - name: Upload to Recce Cloud + run: | + pip install recce-cloud + recce-cloud upload + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} ``` -Find your session ID in Recce Cloud web interface when clicking "Update" on any session. +This sample workflow: -### Required Files +- Runs on every PR targeting main +- Installs Python 3.11, dependencies from `requirements.txt`, and recce-cloud +- **Creates a per-PR schema** (`PR_123`, `PR_456`, etc.) using the dynamic `SNOWFLAKE_SCHEMA` environment variableβ€”this isolates each PR's data so multiple PRs can run simultaneously without conflicts +- **Calls `dbt docs generate --target ci`** to generate artifacts for the PR branch +- **Calls `recce-cloud upload`** to upload the Current metadata, using `GITHUB_TOKEN` for authentication -Recce needs `manifest.json` and `catalog.json` from both **base** and **current** environments for comparison. +To integrate into your own configuration, ensure your workflow includes the bolded steps. -#### Base Metadata +### 4. Merge the CI/CD change -Production environment is commonly used as the baseline, but any environment can serve as the base. +Merge the PR containing the workflow files. After merging: -Choose one method: +- The **Base workflow** automatically uploads your Base to Recce Cloud +- The **Current workflow** is ready to validate future PRs -**Method 1: Generate locally** +In Recce Cloud, verify you see: -``` -dbt docs generate --target-path target-base --target -``` +- GitHub Integration: Connected +- Warehouse Connection: Connected +- Production Metadata: Updated automatically +- PR Sessions: all open PRs appear in the list. Only PRs with uploaded metadata can be launched for review. -**Method 2: dbt Cloud**
-Deploy β†’ Jobs β†’ Production job β†’ Recent run β†’ Download artifacts +![Recce Cloud dashboard after setup](../assets/images/2-getting-started/cloud-onboarding-completed.png){: .shadow} -**Method 3: dbt Docs server**
-Download the artifacts directly from dbt docs server: +### 5. Final Steps -- `/manifest.json` -- `/catalog.json` +You can now: -#### Current Metadata +- See data review summaries in PR comments +- Launch Recce instance to visualize changes +- Review downstream impacts before merging -Use development environment or PR branch as current to compare against the base. +--- -Choose one method: +## Verification Checklist -**Method 1: Generate locally** +- [x] **Base workflow**: Trigger manually, check Base metadata appears in Recce Cloud +- [x] **Current workflow**: Create a test PR, verify PR session appears +- [x] **Data diff**: Open PR session, run Row Count Diff -``` -dbt docs generate --target -``` +## Troubleshooting -**Method 2: dbt Cloud**
-Deploy β†’ Jobs β†’ CI job β†’ Recent run β†’ Download artifacts +| Issue | Solution | +| --- | --- | +| Authentication errors | Confirm repository is connected in Recce Cloud settings | +| Push to main blocked | Check branch protection rules | +| Secret names don't match | Update template to use your existing secret names | +| Workflow fails | Check secrets are configured correctly | +| Artifacts missing | Ensure `dbt docs generate` completes before upload | +| Warehouse connection fails | Check IP whitelisting; add GitHub Actions IP ranges | +## Related Resources +- [CI/CD Getting Started](../7-cicd/ci-cd-getting-started.md) +- [Setup CD](../7-cicd/setup-cd.md) +- [Setup CI](../7-cicd/setup-ci.md) +- [Best Practices for Preparing Environments](../7-cicd/best-practices-prep-env.md) diff --git a/docs/7-cicd/ci-cd-getting-started.md b/docs/7-cicd/ci-cd-getting-started.md index bec9faaa..eda39817 100644 --- a/docs/7-cicd/ci-cd-getting-started.md +++ b/docs/7-cicd/ci-cd-getting-started.md @@ -55,10 +55,10 @@ Recce integrates with both GitHub Actions and GitLab CI/CD using the lightweight Before setting up, ensure you have: -- βœ… **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) -- βœ… **Repository connected** to Recce Cloud - [Git integration guide](../2-getting-started/start-free-with-cloud.md#git-integration) +- [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) +- [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider) - For GitLab: [Create a Personal Access Token](../2-getting-started/gitlab-pat-guide.md) if not already done -- βœ… **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your project +- [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your project ## Setup Steps diff --git a/docs/7-cicd/setup-cd.md b/docs/7-cicd/setup-cd.md index 9541936c..075d0d55 100644 --- a/docs/7-cicd/setup-cd.md +++ b/docs/7-cicd/setup-cd.md @@ -18,24 +18,24 @@ Set up automatic updates for your Recce Cloud base sessions. Keep your data comp Before setting up CD, ensure you have: -- βœ… **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) -- βœ… **Repository connected** to Recce Cloud - [Git integration guide](../2-getting-started/start-free-with-cloud.md#git-integration) -- βœ… **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project +- [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) +- [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider) +- [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project ## Setup ### GitHub Actions -Create `.github/workflows/cd-workflow.yml`: +Create `.github/workflows/base-workflow.yml`: -```yaml linenums="1" hl_lines="42-43" -name: Update Base Recce Session +```yaml linenums="1" +name: Update Base Metadata on: push: branches: ["main"] schedule: - - cron: "0 2 * * *" # Daily at 2 AM UTC + - cron: "0 2 * * *" workflow_dispatch: concurrency: @@ -65,10 +65,14 @@ jobs: - name: Prepare dbt artifacts run: | dbt deps - # Optional: dbt build --target prod + dbt build --target prod dbt docs generate --target prod env: - DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }} + SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} + SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }} + SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} + SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }} + SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} - name: Upload to Recce Cloud run: | @@ -80,9 +84,9 @@ jobs: **Key points:** -- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) is passed explicitly for Recce Cloud authentication -- `recce-cloud upload --type prod` tells Recce this is a baseline session -- `dbt docs generate` creates the required `manifest.json` and `catalog.json` +- `dbt build` and `dbt docs generate` create the required artifacts (`manifest.json` and `catalog.json`) +- `recce-cloud upload --type prod` uploads the Base metadata to Recce Cloud +- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) authenticates with Recce Cloud ### GitLab CI/CD @@ -137,7 +141,7 @@ recce-upload-prod: | Aspect | GitHub Actions | GitLab CI/CD | | -------------------- | ----------------------------------- | ------------------------------------------------------------------------------ | -| **Config file** | `.github/workflows/cd-workflow.yml` | `.gitlab-ci.yml` | +| **Config file** | `.github/workflows/base-workflow.yml` | `.gitlab-ci.yml` | | **Trigger on merge** | `on: push: branches: ["main"]` | `if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH` | | **Schedule setup** | In workflow YAML (`schedule:`) | In UI: **CI/CD β†’ Schedules** | | **Authentication** | Explicit (`GITHUB_TOKEN`) | Automatic (`CI_JOB_TOKEN`) | @@ -161,8 +165,8 @@ recce-upload-prod: Look for these indicators: -- βœ… **Workflow/Pipeline completes** without errors -- βœ… **Base session updated** in [Recce Cloud](https://cloud.reccehq.com) +- [x] **Workflow/Pipeline completes** without errors +- [x] **Base session updated** in [Recce Cloud](https://cloud.reccehq.com) **GitHub:** diff --git a/docs/7-cicd/setup-ci.md b/docs/7-cicd/setup-ci.md index 288d545d..9310ee03 100644 --- a/docs/7-cicd/setup-ci.md +++ b/docs/7-cicd/setup-ci.md @@ -18,18 +18,18 @@ Automatically validate your data changes in every pull request or merge request Before setting up CI, ensure you have: -- βœ… **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) -- βœ… **Repository connected** to Recce Cloud - [Git integration guide](../2-getting-started/start-free-with-cloud.md#git-integration) -- βœ… **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project -- βœ… **CD configured** - [Setup CD](./setup-cd.md) to establish baseline for comparisons +- [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/) +- [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider) +- [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project +- [x] **CD configured** - [Setup CD](./setup-cd.md) to establish baseline for comparisons ## Setup ### GitHub Actions -Create `.github/workflows/ci-workflow.yml`: +Create `.github/workflows/pr-workflow.yml`: -```yaml linenums="1" hl_lines="41-42" +```yaml linenums="1" name: Validate PR Changes on: @@ -66,10 +66,15 @@ jobs: - name: Build current branch artifacts run: | dbt deps - # Optional: dbt build --target ci + dbt build --target ci dbt docs generate --target ci env: - DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }} + SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }} + SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }} + SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }} + SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }} + SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }} + SNOWFLAKE_SCHEMA: "PR_${{ github.event.pull_request.number }}" - name: Upload to Recce Cloud run: | @@ -81,9 +86,10 @@ jobs: **Key points:** -- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) is passed explicitly for Recce Cloud authentication +- Creates a per-PR schema (`PR_123`, `PR_456`, etc.) using the dynamic `SNOWFLAKE_SCHEMA` environment variable to isolate each PR's data +- `dbt build` and `dbt docs generate` create the required artifacts (`manifest.json` and `catalog.json`) - `recce-cloud upload` (without `--type`) auto-detects this is a PR session -- `dbt docs generate` creates the required `manifest.json` and `catalog.json` +- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) authenticates with Recce Cloud ### GitLab CI/CD @@ -136,7 +142,7 @@ recce-upload: | Aspect | GitHub Actions | GitLab CI/CD | | -------------------- | ----------------------------------- | -------------------------------------------------- | -| **Config file** | `.github/workflows/ci-workflow.yml` | `.gitlab-ci.yml` | +| **Config file** | `.github/workflows/pr-workflow.yml` | `.gitlab-ci.yml` | | **Trigger** | `on: pull_request:` | `if: $CI_PIPELINE_SOURCE == "merge_request_event"` | | **Authentication** | Explicit (`GITHUB_TOKEN`) | Automatic (`CI_JOB_TOKEN`) | | **Session type** | Auto-detected from PR context | Auto-detected from MR context | @@ -162,9 +168,9 @@ recce-upload: Look for these indicators: -- βœ… **Workflow/Pipeline completes** without errors -- βœ… **PR/MR session created** in [Recce Cloud](https://cloud.reccehq.com) -- βœ… **Session URL** appears in workflow/pipeline output +- [x] **Workflow/Pipeline completes** without errors +- [x] **PR/MR session created** in [Recce Cloud](https://cloud.reccehq.com) +- [x] **Session URL** appears in workflow/pipeline output **GitHub:** diff --git a/docs/assets/images/2-getting-started/cloud-onboarding-completed.png b/docs/assets/images/2-getting-started/cloud-onboarding-completed.png new file mode 100644 index 00000000..50d87422 Binary files /dev/null and b/docs/assets/images/2-getting-started/cloud-onboarding-completed.png differ diff --git a/mkdocs.yml b/mkdocs.yml index 92260c1f..98471d62 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -119,6 +119,8 @@ markdown_extensions: - name: mermaid class: mermaid format: !!python/name:pymdownx.superfences.fence_code_format + - pymdownx.tasklist: + custom_checkbox: true - pymdownx.tabbed: alternate_style: true