Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions docs/2-getting-started/cloud-5min-tutorial.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/2-getting-started/gitlab-pat-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,5 +80,5 @@ Once you have your Personal Access Token:

## Related Documentation

- [Cloud-based Git Platform Integration](./start-free-with-cloud.md#git-integration)
- [Connect Git Provider](./start-free-with-cloud.md#2-connect-git-provider)
- [CI/CD Getting Started](../7-cicd/ci-cd-getting-started.md)
386 changes: 268 additions & 118 deletions docs/2-getting-started/start-free-with-cloud.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions docs/7-cicd/ci-cd-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,10 +55,10 @@ Recce integrates with both GitHub Actions and GitLab CI/CD using the lightweight

Before setting up, ensure you have:

- **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/)
- **Repository connected** to Recce Cloud - [Git integration guide](../2-getting-started/start-free-with-cloud.md#git-integration)
- [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/)
- [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider)
- For GitLab: [Create a Personal Access Token](../2-getting-started/gitlab-pat-guide.md) if not already done
- **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your project
- [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your project

## Setup Steps

Expand Down
34 changes: 19 additions & 15 deletions docs/7-cicd/setup-cd.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,24 @@ Set up automatic updates for your Recce Cloud base sessions. Keep your data comp

Before setting up CD, ensure you have:

- **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/)
- **Repository connected** to Recce Cloud - [Git integration guide](../2-getting-started/start-free-with-cloud.md#git-integration)
- **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project
- [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/)
- [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider)
- [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project

## Setup

### GitHub Actions

Create `.github/workflows/cd-workflow.yml`:
Create `.github/workflows/base-workflow.yml`:

```yaml linenums="1" hl_lines="42-43"
name: Update Base Recce Session
```yaml linenums="1"
name: Update Base Metadata

on:
push:
branches: ["main"]
schedule:
- cron: "0 2 * * *" # Daily at 2 AM UTC
- cron: "0 2 * * *"
workflow_dispatch:

concurrency:
Expand Down Expand Up @@ -65,10 +65,14 @@ jobs:
- name: Prepare dbt artifacts
run: |
dbt deps
# Optional: dbt build --target prod
dbt build --target prod
dbt docs generate --target prod
env:
DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }}
SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }}
SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }}

- name: Upload to Recce Cloud
run: |
Expand All @@ -80,9 +84,9 @@ jobs:

**Key points:**

- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) is passed explicitly for Recce Cloud authentication
- `recce-cloud upload --type prod` tells Recce this is a baseline session
- `dbt docs generate` creates the required `manifest.json` and `catalog.json`
- `dbt build` and `dbt docs generate` create the required artifacts (`manifest.json` and `catalog.json`)
- `recce-cloud upload --type prod` uploads the Base metadata to Recce Cloud
- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) authenticates with Recce Cloud

### GitLab CI/CD

Expand Down Expand Up @@ -137,7 +141,7 @@ recce-upload-prod:

| Aspect | GitHub Actions | GitLab CI/CD |
| -------------------- | ----------------------------------- | ------------------------------------------------------------------------------ |
| **Config file** | `.github/workflows/cd-workflow.yml` | `.gitlab-ci.yml` |
| **Config file** | `.github/workflows/base-workflow.yml` | `.gitlab-ci.yml` |
| **Trigger on merge** | `on: push: branches: ["main"]` | `if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH` |
| **Schedule setup** | In workflow YAML (`schedule:`) | In UI: **CI/CD → Schedules** |
| **Authentication** | Explicit (`GITHUB_TOKEN`) | Automatic (`CI_JOB_TOKEN`) |
Expand All @@ -161,8 +165,8 @@ recce-upload-prod:

Look for these indicators:

- **Workflow/Pipeline completes** without errors
- **Base session updated** in [Recce Cloud](https://cloud.reccehq.com)
- [x] **Workflow/Pipeline completes** without errors
- [x] **Base session updated** in [Recce Cloud](https://cloud.reccehq.com)

**GitHub:**

Expand Down
34 changes: 20 additions & 14 deletions docs/7-cicd/setup-ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,18 @@ Automatically validate your data changes in every pull request or merge request

Before setting up CI, ensure you have:

- **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/)
- **Repository connected** to Recce Cloud - [Git integration guide](../2-getting-started/start-free-with-cloud.md#git-integration)
- **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project
- **CD configured** - [Setup CD](./setup-cd.md) to establish baseline for comparisons
- [x] **Recce Cloud account** - [Start free trial](https://cloud.reccehq.com/)
- [x] **Repository connected** to Recce Cloud - [Connect Git Provider](../2-getting-started/start-free-with-cloud.md#2-connect-git-provider)
- [x] **dbt artifacts** - Know how to generate `manifest.json` and `catalog.json` from your dbt project
- [x] **CD configured** - [Setup CD](./setup-cd.md) to establish baseline for comparisons

## Setup

### GitHub Actions

Create `.github/workflows/ci-workflow.yml`:
Create `.github/workflows/pr-workflow.yml`:

```yaml linenums="1" hl_lines="41-42"
```yaml linenums="1"
name: Validate PR Changes

on:
Expand Down Expand Up @@ -66,10 +66,15 @@ jobs:
- name: Build current branch artifacts
run: |
dbt deps
# Optional: dbt build --target ci
dbt build --target ci
dbt docs generate --target ci
env:
DBT_ENV_SECRET_KEY: ${{ secrets.DBT_ENV_SECRET_KEY }}
SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
SNOWFLAKE_DATABASE: ${{ secrets.SNOWFLAKE_DATABASE }}
SNOWFLAKE_WAREHOUSE: ${{ secrets.SNOWFLAKE_WAREHOUSE }}
SNOWFLAKE_SCHEMA: "PR_${{ github.event.pull_request.number }}"

- name: Upload to Recce Cloud
run: |
Expand All @@ -81,9 +86,10 @@ jobs:

**Key points:**

- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) is passed explicitly for Recce Cloud authentication
- Creates a per-PR schema (`PR_123`, `PR_456`, etc.) using the dynamic `SNOWFLAKE_SCHEMA` environment variable to isolate each PR's data
- `dbt build` and `dbt docs generate` create the required artifacts (`manifest.json` and `catalog.json`)
- `recce-cloud upload` (without `--type`) auto-detects this is a PR session
- `dbt docs generate` creates the required `manifest.json` and `catalog.json`
- [`GITHUB_TOKEN`](https://docs.github.com/en/actions/concepts/security/github_token) authenticates with Recce Cloud

### GitLab CI/CD

Expand Down Expand Up @@ -136,7 +142,7 @@ recce-upload:

| Aspect | GitHub Actions | GitLab CI/CD |
| -------------------- | ----------------------------------- | -------------------------------------------------- |
| **Config file** | `.github/workflows/ci-workflow.yml` | `.gitlab-ci.yml` |
| **Config file** | `.github/workflows/pr-workflow.yml` | `.gitlab-ci.yml` |
| **Trigger** | `on: pull_request:` | `if: $CI_PIPELINE_SOURCE == "merge_request_event"` |
| **Authentication** | Explicit (`GITHUB_TOKEN`) | Automatic (`CI_JOB_TOKEN`) |
| **Session type** | Auto-detected from PR context | Auto-detected from MR context |
Expand All @@ -162,9 +168,9 @@ recce-upload:

Look for these indicators:

- **Workflow/Pipeline completes** without errors
- **PR/MR session created** in [Recce Cloud](https://cloud.reccehq.com)
- **Session URL** appears in workflow/pipeline output
- [x] **Workflow/Pipeline completes** without errors
- [x] **PR/MR session created** in [Recce Cloud](https://cloud.reccehq.com)
- [x] **Session URL** appears in workflow/pipeline output

**GitHub:**

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ markdown_extensions:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format
- pymdownx.tasklist:
custom_checkbox: true
- pymdownx.tabbed:
alternate_style: true

Expand Down