Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/2-getting-started/oss-vs-cloud.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
---
title: Choose Your Approach
title: Choose best for you
---

# Choose Your Approach
# Choose what's best for you

Recce offers two ways to validate your data changes. **We recommend starting with Recce Cloud** for the easiest setup and team collaboration.
Recce offers two ways to validate your data changes: Open Source and Recce Cloud. **We recommend starting with Recce Cloud** for the easiest setup and team collaboration.

## Quick Comparison

Expand Down
207 changes: 98 additions & 109 deletions docs/2-getting-started/start-free-with-cloud.md
Original file line number Diff line number Diff line change
@@ -1,92 +1,77 @@
---
title: Start Free with Cloud
title: Start with Recce Cloud
---

# Start Free with Recce Cloud
# Start with Recce Cloud

**Launch Recce in under 2 minutes**. Each following feature provides additional value. The progressive features help you
get more value from Recce over time.
Validate data changes and collaborate with your team with automation.

👉 **[Start Free →](https://cloud.reccehq.com){target="_blank"}**
👉 **[Start with Recce Cloud](https://cloud.reccehq.com){target="_blank"}**

## Model Changes and Impact Analysis
Setup steps:

Recce shows what changed between **base** and **current** environments and helps assess potential impact. The most
common case is comparing your development branch against your production or main branch to see what your changes will
impact.
1. [Git Platform Integration](#git-integration)
2. [Data Warehouse Diffing](#data-diffing)
3. [CI/CD Automation](#cicd-automation)

You can:
Fall back to [manual](#manual) if you unable to finish the setup.

- Explore with the pre-loaded Jaffle Shop data
- Upload your metadata (see below)
- **Skip manual upload go directly to [CI/CD automation](#cicd-automation)**

<!-- insert a video -->
## Prerequisites
1. Admin access for git platform
2. Data warehouse credentials with read access
3.

### Upload Metadata

- Web interface: Click "Update" on the session you want to update in Recce Cloud.
1. Click "Update" in base session to upload baseline metadata
2. Click "Update" in current session to upload comparison metadata
3. Click "Launch" to compare current against base
- CLI command:

```
recce upload-session --session-id <your_session_id>
```

Find your session ID in Recce Cloud web interface when clicking "Update" on any session.

### Required Files

Recce needs `manifest.json` and `catalog.json` from both **base** and **current** environments for comparison.
## Git Platform Integration {#git-integration}

#### Base Metadata
Connect your GitHub or GitLab repository to see all PRs/MRs in one place and validate changes before they hit
production.

Production environment is commonly used as the baseline, but any environment can serve as the base.
### Setup Requirements

Choose one method:
| GitHub | GitLab |
|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| • GitHub repository with dbt project<br>• Repository admin access for initial setup<br>• Active PRs with model changes | • GitLab project with dbt project<br>• Project maintainer or owner access for initial setup<br>• Active merge requests with model changes |

**Method 1: Generate locally**
!!!Note
You'll need administrative access to the Organization/Group you want to connect. Please ensure you have the necessary
permissions for **App installations** (GitHub) or **Providing a Personal Access Token** (GitLab).

```
dbt docs generate --target-path target-base --target <your_prod_target>
```
### Connection

**Method 2: dbt Cloud**<br>
Deploy → Jobs → Production job → Recent run → Download artifacts
Connect your repository to track pull requests/merge requests and validate changes.

**Method 3: dbt Docs server**<br>
Download the artifacts directly from dbt docs server:
!!!Note
Keep **Connection setup** note the same as before if there was one specific to this section.

- `<dbt_docs_url>/manifest.json`
- `<dbt_docs_url>/catalog.json`
**Connection setup:**

#### Current Metadata
| GitHub | GitLab |
|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1. Navigate to settings<br>2. Connect GitHub repository<br>3. Authorize Recce access<br>4. Select repository | 1. Navigate to settings<br>2. Connect GitLab by providing a Personal Access Token ([see our directions here](../7-cicd/gitlab-pat-guide.md))<br>3. Connect a project by adding a GitLab Project or URL to a Recce Project |

Use development environment or PR branch as current to compare against the base.

Choose one method:
### How to Use PR/MR Tracking

**Method 1: Generate locally**
Once connected, Recce displays all open and draft PRs/MRs in your Recce Cloud project.

```
dbt docs generate --target <your_dev_target>
```
### PR/MR Validation Workflow

**Method 2: dbt Cloud**<br>
Deploy → Jobs → CI job → Recent run → Download artifacts
- View open/draft PRs/MRs in dashboard
- Select PR/MR to validate
- Upload PR/MR metadata (until CI/CD is configured)
- Launch Recce to analyze changes

## Data Warehouse Diffing {#data-diffing}

Go beyond metadata to see how changes affect your actual data. Configure your data warehouse connection to compare query
Do data diffing to see how changes affect your actual data. Configure your data warehouse connection to compare query
results and catch issues before they impact production.

### Setup Requirements

- Data warehouse credentials with read access
- Network connectivity to your warehouse
- Base and current environments configured in previous step

### Supported Warehouses

Expand All @@ -100,66 +85,13 @@ Configure connection to your data warehouse to enable query result comparisons.

**Quick setup:**

1. In your [project home](https://cloud.datarecce.io/), click the gear icon next to **Warehouse Connection**
1. In your [project home](https://cloud.reccehq.com/), click the gear icon next to **Warehouse Connection**
2. Create a new connection or select an existing one from the dropdown
3. Your connection is linked and ready to use

For detailed connection settings, see [Connect to Warehouse](../5-data-diffing/connect-to-warehouse.md). Connection
credentials are encrypted and secure - see our [security practices](https://reccehq.com/security/).

<!-- insert a video -->

### How to Use Data Diffing

Recce supports several data diffing methods. See Data Diffing sections for details:

- [Row Count Diff](/5-data-diffing/row-count-diff)
- [Profile Diff](/5-data-diffing/profile-diff/)
- [Value Diff](/5-data-diffing/value-diff/)
- [Top-K Diff](/5-data-diffing/topK-diff/)
- [Histogram Diff](/5-data-diffing/histogram-diff/)
- [Query](/5-data-diffing/query/)
credentials are encrypted and secure, see our [security practices](https://reccehq.com/security/).

## Cloud-based Git Platform Integration {#git-integration}

Connect your GitHub or GitLab repository to see all PRs/MRs in one place and validate changes before they hit
production.

### Setup Requirements

| GitHub | GitLab |
|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| • GitHub repository with dbt project<br>• Repository admin access for initial setup<br>• Active PRs with model changes | • GitLab project with dbt project<br>• Project maintainer or owner access for initial setup<br>• Active merge requests with model changes |

!!!Note
You'll need administrative access to the Organization/Group you want to connect. Please ensure you have the necessary
permissions for **App installations** (GitHub) or **Providing a Personal Access Token** (GitLab).

### Connection

Connect your repository to track pull requests/merge requests and validate changes.

!!!Note
Keep **Connection setup** note the same as before if there was one specific to this section.

**Connection setup:**

| GitHub | GitLab |
|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1. Navigate to settings<br>2. Connect GitHub repository<br>3. Authorize Recce access<br>4. Select repository | 1. Navigate to settings<br>2. Connect GitLab by providing a Personal Access Token ([see our directions here](../7-cicd/gitlab-pat-guide.md))<br>3. Connect a project by adding a GitLab Project or URL to a Recce Project |

<!-- insert a video -->

### How to Use PR/MR Tracking

Once connected, Recce displays all open and draft PRs/MRs in your dashboard.

### PR/MR Validation Workflow

- View open/draft PRs/MRs in dashboard
- Select PR/MR to validate
- Upload PR/MR metadata (until CI/CD is configured)
- Launch Recce to analyze changes

## CI/CD Automation {#cicd-automation}

Expand Down Expand Up @@ -189,5 +121,62 @@ See the CI/CD sections for complete setup guides:
- Validation results directly in PR


## Manual Upload Metadata {#manual}

Recce shows what changed between base and current environments and helps assess potential impact. The most common case is comparing your development branch against your production or main branch to see what your changes will impact.

If you don’t use the GitHub/GitLab or havn’t setup CI/CD yet, you can manual upload

- Web interface: Click "Update" on the session you want to update in Recce Cloud.
1. Click "Update" in base session to upload baseline metadata
2. Click "Update" in current session to upload comparison metadata
3. Click "Launch" to compare current against base
- CLI command:

```
recce upload-session --session-id <your_session_id>
```

Find your session ID in Recce Cloud web interface when clicking "Update" on any session.

### Required Files

Recce needs `manifest.json` and `catalog.json` from both **base** and **current** environments for comparison.

#### Base Metadata

Production environment is commonly used as the baseline, but any environment can serve as the base.

Choose one method:

**Method 1: Generate locally**

```
dbt docs generate --target-path target-base --target <your_prod_target>
```

**Method 2: dbt Cloud**<br>
Deploy → Jobs → Production job → Recent run → Download artifacts

**Method 3: dbt Docs server**<br>
Download the artifacts directly from dbt docs server:

- `<dbt_docs_url>/manifest.json`
- `<dbt_docs_url>/catalog.json`

#### Current Metadata

Use development environment or PR branch as current to compare against the base.

Choose one method:

**Method 1: Generate locally**

```
dbt docs generate --target <your_dev_target>
```

**Method 2: dbt Cloud**<br>
Deploy → Jobs → CI job → Recent run → Download artifacts


78 changes: 37 additions & 41 deletions docs/7-cicd/ci-cd-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,82 +10,78 @@ Automate data validation in your development workflow. Catch data issues before

Set up automated workflows that:

- **Maintain current baselines** - Auto-update comparison baselines on every merge to main
- **Validate every PR/MR** - Run data validation checks automatically when changes are proposed
- **Save time on reviews** - Eliminate manual validation steps for every change
- **Run data validations on every pull request/merge request** - Run data validation checks automatically when changes are proposed
- **Prevent regressions** - Catch data quality issues before they reach production
- **Save team time** - Eliminate manual validation steps for every change

!!!note
CI/CD automation requires Recce Cloud Team plan. A free trial is available.
CI/CD automation requires a Cloud Plan. Get started for free [here](https://cloud.reccehq.com/).

## Understanding CI vs CD
## What is CI/CD?

Recce uses both continuous integration and continuous delivery to automate data validation:
Recce uses both continuous integration (CI) and continuous delivery (CD) to automate data validation:

**Continuous Integration (CI)**

- **When**: Runs on every PR/MR update
- **Purpose**: Validates proposed changes against baseline
- **When**: Runs when you open a new or update a Pull Request/Merge Request
- **Purpose**: Validates proposed changes against baseline (typically this mean production)
- **Benefit**: Catches issues before merge, with results in your PR/MR

**Continuous Delivery (CD)**

- **When**: Runs after merge to main branch
- **Purpose**: Updates your baseline Recce session with latest production state
- **Purpose**: Updates baseline artifacts Recce uses to with latest production state
- **Benefit**: Ensures future comparisons use current baseline

## Choose your platform
## What does look like with Recce?

Recce integrates with both GitHub Actions and GitLab CI/CD.

Select your Git platform to get started:
Both CI and CD workflows follow the same pattern:

### GitHub
If your dbt project uses GitHub:
1. **Trigger event** (merge to main, or PR/MR opened/updated)
2. **Generate dbt artifacts** (`dbt docs generate` or external source)
3. **Upload to Recce Cloud** (automatic via workflow action)
4. **Validation results** appear in Recce dashboard and PR/MR

1. [Setup CI](./github/setup-ci.md) - Auto-validate changes in every PR
2. [Setup CD](./github/setup-cd.md) - Auto-update baseline on merge to main
<figure markdown>
![Recce CI/CD architecture](../assets/images/7-cicd/ci-cd.png){: .shadow}
<figcaption>Automated validation workflow for pull requests</figcaption>
</figure>

### GitLab
If your dbt project uses GitLab:
## Getting Started with your CI/CD

2. [Setup CI](./gitlab/setup-ci.md) - Auto-validate changes in every MR
1. [Setup CD](./gitlab/setup-cd.md) - Auto-update baseline on merge to main
3. [GitLab Personal Access Token Guide](./gitlab/gitlab-pat-guide.md) - Required for GitLab integration
Recce currently integrates with both GitHub Actions and GitLab CI/CD. If you use another CI/CD product and interested in Recce, [let us know](https://cal.com/team/recce/chat).

## Prerequisites

Before setting up, ensure you have:

- **Recce Cloud account** with Team plan or free trial
- **Recce Cloud account** You can signup and start your free trial [here](https://cloud.reccehq.com/)
- **Repository connected** to Recce Cloud ([setup guide](../2-getting-started/start-free-with-cloud.md#git-integration))
- **dbt artifacts** (`manifest.json` and `catalog.json`) from your project
- **dbt artifacts generated** (`manifest.json` and `catalog.json`) from your project

## Architecture overview
### GitHub
If your dbt project uses GitHub:

Both CI and CD workflows follow the same pattern:
1. [Setup CD](./github/setup-cd.md) - Auto-update baseline on merge to main
2. [Setup CI](./github/setup-ci.md) - Auto-validate changes in every PR

1. **Trigger event** (merge to main, or PR/MR opened/updated)
2. **Generate dbt artifacts** (`dbt docs generate` or external source)
3. **Upload to Recce Cloud** (automatic via workflow action)
4. **Validation results** appear in Recce dashboard and PR/MR
### GitLab
If your dbt project uses GitLab:

<figure markdown>
![Recce CI/CD architecture](../assets/images/7-cicd/ci-cd.png){: .shadow}
<figcaption>Automated validation workflow for pull requests</figcaption>
</figure>
1. [Setup CD](./gitlab/setup-cd.md) - Auto-update baseline on merge to main
2. [Setup CI](./gitlab/setup-ci.md) - Auto-validate changes in every MR
3. [GitLab Personal Access Token Guide](./gitlab/gitlab-pat-guide.md) - Required for GitLab integration

## Next steps

1. Choose your platform (GitHub or GitLab)
2. Start with CD setup to establish baseline updates
3. Add CI setup to enable PR/MR validation
4. Review [best practices](./best-practices-prep-env.md) for environment preparation
1. Start with relevant CD setup ([Gitlab](./gitlab/setup-cd.md) or [Github](./github/setup-cd.md)) to establish automatic baseline (production artifacts) updates.
2. Configure CI setup ([Gitlab](./gitlab/setup-ci.md) or [Github](./github/setup-ci.md)) to enable PR/MR validation
3. Review [best practices](./best-practices-prep-env.md) for environment preparation

## Related workflows

After setting up CI/CD automation, explore these workflow guides:

- [Development workflow](./scenario-dev.md) - Validate changes during development
- [PR/MR review workflow](./scenario-pr-review.md) - Collaborate on validation results
- [Preset checks](./preset-checks.md) - Configure automatic validation checks
- [Development workflow](./scenario-dev.md) - How to validate data impact during development (pre-PR/MR)
- [PR/MR review workflow](./scenario-pr-review.md) - How to collaborate with teammates using Recce in PRs/MRs
- [Preset checks](./preset-checks.md) - How to configure automatic validation checks