diff --git a/docs/2-getting-started/oss-vs-cloud.md b/docs/2-getting-started/oss-vs-cloud.md index bc07f91..b88f5ba 100644 --- a/docs/2-getting-started/oss-vs-cloud.md +++ b/docs/2-getting-started/oss-vs-cloud.md @@ -1,10 +1,10 @@ --- -title: Choose Your Approach +title: Choose best for you --- -# Choose Your Approach +# Choose what's best for you -Recce offers two ways to validate your data changes. **We recommend starting with Recce Cloud** for the easiest setup and team collaboration. +Recce offers two ways to validate your data changes: Open Source and Recce Cloud. **We recommend starting with Recce Cloud** for the easiest setup and team collaboration. ## Quick Comparison diff --git a/docs/2-getting-started/start-free-with-cloud.md b/docs/2-getting-started/start-free-with-cloud.md index cefeb57..dc2152e 100644 --- a/docs/2-getting-started/start-free-with-cloud.md +++ b/docs/2-getting-started/start-free-with-cloud.md @@ -1,92 +1,77 @@ --- -title: Start Free with Cloud +title: Start with Recce Cloud --- -# Start Free with Recce Cloud +# Start with Recce Cloud -**Launch Recce in under 2 minutes**. Each following feature provides additional value. The progressive features help you -get more value from Recce over time. +Validate data changes and collaborate with your team with automation. -πŸ‘‰ **[Start Free β†’](https://cloud.reccehq.com){target="_blank"}** +πŸ‘‰ **[Start with Recce Cloud](https://cloud.reccehq.com){target="_blank"}** -## Model Changes and Impact Analysis +Setup steps: -Recce shows what changed between **base** and **current** environments and helps assess potential impact. The most -common case is comparing your development branch against your production or main branch to see what your changes will -impact. +1. [Git Platform Integration](#git-integration) +2. [Data Warehouse Diffing](#data-diffing) +3. [CI/CD Automation](#cicd-automation) -You can: +Fall back to [manual](#manual) if you unable to finish the setup. -- Explore with the pre-loaded Jaffle Shop data -- Upload your metadata (see below) -- **Skip manual upload go directly to [CI/CD automation](#cicd-automation)** - +## Prerequisites +1. Admin access for git platform +2. Data warehouse credentials with read access +3. -### Upload Metadata -- Web interface: Click "Update" on the session you want to update in Recce Cloud. - 1. Click "Update" in base session to upload baseline metadata - 2. Click "Update" in current session to upload comparison metadata - 3. Click "Launch" to compare current against base -- CLI command: - -``` -recce upload-session --session-id -``` - -Find your session ID in Recce Cloud web interface when clicking "Update" on any session. - -### Required Files - -Recce needs `manifest.json` and `catalog.json` from both **base** and **current** environments for comparison. +## Git Platform Integration {#git-integration} -#### Base Metadata +Connect your GitHub or GitLab repository to see all PRs/MRs in one place and validate changes before they hit +production. -Production environment is commonly used as the baseline, but any environment can serve as the base. +### Setup Requirements -Choose one method: +| GitHub | GitLab | +|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| +| β€’ GitHub repository with dbt project
β€’ Repository admin access for initial setup
β€’ Active PRs with model changes | β€’ GitLab project with dbt project
β€’ Project maintainer or owner access for initial setup
β€’ Active merge requests with model changes | -**Method 1: Generate locally** +!!!Note + You'll need administrative access to the Organization/Group you want to connect. Please ensure you have the necessary + permissions for **App installations** (GitHub) or **Providing a Personal Access Token** (GitLab). -``` -dbt docs generate --target-path target-base --target -``` +### Connection -**Method 2: dbt Cloud**
-Deploy β†’ Jobs β†’ Production job β†’ Recent run β†’ Download artifacts +Connect your repository to track pull requests/merge requests and validate changes. -**Method 3: dbt Docs server**
-Download the artifacts directly from dbt docs server: +!!!Note + Keep **Connection setup** note the same as before if there was one specific to this section. -- `/manifest.json` -- `/catalog.json` +**Connection setup:** -#### Current Metadata +| GitHub | GitLab | +|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 1. Navigate to settings
2. Connect GitHub repository
3. Authorize Recce access
4. Select repository | 1. Navigate to settings
2. Connect GitLab by providing a Personal Access Token ([see our directions here](../7-cicd/gitlab-pat-guide.md))
3. Connect a project by adding a GitLab Project or URL to a Recce Project | -Use development environment or PR branch as current to compare against the base. -Choose one method: +### How to Use PR/MR Tracking -**Method 1: Generate locally** +Once connected, Recce displays all open and draft PRs/MRs in your Recce Cloud project. -``` -dbt docs generate --target -``` +### PR/MR Validation Workflow -**Method 2: dbt Cloud**
-Deploy β†’ Jobs β†’ CI job β†’ Recent run β†’ Download artifacts +- View open/draft PRs/MRs in dashboard +- Select PR/MR to validate +- Upload PR/MR metadata (until CI/CD is configured) +- Launch Recce to analyze changes ## Data Warehouse Diffing {#data-diffing} -Go beyond metadata to see how changes affect your actual data. Configure your data warehouse connection to compare query +Do data diffing to see how changes affect your actual data. Configure your data warehouse connection to compare query results and catch issues before they impact production. ### Setup Requirements - Data warehouse credentials with read access - Network connectivity to your warehouse -- Base and current environments configured in previous step ### Supported Warehouses @@ -100,66 +85,13 @@ Configure connection to your data warehouse to enable query result comparisons. **Quick setup:** -1. In your [project home](https://cloud.datarecce.io/), click the gear icon next to **Warehouse Connection** +1. In your [project home](https://cloud.reccehq.com/), click the gear icon next to **Warehouse Connection** 2. Create a new connection or select an existing one from the dropdown 3. Your connection is linked and ready to use For detailed connection settings, see [Connect to Warehouse](../5-data-diffing/connect-to-warehouse.md). Connection -credentials are encrypted and secure - see our [security practices](https://reccehq.com/security/). - - - -### How to Use Data Diffing - -Recce supports several data diffing methods. See Data Diffing sections for details: - -- [Row Count Diff](/5-data-diffing/row-count-diff) -- [Profile Diff](/5-data-diffing/profile-diff/) -- [Value Diff](/5-data-diffing/value-diff/) -- [Top-K Diff](/5-data-diffing/topK-diff/) -- [Histogram Diff](/5-data-diffing/histogram-diff/) -- [Query](/5-data-diffing/query/) +credentials are encrypted and secure, see our [security practices](https://reccehq.com/security/). -## Cloud-based Git Platform Integration {#git-integration} - -Connect your GitHub or GitLab repository to see all PRs/MRs in one place and validate changes before they hit -production. - -### Setup Requirements - -| GitHub | GitLab | -|------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| -| β€’ GitHub repository with dbt project
β€’ Repository admin access for initial setup
β€’ Active PRs with model changes | β€’ GitLab project with dbt project
β€’ Project maintainer or owner access for initial setup
β€’ Active merge requests with model changes | - -!!!Note - You'll need administrative access to the Organization/Group you want to connect. Please ensure you have the necessary - permissions for **App installations** (GitHub) or **Providing a Personal Access Token** (GitLab). - -### Connection - -Connect your repository to track pull requests/merge requests and validate changes. - -!!!Note - Keep **Connection setup** note the same as before if there was one specific to this section. - -**Connection setup:** - -| GitHub | GitLab | -|--------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 1. Navigate to settings
2. Connect GitHub repository
3. Authorize Recce access
4. Select repository | 1. Navigate to settings
2. Connect GitLab by providing a Personal Access Token ([see our directions here](../7-cicd/gitlab-pat-guide.md))
3. Connect a project by adding a GitLab Project or URL to a Recce Project | - - - -### How to Use PR/MR Tracking - -Once connected, Recce displays all open and draft PRs/MRs in your dashboard. - -### PR/MR Validation Workflow - -- View open/draft PRs/MRs in dashboard -- Select PR/MR to validate -- Upload PR/MR metadata (until CI/CD is configured) -- Launch Recce to analyze changes ## CI/CD Automation {#cicd-automation} @@ -189,5 +121,62 @@ See the CI/CD sections for complete setup guides: - Validation results directly in PR +## Manual Upload Metadata {#manual} + +Recce shows what changed between base and current environments and helps assess potential impact. The most common case is comparing your development branch against your production or main branch to see what your changes will impact. + +If you don’t use the GitHub/GitLab or havn’t setup CI/CD yet, you can manual upload + +- Web interface: Click "Update" on the session you want to update in Recce Cloud. + 1. Click "Update" in base session to upload baseline metadata + 2. Click "Update" in current session to upload comparison metadata + 3. Click "Launch" to compare current against base +- CLI command: + +``` +recce upload-session --session-id +``` + +Find your session ID in Recce Cloud web interface when clicking "Update" on any session. + +### Required Files + +Recce needs `manifest.json` and `catalog.json` from both **base** and **current** environments for comparison. + +#### Base Metadata + +Production environment is commonly used as the baseline, but any environment can serve as the base. + +Choose one method: + +**Method 1: Generate locally** + +``` +dbt docs generate --target-path target-base --target +``` + +**Method 2: dbt Cloud**
+Deploy β†’ Jobs β†’ Production job β†’ Recent run β†’ Download artifacts + +**Method 3: dbt Docs server**
+Download the artifacts directly from dbt docs server: + +- `/manifest.json` +- `/catalog.json` + +#### Current Metadata + +Use development environment or PR branch as current to compare against the base. + +Choose one method: + +**Method 1: Generate locally** + +``` +dbt docs generate --target +``` + +**Method 2: dbt Cloud**
+Deploy β†’ Jobs β†’ CI job β†’ Recent run β†’ Download artifacts diff --git a/docs/7-cicd/ci-cd-getting-started.md b/docs/7-cicd/ci-cd-getting-started.md index 395a81c..1798cad 100644 --- a/docs/7-cicd/ci-cd-getting-started.md +++ b/docs/7-cicd/ci-cd-getting-started.md @@ -10,82 +10,78 @@ Automate data validation in your development workflow. Catch data issues before Set up automated workflows that: -- **Maintain current baselines** - Auto-update comparison baselines on every merge to main -- **Validate every PR/MR** - Run data validation checks automatically when changes are proposed +- **Save time on reviews** - Eliminate manual validation steps for every change +- **Run data validations on every pull request/merge request** - Run data validation checks automatically when changes are proposed - **Prevent regressions** - Catch data quality issues before they reach production -- **Save team time** - Eliminate manual validation steps for every change !!!note - CI/CD automation requires Recce Cloud Team plan. A free trial is available. + CI/CD automation requires a Cloud Plan. Get started for free [here](https://cloud.reccehq.com/). -## Understanding CI vs CD +## What is CI/CD? -Recce uses both continuous integration and continuous delivery to automate data validation: +Recce uses both continuous integration (CI) and continuous delivery (CD) to automate data validation: **Continuous Integration (CI)** -- **When**: Runs on every PR/MR update -- **Purpose**: Validates proposed changes against baseline +- **When**: Runs when you open a new or update a Pull Request/Merge Request +- **Purpose**: Validates proposed changes against baseline (typically this mean production) - **Benefit**: Catches issues before merge, with results in your PR/MR **Continuous Delivery (CD)** - **When**: Runs after merge to main branch -- **Purpose**: Updates your baseline Recce session with latest production state +- **Purpose**: Updates baseline artifacts Recce uses to with latest production state - **Benefit**: Ensures future comparisons use current baseline -## Choose your platform +## What does look like with Recce? -Recce integrates with both GitHub Actions and GitLab CI/CD. - -Select your Git platform to get started: +Both CI and CD workflows follow the same pattern: -### GitHub -If your dbt project uses GitHub: +1. **Trigger event** (merge to main, or PR/MR opened/updated) +2. **Generate dbt artifacts** (`dbt docs generate` or external source) +3. **Upload to Recce Cloud** (automatic via workflow action) +4. **Validation results** appear in Recce dashboard and PR/MR -1. [Setup CI](./github/setup-ci.md) - Auto-validate changes in every PR -2. [Setup CD](./github/setup-cd.md) - Auto-update baseline on merge to main +
+ ![Recce CI/CD architecture](../assets/images/7-cicd/ci-cd.png){: .shadow} +
Automated validation workflow for pull requests
+
-### GitLab -If your dbt project uses GitLab: +## Getting Started with your CI/CD -2. [Setup CI](./gitlab/setup-ci.md) - Auto-validate changes in every MR -1. [Setup CD](./gitlab/setup-cd.md) - Auto-update baseline on merge to main -3. [GitLab Personal Access Token Guide](./gitlab/gitlab-pat-guide.md) - Required for GitLab integration +Recce currently integrates with both GitHub Actions and GitLab CI/CD. If you use another CI/CD product and interested in Recce, [let us know](https://cal.com/team/recce/chat). ## Prerequisites Before setting up, ensure you have: -- **Recce Cloud account** with Team plan or free trial +- **Recce Cloud account** You can signup and start your free trial [here](https://cloud.reccehq.com/) - **Repository connected** to Recce Cloud ([setup guide](../2-getting-started/start-free-with-cloud.md#git-integration)) -- **dbt artifacts** (`manifest.json` and `catalog.json`) from your project +- **dbt artifacts generated** (`manifest.json` and `catalog.json`) from your project -## Architecture overview +### GitHub +If your dbt project uses GitHub: -Both CI and CD workflows follow the same pattern: +1. [Setup CD](./github/setup-cd.md) - Auto-update baseline on merge to main +2. [Setup CI](./github/setup-ci.md) - Auto-validate changes in every PR -1. **Trigger event** (merge to main, or PR/MR opened/updated) -2. **Generate dbt artifacts** (`dbt docs generate` or external source) -3. **Upload to Recce Cloud** (automatic via workflow action) -4. **Validation results** appear in Recce dashboard and PR/MR +### GitLab +If your dbt project uses GitLab: -
- ![Recce CI/CD architecture](../assets/images/7-cicd/ci-cd.png){: .shadow} -
Automated validation workflow for pull requests
-
+1. [Setup CD](./gitlab/setup-cd.md) - Auto-update baseline on merge to main +2. [Setup CI](./gitlab/setup-ci.md) - Auto-validate changes in every MR +3. [GitLab Personal Access Token Guide](./gitlab/gitlab-pat-guide.md) - Required for GitLab integration ## Next steps -1. Choose your platform (GitHub or GitLab) -2. Start with CD setup to establish baseline updates -3. Add CI setup to enable PR/MR validation -4. Review [best practices](./best-practices-prep-env.md) for environment preparation +1. Start with relevant CD setup ([Gitlab](./gitlab/setup-cd.md) or [Github](./github/setup-cd.md)) to establish automatic baseline (production artifacts) updates. +2. Configure CI setup ([Gitlab](./gitlab/setup-ci.md) or [Github](./github/setup-ci.md)) to enable PR/MR validation +3. Review [best practices](./best-practices-prep-env.md) for environment preparation ## Related workflows After setting up CI/CD automation, explore these workflow guides: -- [Development workflow](./scenario-dev.md) - Validate changes during development -- [PR/MR review workflow](./scenario-pr-review.md) - Collaborate on validation results -- [Preset checks](./preset-checks.md) - Configure automatic validation checks +- [Development workflow](./scenario-dev.md) - How to validate data impact during development (pre-PR/MR) +- [PR/MR review workflow](./scenario-pr-review.md) - How to collaborate with teammates using Recce in PRs/MRs +- [Preset checks](./preset-checks.md) - How to configure automatic validation checks