From 5167c9f6a35347c13468bf2bbb660b1ddfc956d9 Mon Sep 17 00:00:00 2001 From: Rotem Tamir Date: Thu, 10 Aug 2023 11:45:09 +0300 Subject: [PATCH 1/5] doc/md/cloud: setting up ci with atlas cloud guide --- doc/md/cloud/setup-ci.mdx | 322 ++++++++++++++++++++++++++++++++++++++ doc/website/sidebars.js | 1 + 2 files changed, 323 insertions(+) create mode 100644 doc/md/cloud/setup-ci.mdx diff --git a/doc/md/cloud/setup-ci.mdx b/doc/md/cloud/setup-ci.mdx new file mode 100644 index 00000000000..05cfb4da7cb --- /dev/null +++ b/doc/md/cloud/setup-ci.mdx @@ -0,0 +1,322 @@ +--- +id: setup-ci +title: Setting up CI for your Database with Atlas Cloud +--- + +As your application evolves, so does your database schema. If you're not careful +with schema changes (migrations) you can end up introducing all sorts of issues +that are painful and expensive to fix. + +To mitigate the risk of deploying dangerous changes to database schemas, many teams +apply CI/CD practices to their database. This means that every change to the database +schema is automatically reviewed and tested before it is deployed to production. + +In this guide, we'll show you how to set up CI for your database using Atlas Cloud. + +## Push or Pull Workflows + +There are two ways to set up CI for your database with Atlas Cloud: + +* **Pull (GitHub App)** - You install the Atlas Cloud GitHub App on your GitHub + repository and Atlas Cloud will automatically pull changes from your GitHub + repository and run tests on them. + + This is the fastest way to set Atlas Cloud up, but it does require that you + have the necessary permissions to install the GitHub App on your repository. + +* **Push (GitHub Action)** - You install the [`ariga/atlas-action`](https://github.com/ariga/atlas-action) + GitHub Action on your GitHub repository and configure it to run on any pull request + that modifies your database schema. + + This is a bit more work to set up, but it doesn't require you to grant Atlas + Cloud any permissions on your repository. + +## Prerequisites + +1. A GitHub repository containing an Atlas migration directory. If you don't have one handy + and would like to experiment with this workflow you can use the + [`ariga/atlas-template`](https://github.com/ariga/atlas-template) template repo + to create one for you. +2. An [Atlas Cloud](https://atlasgo.cloud) account. If you don't have one, you can + [sign up for free](https://auth.atlasgo.cloud/signup). + +## Pull (GitHub App) Workflow + +### Step 1: Install the Atlas Cloud GitHub App + +After logging in to Atlas Cloud, create a new migration directory by clicking on +"Migrations Directories" in the sidebar and then clicking on the "Add Directory" +button. + +![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-1.png) + +Click "Connect with GitHub" to initiate the GitHub App installation flow. + +![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-2.png) + +### Step 2: Connect Your Repository + +Once you've installed the GitHub App, you need to grant it access to +the relevant repositories. Click on the "Organization" field and choose +"Add Organization" from the dropdown: + +![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-3.png) + +A popup requesting you to grant permission to specific repositories will appear. +Select the repositories you want to grant access to and click "Install and Authorize". + +![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-4.png) + +### Step 3: Configure the Migration Directory + +Fill the rest of the form with the relevant configuration details: +* **Directory Path** - The path to the directory containing your migrations relative to your repository root. +* **Directory Name** - An identifier for the directory. This will be used to identify the directory in the Atlas Cloud and APIs. +* **Migration Engine** - The format of the migrations in the directory. Atlas works best with migration directories in its + format, but also supports `golang-migrate`, Flyway and Liquibase. +* **Database** - the database engine for the target database. + +![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-5.png) + +Click "Connect Directory" to finish the process. + +### Step 4: Test the Integration + +After connecting your directory, let's create a pull request with a destructive +change to make sure that Atlas Cloud catches it. + +Create a new branch: + +``` +git checkout -b test-branch +``` + +Create a new migration file: + +``` +atlas migrate new --edit destructive +``` + +Add the following SQL to the migration file and save: + +```sql +DROP TABLE posts; +``` + +Commit the changes and push them to GitHub: + +``` +git add . +git commit -m "add destructive migration" +git push origin test-branch +``` + +Create a pull request from the `test-branch` branch to your main branch. + +After a few seconds, Atlas Cloud will pick up the changes and run CI against +them. Shortly after, you will see a comment on your pull request with the +results of the CI run: + +![](https://atlasgo.io/uploads/cloud/ci-guide/app-ci.png) + +Amazing! Atlas Cloud caught the destructive change and prevented it from being +accidentally merged to your main branch. + +To see a more detailed report of the CI run, click on the "Full Report on Atlas Cloud" +link in the comment: + +![](https://atlasgo.io/uploads/cloud/ci-guide/app-cloud-report.png) + + +## Push (GitHub Action) Workflow + +### Step 1: Create a Bot Token for Atlas Cloud + +In order to report the results of your CI runs to Atlas Cloud, you will need to +create a bot token for Atlas Cloud to use. Follow [these instructions](/bot.mdx) +to create a token and save it somewhere safe. + +### Step 2: Install the Atlas GitHub CLI Extension + +To streamline the process of configuring the GitHub Action, we've created a +GitHub CLI extension that will do most of the work for you: + +1. Make sure you have the GitHub CLI installed. + + ```shell + brew install gh + ``` + See [here](https://cli.github.com/manual/installation) for more installation options. + +2. Install the Atlas GitHub CLI extension: + + ```shell + gh extension install ariga/atlas + ``` + +### Step 3: Configure the GitHub Action + +1. Make sure you have the necessary permissions to configure your action: + + ```shell + gh auth refresh -s write:packages,workflow + ``` + +2. Make sure your current working directory is the root of your repository. + + ```shell + cd path/to/my/repo + ``` + +3. Run the `init-ci` command to configure the GitHub Action: + + ```shell + gh atlas init-action --token + ``` + Atlas will scan your repository (locally) for directories containing Atlas migrations + and ask you which one you would like to use for CI. Select the desired directory and press "Enter": + ```text + Use the arrow keys to navigate: ↓ ↑ → ← + ? choose migration directory: + ▸ migrations + ``` + + Atlas will then which database driver this directory contains migrations for. Select the + desired driver and press "Enter": + + ```text + Use the arrow keys to navigate: ↓ ↑ → ← + ? choose driver: + ▸ mysql + postgres + mariadb + sqlite + ``` + + Next, the GitHub extension will save your bot token to a GitHub secret and create a + pull request with the necessary configuration for the GitHub Action. + + ![](https://atlasgo.io/uploads/cloud/ci-guide/gh-ext-pr-main.png) + + The PR contains a GitHub Actions workflow similar to this: +```yaml +name: Atlas +on: + push: + branches: + - master + pull_request: + paths: + - 'migrations/*' +# Permissions to write comments on the pull request. +permissions: + contents: read + pull-requests: write +jobs: + lint: + services: + # Spin up a mysql:8 container to be used as the dev-database for analysis. + mysql: + image: mysql:8 + env: + MYSQL_DATABASE: dev + MYSQL_ROOT_PASSWORD: pass + ports: + - 3306:3306 + options: >- + --health-cmd "mysqladmin ping -ppass" + --health-interval 10s + --health-start-period 10s + --health-timeout 5s + --health-retries 10 + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + - uses: ariga/atlas-action@v0 + with: + dir: 'migrations' + dev-url: "mysql://root:pass@localhost:3306/dev" + cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN_PfddoG }} + sync: + needs: lint + if: github.ref == 'refs/heads/master' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ariga/atlas-sync-action@v0 + with: + dir: 'migrations' + driver: mysql + cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN_PfddoG }} +``` + + After reviewing the changes, merge the pull request to enable the GitHub Action. + +### Step 4: Test the Action + +After merging the pull request, the GitHub Action will run `atlas migrate lint` on every pull request +and sync the migrations to Atlas Cloud on every push to `master`. + +1. To test this setup, create a new branch for making some changes to your database schema: + + ```shell + git checkout -b test-ci + ``` + +2. Create a new migration in interactive mode: + + ```shell + atlas migrate new --edit dummy + ``` + +3. Type some broken SQL into the migration file and save it: + + ```sql + CREATE users ( + col varchar(255) + ); + ``` + + (Notice that we're missing the `TABLE` keyword in the `CREATE TABLE` statement.) + +4. Commit the changes and push them to GitHub: + + ```shell + git add migrations + git commit -m "migrations: add dummy migration" + git push origin test-ci + ``` + + Our changes are pushed to GitHub: + ``` + remote: Resolving deltas: 100% (3/3), done. + remote: + remote: Create a pull request for 'test-ci' on GitHub by visiting: + remote: https://github.com/rotemtam/gh-init-demo/pull/new/test-ci + remote: + To github.com:rotemtam/gh-init-demo.git + * [new branch] test-ci -> test-ci + ``` + +5. Open a pull request for the `test-ci` branch and wait for the GitHub Action to run. + + ![](https://atlasgo.io/uploads/cloud/ci-guide/open-pr.png) + +6. Notice that after a few moments, a comment appears on the pull request with the results of the linting run: + + ![](https://atlasgo.io/uploads/cloud/ci-guide/pr-comment.png) + + The linting run failed because of the syntax error we introduced in the migration file. + +7. Click on "Full Report on Atlas Cloud" to see the full report on Atlas Cloud: + + ![](https://atlasgo.io/uploads/cloud/ci-guide/fixed-cloud-report-2.png) + +## Summary + +In this guide, we've shown how to configure Atlas Cloud to apply continuous integration +for our database schema changes. With this setup, whenever a developer proposes a change +to the database schema, Atlas Cloud will verify the safety of the change using various checks +and report back the results. diff --git a/doc/website/sidebars.js b/doc/website/sidebars.js index c9bd4c9270c..f39277b1a90 100644 --- a/doc/website/sidebars.js +++ b/doc/website/sidebars.js @@ -90,6 +90,7 @@ module.exports = { items: [ {type: 'doc', id: 'cloud/getting-started', label: 'Getting Started'}, {type: 'doc', id: 'cloud/directories', label: 'Connected Directories'}, + {type: 'doc', id: 'cloud/setup-ci', label: 'CI Setup'}, {type: 'doc', id: 'cloud/bots', label: 'Creating Bots'}, {type: 'doc', id: 'cloud/deployment', label: 'Deployments'}, ], From 4ef07f90201ae6c82e9b6c91bab4c0948973d577 Mon Sep 17 00:00:00 2001 From: Rotem Tamir Date: Thu, 10 Aug 2023 12:12:18 +0300 Subject: [PATCH 2/5] summaries --- doc/md/cloud/setup-ci.mdx | 155 +++++++++++++++++++++++++------------- 1 file changed, 102 insertions(+), 53 deletions(-) diff --git a/doc/md/cloud/setup-ci.mdx b/doc/md/cloud/setup-ci.mdx index 05cfb4da7cb..d4932a0de18 100644 --- a/doc/md/cloud/setup-ci.mdx +++ b/doc/md/cloud/setup-ci.mdx @@ -48,11 +48,19 @@ After logging in to Atlas Cloud, create a new migration directory by clicking on "Migrations Directories" in the sidebar and then clicking on the "Add Directory" button. +
Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-1.png) +
+ Click "Connect with GitHub" to initiate the GitHub App installation flow. +
+Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-2.png) +
### Step 2: Connect Your Repository @@ -60,12 +68,20 @@ Once you've installed the GitHub App, you need to grant it access to the relevant repositories. Click on the "Organization" field and choose "Add Organization" from the dropdown: +
+Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-3.png) +
A popup requesting you to grant permission to specific repositories will appear. Select the repositories you want to grant access to and click "Install and Authorize". +
+Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-4.png) +
### Step 3: Configure the Migration Directory @@ -76,7 +92,11 @@ Fill the rest of the form with the relevant configuration details: format, but also supports `golang-migrate`, Flyway and Liquibase. * **Database** - the database engine for the target database. +
+Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/create-dir-5.png) +
Click "Connect Directory" to finish the process. @@ -117,7 +137,12 @@ After a few seconds, Atlas Cloud will pick up the changes and run CI against them. Shortly after, you will see a comment on your pull request with the results of the CI run: +
+Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/app-ci.png) +
+ Amazing! Atlas Cloud caught the destructive change and prevented it from being accidentally merged to your main branch. @@ -125,8 +150,11 @@ accidentally merged to your main branch. To see a more detailed report of the CI run, click on the "Full Report on Atlas Cloud" link in the comment: -![](https://atlasgo.io/uploads/cloud/ci-guide/app-cloud-report.png) +
+Screenshot Example +![](https://atlasgo.io/uploads/cloud/ci-guide/app-cloud-report.png) +
## Push (GitHub Action) Workflow @@ -196,61 +224,71 @@ GitHub CLI extension that will do most of the work for you: Next, the GitHub extension will save your bot token to a GitHub secret and create a pull request with the necessary configuration for the GitHub Action. +
+ Screenshot Example + ![](https://atlasgo.io/uploads/cloud/ci-guide/gh-ext-pr-main.png) +
The PR contains a GitHub Actions workflow similar to this: -```yaml -name: Atlas -on: - push: - branches: - - master - pull_request: - paths: - - 'migrations/*' -# Permissions to write comments on the pull request. -permissions: - contents: read - pull-requests: write -jobs: - lint: - services: - # Spin up a mysql:8 container to be used as the dev-database for analysis. - mysql: - image: mysql:8 - env: - MYSQL_DATABASE: dev - MYSQL_ROOT_PASSWORD: pass - ports: - - 3306:3306 - options: >- - --health-cmd "mysqladmin ping -ppass" - --health-interval 10s - --health-start-period 10s - --health-timeout 5s - --health-retries 10 - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - with: - fetch-depth: 0 - - uses: ariga/atlas-action@v0 - with: - dir: 'migrations' - dev-url: "mysql://root:pass@localhost:3306/dev" - cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN_PfddoG }} - sync: - needs: lint - if: github.ref == 'refs/heads/master' - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v3 - - uses: ariga/atlas-sync-action@v0 - with: - dir: 'migrations' - driver: mysql - cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN_PfddoG }} -``` + +
+ + Code Example + + name: Atlas + on: + push: + branches: + - master + pull_request: + paths: + - 'migrations/*' + # Permissions to write comments on the pull request. + permissions: + contents: read + pull-requests: write + jobs: + lint: + services: + # Spin up a mysql:8 container to be used as the dev-database for analysis. + mysql: + image: mysql:8 + env: + MYSQL_DATABASE: dev + MYSQL_ROOT_PASSWORD: pass + ports: + - 3306:3306 + options: >- + --health-cmd "mysqladmin ping -ppass" + --health-interval 10s + --health-start-period 10s + --health-timeout 5s + --health-retries 10 + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 + - uses: ariga/atlas-action@v0 + with: + dir: 'migrations' + dev-url: "mysql://root:pass@localhost:3306/dev" + cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN_PfddoG }} + sync: + needs: lint + if: github.ref == 'refs/heads/master' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ariga/atlas-sync-action@v0 + with: + dir: 'migrations' + driver: mysql + cloud-token: ${{ secrets.ATLAS_CLOUD_TOKEN_PfddoG }} + +
+ After reviewing the changes, merge the pull request to enable the GitHub Action. @@ -301,18 +339,29 @@ and sync the migrations to Atlas Cloud on every push to `master`. ``` 5. Open a pull request for the `test-ci` branch and wait for the GitHub Action to run. +
+ Screenshot Example ![](https://atlasgo.io/uploads/cloud/ci-guide/open-pr.png) +
+ + 6. Notice that after a few moments, a comment appears on the pull request with the results of the linting run: +
+ Screenshot Example ![](https://atlasgo.io/uploads/cloud/ci-guide/pr-comment.png) +
The linting run failed because of the syntax error we introduced in the migration file. 7. Click on "Full Report on Atlas Cloud" to see the full report on Atlas Cloud: +
+ Screenshot Example ![](https://atlasgo.io/uploads/cloud/ci-guide/fixed-cloud-report-2.png) +
## Summary From 1190a7bedc665c858928e68c9669bc33e82d0906 Mon Sep 17 00:00:00 2001 From: Rotem Tamir Date: Thu, 10 Aug 2023 12:16:13 +0300 Subject: [PATCH 3/5] Update doc/md/cloud/setup-ci.mdx Co-authored-by: gedalyah-ariga <139685520+gedalyah-ariga@users.noreply.github.com> --- doc/md/cloud/setup-ci.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/md/cloud/setup-ci.mdx b/doc/md/cloud/setup-ci.mdx index d4932a0de18..488a2d25432 100644 --- a/doc/md/cloud/setup-ci.mdx +++ b/doc/md/cloud/setup-ci.mdx @@ -209,7 +209,7 @@ GitHub CLI extension that will do most of the work for you: ▸ migrations ``` - Atlas will then which database driver this directory contains migrations for. Select the + Atlas will then ask you which database driver this directory contains migrations for. Select the desired driver and press "Enter": ```text From 04e9b0036c2227607d08d9805803dee49335108f Mon Sep 17 00:00:00 2001 From: Rotem Tamir Date: Thu, 10 Aug 2023 12:23:50 +0300 Subject: [PATCH 4/5] Apply suggestions from code review Co-authored-by: gedalyah-ariga <139685520+gedalyah-ariga@users.noreply.github.com> --- doc/md/cloud/setup-ci.mdx | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/md/cloud/setup-ci.mdx b/doc/md/cloud/setup-ci.mdx index 488a2d25432..62315926d01 100644 --- a/doc/md/cloud/setup-ci.mdx +++ b/doc/md/cloud/setup-ci.mdx @@ -4,7 +4,7 @@ title: Setting up CI for your Database with Atlas Cloud --- As your application evolves, so does your database schema. If you're not careful -with schema changes (migrations) you can end up introducing all sorts of issues +with schema changes (migrations), you can end up introducing all sorts of issues that are painful and expensive to fix. To mitigate the risk of deploying dangerous changes to database schemas, many teams @@ -18,8 +18,8 @@ In this guide, we'll show you how to set up CI for your database using Atlas Clo There are two ways to set up CI for your database with Atlas Cloud: * **Pull (GitHub App)** - You install the Atlas Cloud GitHub App on your GitHub - repository and Atlas Cloud will automatically pull changes from your GitHub - repository and run tests on them. + repository. Atlas Cloud will automatically pull changes from your GitHub + repository, then run tests on them. This is the fastest way to set Atlas Cloud up, but it does require that you have the necessary permissions to install the GitHub App on your repository. @@ -28,13 +28,13 @@ There are two ways to set up CI for your database with Atlas Cloud: GitHub Action on your GitHub repository and configure it to run on any pull request that modifies your database schema. - This is a bit more work to set up, but it doesn't require you to grant Atlas + This is a bit more work to set up, but it does _not_ require you to grant Atlas Cloud any permissions on your repository. ## Prerequisites 1. A GitHub repository containing an Atlas migration directory. If you don't have one handy - and would like to experiment with this workflow you can use the + and would like to experiment with this workflow, you can use the [`ariga/atlas-template`](https://github.com/ariga/atlas-template) template repo to create one for you. 2. An [Atlas Cloud](https://atlasgo.cloud) account. If you don't have one, you can @@ -75,7 +75,7 @@ the relevant repositories. Click on the "Organization" field and choose A popup requesting you to grant permission to specific repositories will appear. -Select the repositories you want to grant access to and click "Install and Authorize". +Select the repositories you want to grant access to, then click "Install and Authorize".
Screenshot Example @@ -89,7 +89,7 @@ Fill the rest of the form with the relevant configuration details: * **Directory Path** - The path to the directory containing your migrations relative to your repository root. * **Directory Name** - An identifier for the directory. This will be used to identify the directory in the Atlas Cloud and APIs. * **Migration Engine** - The format of the migrations in the directory. Atlas works best with migration directories in its - format, but also supports `golang-migrate`, Flyway and Liquibase. + format, but also supports `golang-migrate`, Flyway, and Liquibase. * **Database** - the database engine for the target database.
@@ -184,19 +184,19 @@ GitHub CLI extension that will do most of the work for you: ### Step 3: Configure the GitHub Action -1. Make sure you have the necessary permissions to configure your action: +1. **Permissions** - Make sure you have the necessary permissions to configure your action: ```shell gh auth refresh -s write:packages,workflow ``` -2. Make sure your current working directory is the root of your repository. +2. **Go to repo root** - Make sure your current working directory is the root of your repository. ```shell cd path/to/my/repo ``` -3. Run the `init-ci` command to configure the GitHub Action: +3. **Run the extension** - Run the `init-ci` command to configure the GitHub Action: ```shell gh atlas init-action --token From c3d8bb9a580cc727e2fdc5a66022bacfaec54b02 Mon Sep 17 00:00:00 2001 From: Rotem Tamir Date: Thu, 10 Aug 2023 12:34:08 +0300 Subject: [PATCH 5/5] fix bot link --- doc/md/cloud/setup-ci.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/md/cloud/setup-ci.mdx b/doc/md/cloud/setup-ci.mdx index 62315926d01..1b80e369293 100644 --- a/doc/md/cloud/setup-ci.mdx +++ b/doc/md/cloud/setup-ci.mdx @@ -161,7 +161,7 @@ link in the comment: ### Step 1: Create a Bot Token for Atlas Cloud In order to report the results of your CI runs to Atlas Cloud, you will need to -create a bot token for Atlas Cloud to use. Follow [these instructions](/bot.mdx) +create a bot token for Atlas Cloud to use. Follow [these instructions](/cloud/bot.mdx) to create a token and save it somewhere safe. ### Step 2: Install the Atlas GitHub CLI Extension