From 0d2aad296dd197327da50635365adec40da568eb Mon Sep 17 00:00:00 2001 From: Edvin Norling Date: Mon, 11 Oct 2021 13:28:23 +0200 Subject: [PATCH 1/4] How to setup terraform CI in github --- docs/xks/operator-guide/github.md | 93 +++++++++++++++++++++++++++++++ sidebars.js | 3 +- 2 files changed, 95 insertions(+), 1 deletion(-) create mode 100644 docs/xks/operator-guide/github.md diff --git a/docs/xks/operator-guide/github.md b/docs/xks/operator-guide/github.md new file mode 100644 index 00000000000..a3e02305c98 --- /dev/null +++ b/docs/xks/operator-guide/github.md @@ -0,0 +1,93 @@ +--- +id: github +title: XKF on Github +--- + +## Terraform + +In GitHub there is no service connection like there is in Azure DevOps. +Instead we utilize a Azure SP to talk to Azure when running our CI pipelines. + +Create a SP with the access that terraform requires to perform all the tasks you want. +You can read more about SP creation in our [operator guide](operator-guide.md) + +The SP should look something like this when your SP is created: + +```.json +{"clientId": "", + "clientSecret": "", + "subscriptionId": "", + "tenantId": ""} +``` + +For example: + +```.json +{"clientId": "00000000-0000-0000-0000-000000000000", + "clientSecret": "super-duper-secret-value", + "subscriptionId": "00000000-0000-0000-0000-000000000000", + "tenantId": "00000000-0000-0000-0000-000000000000"} +``` + +Upload the entire json to your github secret. +We use one secret per environment and the secrets **have** to follow the namestandard that we have set for our secrets, +else the github action workflow won't be able to find the secret. + +The secret name should be **AZURE_CREDENTIALS_\**, for example **AZURE_CREDENTIALS_DEV** + +To upload the secret to github you can use the github UI or you can use the [gh cli](https://github.com/cli/cli) to upload secrets to GitHub. + +Assuming that you are storing the SP json data in a file you could do: + +```shell +gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_DEV < dev-secrets.json +gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_QA < qa-secrets.json +gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_PROD < prod-secrets.json +``` + +### CI/CD + +Just like in azure devops case we have created a +[basic pipeline](https://github.com/XenitAB/azure-devops-templates/terraform-docker-github/README.md) +for easy use. + +Bellow you can find a example pipeline that uses the github action workflow. + +```.github/workflows/core.yaml +name: terraform_core + +on: + push: + branches: + - main + paths: + - core/** + pull_request: + paths: + - core/** + workflow_dispatch: + inputs: + OPA_BLAST_RADIUS: + description: OPA Blast Radius + required: true + default: "50" + +jobs: + terraform: + uses: xenitab/azure-devops-templates/.github/workflows/terraform-docker.yaml@2021.09.3 + with: + DIR: core + runs-on: '["self-hosted", "linux"]' # If you don't want to use the default ubuntu-latest + ENVIRONMENTS: | + { + "environments":[ + {"name":"dev"}, + {"name":"qa"}, + {"name":"prod"} + ] + } + secrets: + AZURE_CREDENTIALS_DEV: ${{ secrets.AZURE_CREDENTIALS_DEV }} + AZURE_CREDENTIALS_QA: ${{ secrets.AZURE_CREDENTIALS_QA }} + AZURE_CREDENTIALS_PROD: ${{ secrets.AZURE_CREDENTIALS_PROD }} +``` diff --git a/sidebars.js b/sidebars.js index f8d5c0ea982..b6b47bad266 100644 --- a/sidebars.js +++ b/sidebars.js @@ -37,7 +37,8 @@ module.exports = { "xks/operator-guide/eks", "xks/operator-guide/blast-radius", "xks/operator-guide/operator-guide", - "xks/operator-guide/azure-devops-agents" + "xks/operator-guide/azure-devops-agents", + "xks/operator-guide/github", ] } ] From dabc4a12facba625cd98a466a84a467960c795f8 Mon Sep 17 00:00:00 2001 From: Edvin Norling Date: Thu, 21 Oct 2021 10:26:43 +0200 Subject: [PATCH 2/4] Refactor github page to explain what it does * Prepare for future AWS support in the way we are writing the doc --- docs/xks/operator-guide/github.md | 89 +++++++++++++++++-------------- 1 file changed, 48 insertions(+), 41 deletions(-) diff --git a/docs/xks/operator-guide/github.md b/docs/xks/operator-guide/github.md index a3e02305c98..eb8f62aaa63 100644 --- a/docs/xks/operator-guide/github.md +++ b/docs/xks/operator-guide/github.md @@ -3,55 +3,24 @@ id: github title: XKF on Github --- -## Terraform - -In GitHub there is no service connection like there is in Azure DevOps. -Instead we utilize a Azure SP to talk to Azure when running our CI pipelines. - -Create a SP with the access that terraform requires to perform all the tasks you want. -You can read more about SP creation in our [operator guide](operator-guide.md) - -The SP should look something like this when your SP is created: - -```.json -{"clientId": "", - "clientSecret": "", - "subscriptionId": "", - "tenantId": ""} -``` - -For example: - -```.json -{"clientId": "00000000-0000-0000-0000-000000000000", - "clientSecret": "super-duper-secret-value", - "subscriptionId": "00000000-0000-0000-0000-000000000000", - "tenantId": "00000000-0000-0000-0000-000000000000"} -``` +As a XKF user you can use both Azure DevOps and GitHub to store +your terraform and gitops repositories. -Upload the entire json to your github secret. -We use one secret per environment and the secrets **have** to follow the namestandard that we have set for our secrets, -else the github action workflow won't be able to find the secret. - -The secret name should be **AZURE_CREDENTIALS_\**, for example **AZURE_CREDENTIALS_DEV** +In this document we will go through how to use XKF on GitHub focusing +on Infrastructure As Code (IAC) using Terraform. -To upload the secret to github you can use the github UI or you can use the [gh cli](https://github.com/cli/cli) to upload secrets to GitHub. - -Assuming that you are storing the SP json data in a file you could do: +## Terraform -```shell -gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_DEV < dev-secrets.json -gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_QA < qa-secrets.json -gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_PROD < prod-secrets.json -``` +How to run Terraform plan and apply through a GitHub action workflow. -### CI/CD +### Workflow Just like in azure devops case we have created a [basic pipeline](https://github.com/XenitAB/azure-devops-templates/terraform-docker-github/README.md) for easy use. -Bellow you can find a example pipeline that uses the github action workflow. +Below you can find a example pipeline that uses the github action workflow. +Read further down to see how to create the secrets needed to run the pipeline. ```.github/workflows/core.yaml name: terraform_core @@ -74,7 +43,7 @@ on: jobs: terraform: - uses: xenitab/azure-devops-templates/.github/workflows/terraform-docker.yaml@2021.09.3 + uses: xenitab/azure-devops-templates/.github/workflows/terraform-docker.yaml@2021.10.1 with: DIR: core runs-on: '["self-hosted", "linux"]' # If you don't want to use the default ubuntu-latest @@ -91,3 +60,41 @@ jobs: AZURE_CREDENTIALS_QA: ${{ secrets.AZURE_CREDENTIALS_QA }} AZURE_CREDENTIALS_PROD: ${{ secrets.AZURE_CREDENTIALS_PROD }} ``` + +### Azure + +Create a Service Principal(SP) with the access that terraform requires to perform all the tasks you want. +You can read more about SP creation in our [operator guide](operator-guide.md) + +The SP should look something like this when your SP is created: + +```.json +{"clientId": "", + "clientSecret": "", + "subscriptionId": "", + "tenantId": ""} +``` + +For example: + +```.json +{"clientId": "00000000-0000-0000-0000-000000000000", + "clientSecret": "super-duper-secret-value", + "subscriptionId": "00000000-0000-0000-0000-000000000000", + "tenantId": "00000000-0000-0000-0000-000000000000"} +``` + +Upload the entire json as your github secret. +We use one secret per environment and it **have** to follow our namestandard or +else the github action workflow won't be able to find the secret. +The secret name should be **AZURE_CREDENTIALS_\**, for example **AZURE_CREDENTIALS_DEV** + +To upload the secret to github you can use the github UI or you can use the [gh cli](https://github.com/cli/cli) to upload secrets to GitHub. + +Assuming that you are storing the SP json data in a file you could do: + +```shell +gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_DEV < dev-secrets.json +gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_QA < qa-secrets.json +gh secret -R ORG/xks-terraform set AZURE_CREDENTIALS_PROD < prod-secrets.json +``` From 32a9ac81c47b69914acf3545c928400548532171 Mon Sep 17 00:00:00 2001 From: Edvin Norling Date: Thu, 21 Oct 2021 10:41:30 +0200 Subject: [PATCH 3/4] Point out the formatting of the SP --- docs/xks/operator-guide/github.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/xks/operator-guide/github.md b/docs/xks/operator-guide/github.md index eb8f62aaa63..479eec44b64 100644 --- a/docs/xks/operator-guide/github.md +++ b/docs/xks/operator-guide/github.md @@ -66,16 +66,12 @@ jobs: Create a Service Principal(SP) with the access that terraform requires to perform all the tasks you want. You can read more about SP creation in our [operator guide](operator-guide.md) -The SP should look something like this when your SP is created: +We are using [Azure Login GitHub Action](https://github.com/marketplace/actions/azure-login#configure-deployment-credentials) +to login to Azure. When uploading your SP to GitHub make sure to follow the formatting in the examples. -```.json -{"clientId": "", - "clientSecret": "", - "subscriptionId": "", - "tenantId": ""} -``` +This is to prevent unnecessary masking of { } in your logs which are in dictionary form. -For example: +**For example, do**: ```.json {"clientId": "00000000-0000-0000-0000-000000000000", @@ -84,6 +80,17 @@ For example: "tenantId": "00000000-0000-0000-0000-000000000000"} ``` +**instead of**: + +```.json +{ + "clientId": "00000000-0000-0000-0000-000000000000", + "clientSecret": "super-duper-secret-value", + "subscriptionId": "00000000-0000-0000-0000-000000000000", + "tenantId": "00000000-0000-0000-0000-000000000000" +} +``` + Upload the entire json as your github secret. We use one secret per environment and it **have** to follow our namestandard or else the github action workflow won't be able to find the secret. From c266ab54f6f546e76b16649161103483b3a9f961 Mon Sep 17 00:00:00 2001 From: Edvin Norling Date: Thu, 21 Oct 2021 11:37:28 +0200 Subject: [PATCH 4/4] Clarify where to store the workflow --- docs/xks/operator-guide/github.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/xks/operator-guide/github.md b/docs/xks/operator-guide/github.md index 479eec44b64..c289be6c33d 100644 --- a/docs/xks/operator-guide/github.md +++ b/docs/xks/operator-guide/github.md @@ -22,6 +22,8 @@ for easy use. Below you can find a example pipeline that uses the github action workflow. Read further down to see how to create the secrets needed to run the pipeline. +You should store this GitHub action in your Terraform repository under `.github/workflows/name.yaml` + ```.github/workflows/core.yaml name: terraform_core @@ -61,12 +63,12 @@ jobs: AZURE_CREDENTIALS_PROD: ${{ secrets.AZURE_CREDENTIALS_PROD }} ``` -### Azure +### Azure Service Principal Create a Service Principal(SP) with the access that terraform requires to perform all the tasks you want. You can read more about SP creation in our [operator guide](operator-guide.md) -We are using [Azure Login GitHub Action](https://github.com/marketplace/actions/azure-login#configure-deployment-credentials) +The workflow is using [Azure Login GitHub Action](https://github.com/marketplace/actions/azure-login#configure-deployment-credentials) to login to Azure. When uploading your SP to GitHub make sure to follow the formatting in the examples. This is to prevent unnecessary masking of { } in your logs which are in dictionary form. @@ -92,9 +94,8 @@ This is to prevent unnecessary masking of { } in your logs which are in dictiona ``` Upload the entire json as your github secret. -We use one secret per environment and it **have** to follow our namestandard or -else the github action workflow won't be able to find the secret. -The secret name should be **AZURE_CREDENTIALS_\**, for example **AZURE_CREDENTIALS_DEV** +The workflow uses one secret per environment and we recommend that you follow our namestandard. +The secret name the workflow use is **AZURE_CREDENTIALS_\**, for example **AZURE_CREDENTIALS_DEV** To upload the secret to github you can use the github UI or you can use the [gh cli](https://github.com/cli/cli) to upload secrets to GitHub.