diff --git a/_docs-sources/pipelines/hello-world/index.md b/_docs-sources/pipelines/hello-world/index.md index 8f3f06635a..27d7a00bb2 100644 --- a/_docs-sources/pipelines/hello-world/index.md +++ b/_docs-sources/pipelines/hello-world/index.md @@ -10,7 +10,7 @@ By the end, you’ll have: - Two GitHub repositories - `infrastructure-live` — Defines the infrastructure that is deployed to your AWS account - `infrastructure-pipelines` — Contains deployment definitions for your infrastructure -- An IAM role in your AWS account that allows GitHub Actions to assume a role in your AWS account using OIDC +- A group of IAM roles in your AWS account that allow GitHub Actions modify your account through OIDC - An S3 Bucket deployed automatically by Gruntwork Pipelines ## Prerequisites @@ -20,17 +20,20 @@ Before you begin, make sure you have: - Permissions to create and administer repositories in GitHub - A sandbox or development AWS account - Valid [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) for a user with AdministratorAccess to the AWS account mentioned above -- [Boilerplate](https://github.com/gruntwork-io/boilerplate#install) installed on your system (requires Gruntwork subscription) - [Terragrunt](https://terragrunt.gruntwork.io/) installed on your system -- A [classic GitHub PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with `repo` and `workflow` scopes and access to Gruntwork modules +- A GitHub user with an active **Gruntwork Subscription** for creating a [classic GitHub PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with `repo` & `workflow` scopes as well as access to Gruntwork packages + :::info + To create a classic GitHub PAT: + 1. Navigate to https://github.com/settings/tokens + 1. Choose `Generate new token (classic)`. + 1. In the "Note" field, enter "Gruntwork Pipelines POC" (or something similar), select the `repo` and `workflow` scope checkboxes, then click `Generate token`. -:::info -To create a classic GitHub PAT, go to https://github.com/settings/profile, click on `Developer Settings`, then `Personal access tokens`, then `Tokens (classic)`, then `Generate new token (classic)`. In the "Note" field, enter "Gruntwork Pipelines POC" (or something similar), select the `repo` and `workflow` scope checkboxes, then click `Generate token`. Keep your token handy; we'll be using it shortly. -::: + **Keep your token handy; we'll be using it shortly.** + ::: ## Setting up the repositories -First, you’ll set up two git repositories: +In this section, you’ll set up two git repositories: - The `infrastructure-live` repository will contain the definitions of your infrastructure as code (IaC) - The `infrastructure-pipelines` repository will define how your IaC will be deployed. @@ -44,6 +47,11 @@ Finally, you’ll set up your PAT as a GitHub Actions secret in each repository. ### Create the repositories +Create a new repository from the templates provided below: + +1. [gruntwork-infra-live-standalone-template](https://github.com/gruntwork-io/gruntwork-infra-live-standalone-template) +1. [gruntwork-pipelines-standalone-template](https://github.com/gruntwork-io/gruntwork-pipelines-standalone-template) + :::warning In this tutorial, we will use the default GitHub repo configuration. @@ -52,105 +60,77 @@ In a production environment, we recommend setting up for your `main` branch as described in [Branch Protection](../security/branch-protection.md#recommended-settings). ::: -Navigate to the repositories tab of your organization or personal GitHub account in your web browser. Repeat the following steps twice to create one repository named `infrastructure-live` and one repository named `infrastructure-pipelines`. +For each of the template repositories above: -1. Click the `New Repository` button. -1. In the `Repository name` field enter either `infrastructure-live` or `infrastructure-pipelines` -1. Select the `Private` option -1. Do not initialize the repo with a README, gitignore, or license +1. Click `Use this template` +1. Click `Create a new repository` +1. Select your preferred organization from the owner dropdown +1. Name the repo anything you like and make note of the name. +1. Select the `Private` radio button +1. Click `Create Repository` +1. Repeat for the second template repository ![GitHub form for creating a new repository](/img/pipelines/tutorial/create_new_repo_form.png) ### Setting up secrets -:::warning -In this tutorial, we will use a single GitHub Personal Access Token (PAT) with broad access. - -In a production environment, we recommend using a mix of fine-grained and classic PATs as described in [Machine Users](../security/machine-users.md). -::: - -Next, you're going to configure GitHub Actions secrets for each repository. Our goal here is to enable: +In this section, you're going to configure GitHub Actions secrets for each repository. Our goal here is to enable: - `infrastructure-live` to kick off GitHub Actions workflows in the `infrastructure-pipeline` repository - `infrastructure-pipelines` to clone the `infrastructure-live` repository - `infrastructure-pipelines` to access Gruntwork modules -First, navigate to the `infrastructure-live` repository. Select the `Settings` tab, select the `Secrets and variables` drop down on the left side panel, then select `Actions`. Create the following secrets and use your GitHub PAT as the value for all of them: +:::warning +In this tutorial, we will use a single GitHub Personal Access Token (PAT) with broad access in secrets. -- `PIPELINES_DISPATCH_TOKEN` -- `INFRA_LIVE_ACCESS_TOKEN` -- `GRUNTWORK_CODE_ACCESS_TOKEN` +In a production environment, we recommend using a mix of fine-grained and classic PATs as described in [Machine Users](../security/machine-users.md). +::: -Next, navigate to the `infrastructure-pipelines` repository. Select the `Settings` tab, select the `Secrets and variables` drop down on the left side panel, then select `Actions`. Create the following secrets and use your GitHub PAT as the value for all of them: +Copy the script below and edit the exports at the top to match your git repo names and GitHub PAT, then run it while authenticated using the **GitHub CLI** -- `INFRA_LIVE_ACCESS_TOKEN` -- `GRUNTWORK_CODE_ACCESS_TOKEN` +```bash +export INFRA_LIVE_REPO_NAME= +export INFRA_PIPELINES_REPO_NAME= +export GHA_TOKEN= # This is the GitHub PAT you created in the prerequisites section -## Generating code +gh secret set INFRA_LIVE_ACCESS_TOKEN -a actions --repo $INFRA_LIVE_REPO_NAME -b $GHA_TOKEN; +gh secret set GRUNTWORK_CODE_ACCESS_TOKEN -a actions --repo $INFRA_LIVE_REPO_NAME -b $GHA_TOKEN; +gh secret set PIPELINES_DISPATCH_TOKEN -a actions --repo $INFRA_LIVE_REPO_NAME -b $GHA_TOKEN; -Next, you’ll write the IaC and GitHub Actions workflow code required to run Gruntwork Pipelines. +gh secret set INFRA_LIVE_ACCESS_TOKEN -a actions --repo $INFRA_PIPELINES_REPO_NAME -b $GHA_TOKEN; +gh secret set GRUNTWORK_CODE_ACCESS_TOKEN -a actions --repo $INFRA_PIPELINES_REPO_NAME -b $GHA_TOKEN; +gh secret set PIPELINES_BOOTSTRAP_TOKEN -a actions --repo $INFRA_PIPELINES_REPO_NAME -b $GHA_TOKEN; +``` -Rather than copy & pasting this code from documentation, we're going to generate the code using [Boilerplate](https://github.com/gruntwork-io/boilerplate), a code generation tool authored by Gruntwork. The generated code will: +## Generating code -- Provision an AWS IAM role (which Pipelines will assume to manage AWS resources) -- Configure one GitHub Actions (GHA) workflow for the `infrastructure-live` repository -- Configure one GHA workflow for the `infrastructure-pipelines` repository +In this section, you’ll generate the IaC and GitHub Actions workflow code required to run Gruntwork Pipelines using our Bootstrap GitHub Action Workflow +included in your repository that was generated with the Gruntwork template repos. -The code generation template for `infrastructure-live` sets up a folder structure that follows Gruntwork’s recommended folder structure for Terragrunt, so you could continue to use the generated code in perpetuity and simply add more AWS accounts and resource definitions if you so choose! ### Infrastructure-pipelines -First, generate the `infrastructure-pipelines` repository code using [Boilerplate](https://github.com/gruntwork-io/boilerplate). On your local computer, git clone the newly created `infrastructure-pipelines` repository, `cd` into the repo directory, then use the following command replacing `` with the name of your GitHub organization. - -```bash -boilerplate --template-url "github.com/gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/infrastructure-pipelines?ref=v1.0.1" \ - --output-folder . \ - --var InfraLiveRepoName="infrastructure-live" \ - --var GithubOrg="" \ - --non-interactive -``` - -Push your changes to the `infrastructure-pipelines` repository you created in [Create the repositories](#create-the-repositories). +1. Navigate to `Actions` +1. Click `Infrastructure Pipelines Bootstrap` in the sidebar on the left +1. Select the `Run Workflow` dropdown and enter the name of your `infrastructure-live` repository. + ![Infrastructure-pipelines Workflow Form](/img/pipelines/tutorial/pipelines_run_workflow.png) +1. Click `Run Workflow` to start the workflow. +1. A Pull Request(PR) will be created after the workflow completes. +1. Review the changes, follow the instructions in the PR and then merge. ### Infrastructure-live -Next, generate the `infrastructure-live` repository code using Boilerplate. On your local computer, git clone the newly created `infrastructure-live` repository, `cd` into the repo directory, then use the following command, replacing the values wrapped in `<>` with real values for your organization. - -The AWSAccount* variables correspond to the AWS account in which your `infrastructure-live` repository will manage AWS resources. The `OrgNamePrefix` is used to prefix your resource names with your org name so that they are identifiable. For example, we will create an S3 bucket to store Terraform/OpenTofu state, whose name begins with the value of `OrgNamePrefix`. - -```bash -boilerplate --template-url "github.com/gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/single-account-pipeline?ref=v1.0.1" \ - --output-folder . \ - --var AwsAccountName="" \ - --var AwsAccountId="''" \ - --var AwsAccountEmail="" \ - --var InfraLiveRepoName="infrastructure-live" \ - --var InfraPipelinesRepoName="infrastructure-pipelines" \ - --var GithubOrg="" \ - --var OrgNamePrefix="" \ - --var RequestingTeamName="" \ - --non-interactive -``` - -The generated code creates a full Terragrunt `infrastructure-live` folder structure that creates a single AWS resource, which is an AWS IAM role that Pipelines will use to deploy resources in your AWS account. More specifically, it will create an instance of the Gruntwork [github-actions-iam-role module](https://github.com/gruntwork-io/terraform-aws-security/tree/main/modules/github-actions-iam-role), which creates an IAM role that is assumable by GitHub using OIDC, and which can only be assumed when running a GitHub Actions workflow from the `main` branch. - -With Pipelines, all `terragrunt` operations will happen in GitHub Actions in response to a Pull Request or `git push`. But before that will work, you will first need to run an `apply` locally to provision the IAM role. This should be the only time you need to manually run `apply` to provision resources for this account, moving forward Pipelines will handle the lifecycle of all resources for you, based on the code you commit to your repository. - -First, run a `run-all plan` in the newly created directory named after the account to see the resources that will be provisioned. Replace `` with the value you used for `AwsAccountName` in the boilerplate command above. - -```bash -cd / -terragrunt run-all plan -``` -Terragrunt will prompt you to create the Terragrunt state and logs buckets, enter `y` when prompted, then hit enter. - -Terragrunt will then run the plan. Once you have reviewed the new resources to be created, run `terragrunt run-all apply` to create the resources. - -Finally, git push your changes to the `infrastructure-live` repository you created in [Create the repositories](#create-the-repositories). +1. Navigate to `Actions` +1. Click `Infrastructure Live Bootstrap` in the sidebar on the left +1. Select the `Run Workflow` dropdown and enter the requested details. + ![Infrastructure-live Workflow Form](/img/pipelines/tutorial/infra_live_workflow.png) +1. Click `Run Workflow` to start the workflow. +1. A Pull Request(PR) will be created after the workflow completes. +1. Review the changes, follow the instructions in the PR and then merge. ## Running your first pipeline -Next you’ll create a resource in your AWS account using Pipelines and GitOps workflows. You’ll define a `terragrunt.hcl` file that creates an AWS S3 bucket in your AWS account, push your changes and create a pull request (PR) to run a `plan` action, then run an `apply` action to create the bucket by merging your PR. +In this section, you’ll create a resource in your AWS account using Pipelines and GitOps workflows by defining a `terragrunt.hcl` file that creates an AWS S3 bucket in your AWS account, pushing your changes and creating a Pull Request (PR) to run a `plan` action, then run an `apply` action to create the bucket by merging your PR. ### Adding a new S3 bucket @@ -170,13 +150,8 @@ locals { } ``` -Next, add the terragrunt code to create an S3 bucket. Copy the terragrunt code below, replacing `` with your desired bucket name. S3 bucket names need to be globally unique, so we've provided a helper script below to help generate the name of your bucket. You may name the bucket whatever you like, just make sure it’s unique. +Next, add the terragrunt code to create an S3 bucket. Copy the terragrunt code below, replacing `` with your desired bucket name. You may name the bucket whatever you like, just make sure it’s unique. -```bash -export export UNIQUE_ID=$(uuidgen | tr 'A-Z' 'a-z') -export DATE_NOW=$(date "+%F") -echo "gwp-bucket-${UNIQUE_ID}-${DATE_NOW}" -``` ```hcl title="///data-storage/s3/terragrunt.hcl" # ------------------------------------------------------------------------------------------------------ diff --git a/_docs-sources/pipelines/security/machine-users.md b/_docs-sources/pipelines/security/machine-users.md index a1bed0d697..ad71e0133c 100644 --- a/_docs-sources/pipelines/security/machine-users.md +++ b/_docs-sources/pipelines/security/machine-users.md @@ -1,49 +1,73 @@ -# Machine Users +# Machine users Gruntwork recommends using CI users in Gruntwork Pipelines, separate from human users in your organization. Using a CI user ensures that a workflow won't break due to an employee leaving your company. Further, using CI users allow you to apply granular permissions that may normally be too restrictive for a normal employee to do their daily work. +:::info + + This guide will take approximately 25 minutes to complete. + +::: + ## Creating machine users -Gruntwork recommends using two ci-users - one with the ability to open pull requests and run workflows on your behalf and another that can only read code from GitHub. Restrictive permissions are then applied to each user to limit them to only perform the required actions to accomplish their tasks. This means that in order to actually run a pipeline job, both users must be involved at separate stages. +Gruntwork Pipelines requires using two machine users; one with the ability to open pull requests and run workflows on your behalf and another that can only read code from GitHub. Restrictive permissions are then applied to each user to limit them to only perform the required actions to accomplish their tasks. This means that in order to actually run a pipeline job, both users must be involved at separate stages. We’ll refer to this user as `ci-user` and `ci-read-only-user`, but you may name them anything you like. These users **must**: - 1. Be a member of your GitHub Organization - 1. Have permission to create Pull requests within your GitHub Organization - 1. Be a member of your team in **Gruntwork**’s GitHub Organization (See [instructions on inviting a user to your team](https://docs.gruntwork.io/developer-portal/invite-team#inviting-team-members) and [linking the user’s GitHub ID to Gruntwork](https://docs.gruntwork.io/developer-portal/link-github-id)) +1. Both be members of your GitHub Organization +1. Both be members of your team in **Gruntwork**’s GitHub Organization (See [instructions on inviting a user to your team](https://docs.gruntwork.io/developer-portal/invite-team#inviting-team-members) and [linking the user’s GitHub ID to Gruntwork](https://docs.gruntwork.io/developer-portal/link-github-id)) + +## Storing secrets +During this setup, you will need to generate and securely store three GitHub tokens for two GitHub users. You will need a temporary location for these sensitive values between generating them and storing them in GitHub Actions. Do so according to your company's recommended security best practices (e.g., do not store them in Slack, a sticky note, etc., during this exercise.) + +:::note +Your organization is required to rotate the GitHub tokens and update all GitHub secrets that use them. +::: ### ci-user The `ci-user` orchestrates workflows, can open pull requests from automated code generation, and leave comments on pull requests. This user should have two GitHub Fine Grained [Personal Access Tokens (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#fine-grained-personal-access-tokens)s with the following permissions. -1. `INFRA_LIVE_ACCESS`. This token will be used to grant GitHub Actions access to clone your `infrastructure-live` repository, open PRs, and create comments & issues. +First, invite the `ci-user` to both your `infrastructure-live` and `infrastructure-pipelines` repositories with collaborator access. Then, create the following access tokens in the `ci-user`'s GitHub account: - This token **must** have: +1. `INFRA_LIVE_ACCESS`. This is a fine-grained GitHub access token and will be used to grant GitHub Actions access to clone your `infrastructure-live` repository, open PRs, and create comments & issues. - - Content (read & write) access - - Pull Requests (read & write) access + This token **must** have the following permissions to your **`infrastructure-live`** repo in your GitHub Organization: - to your **`infrastructure-live`** repo in your GitHub Organization. + - Content read & write access + - Issues read & write access + - Metadata read access + - Pull Requests read & write access ![INFRA_LIVE_ACCESS PAT Configuration](/img/pipelines/security/INFRA_LIVE_ACCESS.png) -1. `PIPELINES_DISPATCH`. This token will be used to grant GitHub Actions permission to trigger workflows in your `infrastructure-pipelines` repository. +1. `PIPELINES_DISPATCH`. This token will be used to grant GitHub Actions permission to trigger workflows in your `infrastructure-pipelines` repository to your **`infrastructure-pipelines`** repo in your GitHub Organization: This token **must** have: - - Contents (read only) access - - Actions (read & write) access - - to your **`infrastructure-pipelines`** repo in your GitHub Organization. + - Actions read & write access + - Contents read only access + - Metadata read access ![PIPELINES_DISPATCH PAT Configuration](/img/pipelines/security/PIPELINES_DISPATCH.png) +1. `PIPELINES_BOOTSTRAP`. This token will be used to grant GitHub Actions permission to create open PRs and create comments in your `infrastructure-pipelines` repository in your GitHub Organization. **This is only required during the bootstrap process of the pipelines repository and should be removed immediately after.** + + This token **must** have: + + - Contents read & write access + - Metadata read access + - Pull Requests read & write access + + ![PIPELINES_BOOTSTRAP PAT Configuration](/img/pipelines/security/PIPELINES_BOOTSTRAP.png) ### ci-read-only-user -The `ci-read-only-user` is used clone `infrastructure-live` and in terragrunt actions to access Gruntwork Library modules and your own `infrastructure-modules` repository (or repositories). The `ci-read-only-user` should have a single classic token [Personal Access Tokens (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with read permissions. Because classic PATs have coarse granularity, we recommend putting this user in a GitHub team that only has READ access to `infrastructure-live` and `infrastructure-modules` in your own GitHub Organization. By adding this user to the Gruntwork Developer portal, they will automatically gain access to the Gruntwork Library. +This user is created to pull down Terraform/OpenTofu code, but not to apply it. The `ci-read-only-user` is used clone `infrastructure-live` and in Terragrunt actions to access Gruntwork Library modules and your own `infrastructure-modules` repository or any custom module repositories that are private. + +The `ci-read-only-user` should have a single classic token [Personal Access Tokens (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with read permissions. Classic PATs have coarse granularity, we recommend putting this user in a GitHub team that only has READ access to `infrastructure-live` and any relevant module repositories in your own GitHub Organization. By adding this user to the Gruntwork Developer portal, they will automatically gain access to the Gruntwork Library. -Create the following token for the `ci-read-only-user`: +Invite `ci-user-read-only` to your `infrastructure-live` repository with collaborator access. Create the following token for the `ci-read-only-user`: 1. `GRUNTWORK_CODE_ACCESS`. This token will be used to manage access to Gruntwork resources during GitHub Action runs. @@ -53,17 +77,31 @@ This token **must** have `repo` scopes. The expiration of this token is up to you and the security posture of your organization, Gruntwork recommend 90 days to avoid having to regularly rotate a token and secrets. +## Invite both machine users to Gruntwork +Ensure both of these machine users are members of your team in **Gruntwork**’s GitHub Organization (See [instructions on inviting a user to your team](https://docs.gruntwork.io/developer-portal/invite-team#inviting-team-members) and [linking the user’s GitHub ID to Gruntwork](https://docs.gruntwork.io/developer-portal/link-github-id)) + ## Repository Secrets Gruntwork Pipelines reads these secrets from GitHub Actions secrets created in the repo. For steps on how to create repository Actions secrets, refer to [creating secrets for a repository](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository). +### infrastructure-live + In the `infrastructure-live` repository create the following secrets: -1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated in the step above as its value. -1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated in the step above as its value. -1. `PIPELINES_DISPATCH_TOKEN`. This should be assigned the _`PIPELINES_DISPATCH`_ token generated in the step above as its value. +1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated by the `ci-read-only-user` in the [secrets section](#ci-read-only-user) as its value. +1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. +1. `PIPELINES_DISPATCH_TOKEN`. This should be assigned the _`PIPELINES_DISPATCH`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. + +### infrastructure-pipelines In the `infrastructure-pipelines` repository create the following secrets: -1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated in the step above as its value. -1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated in the step above as its value. +1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated by the `ci-read-only-user` in the [secrets section](#ci-read-only-user) as its value. +1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. +1. `PIPELINES_BOOTSTRAP_TOKEN`. This should be assigned the _`PIPELINES_BOOTSTRAP`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. + + :::warning + For security purposes, you should delete: + 1. The `PIPELINES_BOOTSTRAP` Personal Access Token from the `ci-user` GitHub account and + 1. The `PIPELINES_BOOTSTRAP_TOKEN` GitHub Actions secret from the `infrastructure-pipelines` repository -Your organization is required to rotate tokens and update all secrets that use them. + after the bootstrap process is complete. + ::: diff --git a/_docs-sources/pipelines/upgrading/index.md b/_docs-sources/pipelines/upgrading/index.md index 1bdd1eba08..cc4d0a3fa0 100644 --- a/_docs-sources/pipelines/upgrading/index.md +++ b/_docs-sources/pipelines/upgrading/index.md @@ -4,7 +4,7 @@ This migration guide is intended for those running the previous version of Grunt To accomplish this task, we'll be using ECS Deploy Runner to deploy Pipelines, then using Pipelines to destroy ECS Deploy Runner. -## What's new +## What's New ECS Deploy Runner was designed as a highly secure approach to CI/CD for infrastructure. With Pipelines, we adapted the security principles built into ECS Deploy Runner to embrace modern CI system functionality, leaning heavily on GitHub Actions native functionality to create a more streamlined setup experience, and better overall UX. The major changes in Pipelines include: @@ -14,61 +14,49 @@ ECS Deploy Runner was designed as a highly secure approach to CI/CD for infrastr ## Prerequisites -- [Boilerplate](https://github.com/gruntwork-io/boilerplate#install) installed on your system (requires Gruntwork subscription) - Ability to create repositories in your GitHub Organization - Ability to add users to your GitHub Organization and the Gruntwork Developer Portal - Permissions to create secrets in GitHub repositories - -## GitHub Personal Access Token Setup - -Pipelines uses several GitHub Personal Access Tokens (PATs) across workflows, with each PAT possessing only the minimal set of permissions necessary for its specific task. To uphold the principle of least privilege for each PAT, we advise maintaining two distinct users: the pre-existing CI user and a new CI Read Only user. The CI Read Only user should have access to read your `infrastructure-live` repository, `infrastructure-modules` repository (if applicable), and the Gruntwork Library. - -### Create a CI Read Only User - -Follow the steps on [signing up a new GitHub account](https://docs.github.com/en/get-started/signing-up-for-github/signing-up-for-a-new-github-account) to create a new user. If you have an email address for your existing CI user, we recommend using plus addressing, such as `ci+read-only@your-domain.com`, as the email address for the new account. - -Add the new user to your GitHub org, adding it to a Team with read only access to your `infrastructure-live` and `infrastructure-modules` repository (if applicable). Follow the steps on [adding a user to the Gruntwork Developer portal](../../developer-portal/invite-team.md) to grant the CI Read Only user access to the Gruntwork Library. - -### Create GitHub Personal Access Tokens for GitHub Actions Workflows - -Next, create the following PATs for the new CI Read Only user. We'll use these tokens shortly. - -- `GRUNTWORK_CODE_ACCESS_TOKEN`: Classic token with READ access to repos. This should should be associated with the `CI Read Only` user created in [creating a CI Read only user](#creating-a-ci-read-only-user). -- `PIPELINES_DISPATCH_TOKEN`: Fine-grained token that only has workflow permissions to execute workflows in the `infrastructure-pipelines` repository. This token should be associated with your existing CI user. -- `INFRA_LIVE_ACCESS_TOKEN`: Fine-grained token with READ and WRITE access to your `infrastructure-live` repository. This token should be associated with your existing CI user. +- [Terragrunt](https://terragrunt.gruntwork.io/) installed on your system ## Create your infrastructure-pipelines repository -Pipelines separates code (IaC) and deployment using two different git repositories. One repository holds both the deployment code and the AWS account access, and assigns write privileges only to a select group of admins. This repository uses OpenID Connect and AWS IAM roles to generate temporary session credentials, avoiding the need to store long-lasting secrets. +Pipelines separates code (IaC) and deployment using two different GitHub repositories. One repository holds both the deployment code and the AWS account access, and assigns write privileges only to a select group of admins. This repository uses OpenID Connect and AWS IAM roles to generate temporary session credentials, avoiding the need to store long-lasting secrets. We strongly recommend naming this repository `infrastructure-pipelines`. All code generated assumes the repository will be located at `/infrastructure-pipelines`, so using a different name will require manual work on your behalf. -### Create the repo +### Initialize infrastructure-pipelines repository + +Create a new Pipelines repository from the template below: -Navigate to the repositories tab of your organization or personal GitHub account in your web browser. +- [gruntwork-pipelines-standalone-template](https://github.com/gruntwork-io/gruntwork-pipelines-standalone-template) -1. Click the **New Repository** button. -1. In the **Repository name** field enter `infrastructure-pipelines` -1. Select the **Private** option -1. Do not initialize the repo with a README, gitignore, or license +1. Click `Use this template` +1. Click `Create a new repository` +1. Select your preferred organization from the owner dropdown +1. Name the repo anything you like and make note of the name. +1. Select the `Private` radio button +1. Click `Create Repository` + +![GitHub form for creating a new repository](/img/pipelines/tutorial/create_new_repo_form.png) :::info For a simple proof of concept, the default repo configuration will suffice. Before using this repository in a production environment, we recommend setting up a [branch protection rule](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) for your `main` branch. At a minimum, we recommend enabling requiring a pull request before merging with at least one reviewer required. ::: -### Use boilerplate to generate code +### Create Temporary Bootstrap Token +Create a temporary PAT with your own GitHub account so it has the ability to open PRs In your `infrastructure-pipelines` repository by using a classic PAT with `repo` and `workflow` access. Once you've generated this token, create a secret called `CUSTOMER_BOOTSTRAP_ACCESS_TOKEN` in `infrastructure-pipelines` and place this value in it. -Next, generate the `infrastructure-pipelines` repository code using [Boilerplate](https://github.com/gruntwork-io/boilerplate), a tool authored by Gruntwork for generating files and folders specially for DevOps use cases. Clone the newly created `infrastructure-pipelines` repository, `cd` into the repo directory, then use the following command replacing `` with the name of your GitHub organization and `` with the name of your infrastructure-live repository. +### Bootstrap the Pipelines Repo -```bash -boilerplate --template-url "git@github.com:gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/infrastructure-pipelines?ref=v1.0.0" \ - --output-folder . \ - --var InfraLiveRepoName="" \ - --var GithubOrg="" \ - --non-interactive -``` +In the newly created repository, go to Actions and run the **Infrastructure Pipelines Bootstrap** and enter in the name of your `infrastructure-live` repository. This will open up a pull request that you can merge. + +### Delete Temporary Bootstrap Token +Once the Pipelines repository is bootstrapped successfully, you can delete the `CUSTOMER_BOOTSTRAP_ACCESS_TOKEN`. + +## GitHub CI Machine Users and Secrets Setup -Push your changes up to the remote repository. +Pipelines utilizes two machine users, one for read-only and another elevated operations. Follow this [guide](../security/machine-users) to create them and and set up the correct access tokens in each of the repositories. ## Create AWS IAM roles for Pipelines @@ -76,50 +64,20 @@ In Pipelines, each of your accounts must have an AWS IAM role. This role shares For more information see [security hardening with OpenID Connect](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect) and [OpenID Connect in AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). -### Generate the code using boilerplate for each account +### Generate the Pipelines IAM code using GitHub Actions for each account -To simplify the migration process, we have developed a template that creates all code and files required to generate the IAM role, OpenID Connect identity provider, and IAM policies. +In `infrastructure-live`, open a pull request and copy this [GitHub Actions workflow file](https://github.com/gruntwork-io/gruntwork-infra-live-standalone-template/blob/main/.github/workflows/bootstrap.yml) into your `.github/workflows` directory. -Use the following command, replacing the values wrapped in `<>` with real values for your organization. The `AwsAccountName` variable should be the name of the top level directory that represents the AWS account that will use Pipelines (e.g, `dev`). You will need to run this command for each account that you would like to upgrade to Pipelines. - -Now `cd` into your `infrastructure-live` repository and run the following command: - -```bash -boilerplate --template-url "git@github.com:gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/pipelines-edr-migration" \ - --output-folder . \ - --var InfraPipelinesRepoName="" \ - --var GithubOrg="" \ - --var AwsAccountName="" \ - --var RequestingTeamName="" \ - --var AwsAccountName="" \ - --var OrgNamePrefix="The name prefix to use for creating resources" \ - --non-interactive -``` - -Create a branch, commit your changes, and push your branch to the remote repository. Then create a pull request targeting your default branch (e.g., `main`). +Once that has been merged into `main`, run the **Infrastructure Live Bootstrap** for each account to generate the Terraform needed to create the IAM Roles and the GitHub OIDC connection to securely do deploys. Each workflow run will open a PR that needs to be merged. :::tip We recommend trying out Pipelines in non-production environments first, before rolling out to production environments. ::: -### Planning and applying - -Create a new branch for your changes, commit your changes to your branch, then push your branch. Next, create a PR to merge your branch into `main` (the default branch in your repository). The ECS Deploy Runner will detect the change and run a `plan` to create the IAM Roles, OpenID Connect providers, and all requires IAM policies. Review the `plan` output and confirm it looks as expected. +The ECS Deploy Runner will detect the change and run a `plan` to create the IAM Roles, OpenID Connect providers, and all requires IAM policies. Review the `plan` output and confirm it looks as expected. Once you have reviewed the `plan` output and gotten any necessary approvals, merge the PR. The EDR version of Pipelines will run `apply` to create the resources. -## Set up secrets - -Pipelines requires the use of GitHub personal access tokens (PAT) to allow workflows in the `infrastructure-live` to run workflows in the `infrastructure-pipelines` repository and workflows in the `infrastructure-pipelines` repository access to the code in `infrastructure-live` as well as repos for the Gruntwork Library. This section will guide you in establishing secrets in both repositories and comprehending the security implications of the multi-PAT configuration. - -### Infrastructure live - -Navigate to your `infrastructure-live` repository (it may be named `-infrastructure-live`). Select the **Settings** tab, select the **Secrets and variables** drop down on the left side panel, then select **Actions**. Create three secrets named `GRUNTWORK_CODE_ACCESS_TOKEN`, `PIPELINES_DISPATCH_TOKEN`, and `INFRA_LIVE_ACCESS_TOKEN`. Use the corresponding value from the tokens you created in [GitHub Personal Access Token Setup](#github-personal-access-token-setup) for the secret value. - -### Infrastructure pipelines - -Navigate to the `infrastructure-pipelines` repository. Select the **Settings** tab, select the **Secrets and variables** drop down on the left side panel, then select **Actions**. Create two secrets named `INFRA_LIVE_ACCESS_TOKEN` and `GRUNTWORK_CODE_ACCESS_TOKEN`. Use the corresponding value from the tokens you created in [GitHub Personal Access Token Setup](#github-personal-access-token-setup) for the secret value. - ## Updating your accounts file As part of a separate product improvement initiative, we have transitioned from the `accounts.json` file to `accounts.yml`. This change not only enhances readability but is also functionally better, with Terraform's built-in YAML formatter better meeting our requirement for regenerating a formatted file to facilitate the addition of new accounts through GitOps workflows. It's important to note that there is no functional distinction between these two file types in either the EDR version of Pipelines or Pipelines. diff --git a/docs/pipelines/hello-world/index.md b/docs/pipelines/hello-world/index.md index c11a0fd8cc..027b74d275 100644 --- a/docs/pipelines/hello-world/index.md +++ b/docs/pipelines/hello-world/index.md @@ -10,7 +10,7 @@ By the end, you’ll have: - Two GitHub repositories - `infrastructure-live` — Defines the infrastructure that is deployed to your AWS account - `infrastructure-pipelines` — Contains deployment definitions for your infrastructure -- An IAM role in your AWS account that allows GitHub Actions to assume a role in your AWS account using OIDC +- A group of IAM roles in your AWS account that allow GitHub Actions modify your account through OIDC - An S3 Bucket deployed automatically by Gruntwork Pipelines ## Prerequisites @@ -20,17 +20,20 @@ Before you begin, make sure you have: - Permissions to create and administer repositories in GitHub - A sandbox or development AWS account - Valid [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) for a user with AdministratorAccess to the AWS account mentioned above -- [Boilerplate](https://github.com/gruntwork-io/boilerplate#install) installed on your system (requires Gruntwork subscription) - [Terragrunt](https://terragrunt.gruntwork.io/) installed on your system -- A [classic GitHub PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with `repo` and `workflow` scopes and access to Gruntwork modules +- A GitHub user with an active **Gruntwork Subscription** for creating a [classic GitHub PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with `repo` & `workflow` scopes as well as access to Gruntwork packages + :::info + To create a classic GitHub PAT: + 1. Navigate to https://github.com/settings/tokens + 1. Choose `Generate new token (classic)`. + 1. In the "Note" field, enter "Gruntwork Pipelines POC" (or something similar), select the `repo` and `workflow` scope checkboxes, then click `Generate token`. -:::info -To create a classic GitHub PAT, go to https://github.com/settings/profile, click on `Developer Settings`, then `Personal access tokens`, then `Tokens (classic)`, then `Generate new token (classic)`. In the "Note" field, enter "Gruntwork Pipelines POC" (or something similar), select the `repo` and `workflow` scope checkboxes, then click `Generate token`. Keep your token handy; we'll be using it shortly. -::: + **Keep your token handy; we'll be using it shortly.** + ::: ## Setting up the repositories -First, you’ll set up two git repositories: +In this section, you’ll set up two git repositories: - The `infrastructure-live` repository will contain the definitions of your infrastructure as code (IaC) - The `infrastructure-pipelines` repository will define how your IaC will be deployed. @@ -44,6 +47,11 @@ Finally, you’ll set up your PAT as a GitHub Actions secret in each repository. ### Create the repositories +Create a new repository from the templates provided below: + +1. [gruntwork-infra-live-standalone-template](https://github.com/gruntwork-io/gruntwork-infra-live-standalone-template) +1. [gruntwork-pipelines-standalone-template](https://github.com/gruntwork-io/gruntwork-pipelines-standalone-template) + :::warning In this tutorial, we will use the default GitHub repo configuration. @@ -52,105 +60,77 @@ In a production environment, we recommend setting up for your `main` branch as described in [Branch Protection](../security/branch-protection.md#recommended-settings). ::: -Navigate to the repositories tab of your organization or personal GitHub account in your web browser. Repeat the following steps twice to create one repository named `infrastructure-live` and one repository named `infrastructure-pipelines`. +For each of the template repositories above: -1. Click the `New Repository` button. -1. In the `Repository name` field enter either `infrastructure-live` or `infrastructure-pipelines` -1. Select the `Private` option -1. Do not initialize the repo with a README, gitignore, or license +1. Click `Use this template` +1. Click `Create a new repository` +1. Select your preferred organization from the owner dropdown +1. Name the repo anything you like and make note of the name. +1. Select the `Private` radio button +1. Click `Create Repository` +1. Repeat for the second template repository ![GitHub form for creating a new repository](/img/pipelines/tutorial/create_new_repo_form.png) ### Setting up secrets -:::warning -In this tutorial, we will use a single GitHub Personal Access Token (PAT) with broad access. - -In a production environment, we recommend using a mix of fine-grained and classic PATs as described in [Machine Users](../security/machine-users.md). -::: - -Next, you're going to configure GitHub Actions secrets for each repository. Our goal here is to enable: +In this section, you're going to configure GitHub Actions secrets for each repository. Our goal here is to enable: - `infrastructure-live` to kick off GitHub Actions workflows in the `infrastructure-pipeline` repository - `infrastructure-pipelines` to clone the `infrastructure-live` repository - `infrastructure-pipelines` to access Gruntwork modules -First, navigate to the `infrastructure-live` repository. Select the `Settings` tab, select the `Secrets and variables` drop down on the left side panel, then select `Actions`. Create the following secrets and use your GitHub PAT as the value for all of them: +:::warning +In this tutorial, we will use a single GitHub Personal Access Token (PAT) with broad access in secrets. -- `PIPELINES_DISPATCH_TOKEN` -- `INFRA_LIVE_ACCESS_TOKEN` -- `GRUNTWORK_CODE_ACCESS_TOKEN` +In a production environment, we recommend using a mix of fine-grained and classic PATs as described in [Machine Users](../security/machine-users.md). +::: -Next, navigate to the `infrastructure-pipelines` repository. Select the `Settings` tab, select the `Secrets and variables` drop down on the left side panel, then select `Actions`. Create the following secrets and use your GitHub PAT as the value for all of them: +Copy the script below and edit the exports at the top to match your git repo names and GitHub PAT, then run it while authenticated using the **GitHub CLI** -- `INFRA_LIVE_ACCESS_TOKEN` -- `GRUNTWORK_CODE_ACCESS_TOKEN` +```bash +export INFRA_LIVE_REPO_NAME= +export INFRA_PIPELINES_REPO_NAME= +export GHA_TOKEN= # This is the GitHub PAT you created in the prerequisites section -## Generating code +gh secret set INFRA_LIVE_ACCESS_TOKEN -a actions --repo $INFRA_LIVE_REPO_NAME -b $GHA_TOKEN; +gh secret set GRUNTWORK_CODE_ACCESS_TOKEN -a actions --repo $INFRA_LIVE_REPO_NAME -b $GHA_TOKEN; +gh secret set PIPELINES_DISPATCH_TOKEN -a actions --repo $INFRA_LIVE_REPO_NAME -b $GHA_TOKEN; -Next, you’ll write the IaC and GitHub Actions workflow code required to run Gruntwork Pipelines. +gh secret set INFRA_LIVE_ACCESS_TOKEN -a actions --repo $INFRA_PIPELINES_REPO_NAME -b $GHA_TOKEN; +gh secret set GRUNTWORK_CODE_ACCESS_TOKEN -a actions --repo $INFRA_PIPELINES_REPO_NAME -b $GHA_TOKEN; +gh secret set PIPELINES_BOOTSTRAP_TOKEN -a actions --repo $INFRA_PIPELINES_REPO_NAME -b $GHA_TOKEN; +``` -Rather than copy & pasting this code from documentation, we're going to generate the code using [Boilerplate](https://github.com/gruntwork-io/boilerplate), a code generation tool authored by Gruntwork. The generated code will: +## Generating code -- Provision an AWS IAM role (which Pipelines will assume to manage AWS resources) -- Configure one GitHub Actions (GHA) workflow for the `infrastructure-live` repository -- Configure one GHA workflow for the `infrastructure-pipelines` repository +In this section, you’ll generate the IaC and GitHub Actions workflow code required to run Gruntwork Pipelines using our Bootstrap GitHub Action Workflow +included in your repository that was generated with the Gruntwork template repos. -The code generation template for `infrastructure-live` sets up a folder structure that follows Gruntwork’s recommended folder structure for Terragrunt, so you could continue to use the generated code in perpetuity and simply add more AWS accounts and resource definitions if you so choose! ### Infrastructure-pipelines -First, generate the `infrastructure-pipelines` repository code using [Boilerplate](https://github.com/gruntwork-io/boilerplate). On your local computer, git clone the newly created `infrastructure-pipelines` repository, `cd` into the repo directory, then use the following command replacing `` with the name of your GitHub organization. - -```bash -boilerplate --template-url "github.com/gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/infrastructure-pipelines?ref=v1.0.1" \ - --output-folder . \ - --var InfraLiveRepoName="infrastructure-live" \ - --var GithubOrg="" \ - --non-interactive -``` - -Push your changes to the `infrastructure-pipelines` repository you created in [Create the repositories](#create-the-repositories). +1. Navigate to `Actions` +1. Click `Infrastructure Pipelines Bootstrap` in the sidebar on the left +1. Select the `Run Workflow` dropdown and enter the name of your `infrastructure-live` repository. + ![Infrastructure-pipelines Workflow Form](/img/pipelines/tutorial/pipelines_run_workflow.png) +1. Click `Run Workflow` to start the workflow. +1. A Pull Request(PR) will be created after the workflow completes. +1. Review the changes, follow the instructions in the PR and then merge. ### Infrastructure-live -Next, generate the `infrastructure-live` repository code using Boilerplate. On your local computer, git clone the newly created `infrastructure-live` repository, `cd` into the repo directory, then use the following command, replacing the values wrapped in `<>` with real values for your organization. - -The AWSAccount* variables correspond to the AWS account in which your `infrastructure-live` repository will manage AWS resources. The `OrgNamePrefix` is used to prefix your resource names with your org name so that they are identifiable. For example, we will create an S3 bucket to store Terraform/OpenTofu state, whose name begins with the value of `OrgNamePrefix`. - -```bash -boilerplate --template-url "github.com/gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/single-account-pipeline?ref=v1.0.1" \ - --output-folder . \ - --var AwsAccountName="" \ - --var AwsAccountId="''" \ - --var AwsAccountEmail="" \ - --var InfraLiveRepoName="infrastructure-live" \ - --var InfraPipelinesRepoName="infrastructure-pipelines" \ - --var GithubOrg="" \ - --var OrgNamePrefix="" \ - --var RequestingTeamName="" \ - --non-interactive -``` - -The generated code creates a full Terragrunt `infrastructure-live` folder structure that creates a single AWS resource, which is an AWS IAM role that Pipelines will use to deploy resources in your AWS account. More specifically, it will create an instance of the Gruntwork [github-actions-iam-role module](https://github.com/gruntwork-io/terraform-aws-security/tree/main/modules/github-actions-iam-role), which creates an IAM role that is assumable by GitHub using OIDC, and which can only be assumed when running a GitHub Actions workflow from the `main` branch. - -With Pipelines, all `terragrunt` operations will happen in GitHub Actions in response to a Pull Request or `git push`. But before that will work, you will first need to run an `apply` locally to provision the IAM role. This should be the only time you need to manually run `apply` to provision resources for this account, moving forward Pipelines will handle the lifecycle of all resources for you, based on the code you commit to your repository. - -First, run a `run-all plan` in the newly created directory named after the account to see the resources that will be provisioned. Replace `` with the value you used for `AwsAccountName` in the boilerplate command above. - -```bash -cd / -terragrunt run-all plan -``` -Terragrunt will prompt you to create the Terragrunt state and logs buckets, enter `y` when prompted, then hit enter. - -Terragrunt will then run the plan. Once you have reviewed the new resources to be created, run `terragrunt run-all apply` to create the resources. - -Finally, git push your changes to the `infrastructure-live` repository you created in [Create the repositories](#create-the-repositories). +1. Navigate to `Actions` +1. Click `Infrastructure Live Bootstrap` in the sidebar on the left +1. Select the `Run Workflow` dropdown and enter the requested details. + ![Infrastructure-live Workflow Form](/img/pipelines/tutorial/infra_live_workflow.png) +1. Click `Run Workflow` to start the workflow. +1. A Pull Request(PR) will be created after the workflow completes. +1. Review the changes, follow the instructions in the PR and then merge. ## Running your first pipeline -Next you’ll create a resource in your AWS account using Pipelines and GitOps workflows. You’ll define a `terragrunt.hcl` file that creates an AWS S3 bucket in your AWS account, push your changes and create a pull request (PR) to run a `plan` action, then run an `apply` action to create the bucket by merging your PR. +In this section, you’ll create a resource in your AWS account using Pipelines and GitOps workflows by defining a `terragrunt.hcl` file that creates an AWS S3 bucket in your AWS account, pushing your changes and creating a Pull Request (PR) to run a `plan` action, then run an `apply` action to create the bucket by merging your PR. ### Adding a new S3 bucket @@ -170,13 +150,8 @@ locals { } ``` -Next, add the terragrunt code to create an S3 bucket. Copy the terragrunt code below, replacing `` with your desired bucket name. S3 bucket names need to be globally unique, so we've provided a helper script below to help generate the name of your bucket. You may name the bucket whatever you like, just make sure it’s unique. +Next, add the terragrunt code to create an S3 bucket. Copy the terragrunt code below, replacing `` with your desired bucket name. You may name the bucket whatever you like, just make sure it’s unique. -```bash -export export UNIQUE_ID=$(uuidgen | tr 'A-Z' 'a-z') -export DATE_NOW=$(date "+%F") -echo "gwp-bucket-${UNIQUE_ID}-${DATE_NOW}" -``` ```hcl title="///data-storage/s3/terragrunt.hcl" # ------------------------------------------------------------------------------------------------------ @@ -234,6 +209,6 @@ If you are not going to continue using Pipelines after this tutorial, clean up t diff --git a/docs/pipelines/security/machine-users.md b/docs/pipelines/security/machine-users.md index 576d05b049..ae7868f82d 100644 --- a/docs/pipelines/security/machine-users.md +++ b/docs/pipelines/security/machine-users.md @@ -1,49 +1,73 @@ -# Machine Users +# Machine users Gruntwork recommends using CI users in Gruntwork Pipelines, separate from human users in your organization. Using a CI user ensures that a workflow won't break due to an employee leaving your company. Further, using CI users allow you to apply granular permissions that may normally be too restrictive for a normal employee to do their daily work. +:::info + + This guide will take approximately 25 minutes to complete. + +::: + ## Creating machine users -Gruntwork recommends using two ci-users - one with the ability to open pull requests and run workflows on your behalf and another that can only read code from GitHub. Restrictive permissions are then applied to each user to limit them to only perform the required actions to accomplish their tasks. This means that in order to actually run a pipeline job, both users must be involved at separate stages. +Gruntwork Pipelines requires using two machine users; one with the ability to open pull requests and run workflows on your behalf and another that can only read code from GitHub. Restrictive permissions are then applied to each user to limit them to only perform the required actions to accomplish their tasks. This means that in order to actually run a pipeline job, both users must be involved at separate stages. We’ll refer to this user as `ci-user` and `ci-read-only-user`, but you may name them anything you like. These users **must**: - 1. Be a member of your GitHub Organization - 1. Have permission to create Pull requests within your GitHub Organization - 1. Be a member of your team in **Gruntwork**’s GitHub Organization (See [instructions on inviting a user to your team](https://docs.gruntwork.io/developer-portal/invite-team#inviting-team-members) and [linking the user’s GitHub ID to Gruntwork](https://docs.gruntwork.io/developer-portal/link-github-id)) +1. Both be members of your GitHub Organization +1. Both be members of your team in **Gruntwork**’s GitHub Organization (See [instructions on inviting a user to your team](https://docs.gruntwork.io/developer-portal/invite-team#inviting-team-members) and [linking the user’s GitHub ID to Gruntwork](https://docs.gruntwork.io/developer-portal/link-github-id)) + +## Storing secrets +During this setup, you will need to generate and securely store three GitHub tokens for two GitHub users. You will need a temporary location for these sensitive values between generating them and storing them in GitHub Actions. Do so according to your company's recommended security best practices (e.g., do not store them in Slack, a sticky note, etc., during this exercise.) + +:::note +Your organization is required to rotate the GitHub tokens and update all GitHub secrets that use them. +::: ### ci-user The `ci-user` orchestrates workflows, can open pull requests from automated code generation, and leave comments on pull requests. This user should have two GitHub Fine Grained [Personal Access Tokens (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#fine-grained-personal-access-tokens)s with the following permissions. -1. `INFRA_LIVE_ACCESS`. This token will be used to grant GitHub Actions access to clone your `infrastructure-live` repository, open PRs, and create comments & issues. +First, invite the `ci-user` to both your `infrastructure-live` and `infrastructure-pipelines` repositories with collaborator access. Then, create the following access tokens in the `ci-user`'s GitHub account: - This token **must** have: +1. `INFRA_LIVE_ACCESS`. This is a fine-grained GitHub access token and will be used to grant GitHub Actions access to clone your `infrastructure-live` repository, open PRs, and create comments & issues. - - Content (read & write) access - - Pull Requests (read & write) access + This token **must** have the following permissions to your **`infrastructure-live`** repo in your GitHub Organization: - to your **`infrastructure-live`** repo in your GitHub Organization. + - Content read & write access + - Issues read & write access + - Metadata read access + - Pull Requests read & write access ![INFRA_LIVE_ACCESS PAT Configuration](/img/pipelines/security/INFRA_LIVE_ACCESS.png) -1. `PIPELINES_DISPATCH`. This token will be used to grant GitHub Actions permission to trigger workflows in your `infrastructure-pipelines` repository. +1. `PIPELINES_DISPATCH`. This token will be used to grant GitHub Actions permission to trigger workflows in your `infrastructure-pipelines` repository to your **`infrastructure-pipelines`** repo in your GitHub Organization: This token **must** have: - - Contents (read only) access - - Actions (read & write) access - - to your **`infrastructure-pipelines`** repo in your GitHub Organization. + - Actions read & write access + - Contents read only access + - Metadata read access ![PIPELINES_DISPATCH PAT Configuration](/img/pipelines/security/PIPELINES_DISPATCH.png) +1. `PIPELINES_BOOTSTRAP`. This token will be used to grant GitHub Actions permission to create open PRs and create comments in your `infrastructure-pipelines` repository in your GitHub Organization. **This is only required during the bootstrap process of the pipelines repository and should be removed immediately after.** + + This token **must** have: + + - Contents read & write access + - Metadata read access + - Pull Requests read & write access + + ![PIPELINES_BOOTSTRAP PAT Configuration](/img/pipelines/security/PIPELINES_BOOTSTRAP.png) ### ci-read-only-user -The `ci-read-only-user` is used clone `infrastructure-live` and in terragrunt actions to access Gruntwork Library modules and your own `infrastructure-modules` repository (or repositories). The `ci-read-only-user` should have a single classic token [Personal Access Tokens (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with read permissions. Because classic PATs have coarse granularity, we recommend putting this user in a GitHub team that only has READ access to `infrastructure-live` and `infrastructure-modules` in your own GitHub Organization. By adding this user to the Gruntwork Developer portal, they will automatically gain access to the Gruntwork Library. +This user is created to pull down Terraform/OpenTofu code, but not to apply it. The `ci-read-only-user` is used clone `infrastructure-live` and in Terragrunt actions to access Gruntwork Library modules and your own `infrastructure-modules` repository or any custom module repositories that are private. + +The `ci-read-only-user` should have a single classic token [Personal Access Tokens (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#personal-access-tokens-classic) with read permissions. Classic PATs have coarse granularity, we recommend putting this user in a GitHub team that only has READ access to `infrastructure-live` and any relevant module repositories in your own GitHub Organization. By adding this user to the Gruntwork Developer portal, they will automatically gain access to the Gruntwork Library. -Create the following token for the `ci-read-only-user`: +Invite `ci-user-read-only` to your `infrastructure-live` repository with collaborator access. Create the following token for the `ci-read-only-user`: 1. `GRUNTWORK_CODE_ACCESS`. This token will be used to manage access to Gruntwork resources during GitHub Action runs. @@ -53,25 +77,39 @@ This token **must** have `repo` scopes. The expiration of this token is up to you and the security posture of your organization, Gruntwork recommend 90 days to avoid having to regularly rotate a token and secrets. +## Invite both machine users to Gruntwork +Ensure both of these machine users are members of your team in **Gruntwork**’s GitHub Organization (See [instructions on inviting a user to your team](https://docs.gruntwork.io/developer-portal/invite-team#inviting-team-members) and [linking the user’s GitHub ID to Gruntwork](https://docs.gruntwork.io/developer-portal/link-github-id)) + ## Repository Secrets Gruntwork Pipelines reads these secrets from GitHub Actions secrets created in the repo. For steps on how to create repository Actions secrets, refer to [creating secrets for a repository](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository). +### infrastructure-live + In the `infrastructure-live` repository create the following secrets: -1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated in the step above as its value. -1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated in the step above as its value. -1. `PIPELINES_DISPATCH_TOKEN`. This should be assigned the _`PIPELINES_DISPATCH`_ token generated in the step above as its value. +1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated by the `ci-read-only-user` in the [secrets section](#ci-read-only-user) as its value. +1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. +1. `PIPELINES_DISPATCH_TOKEN`. This should be assigned the _`PIPELINES_DISPATCH`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. + +### infrastructure-pipelines In the `infrastructure-pipelines` repository create the following secrets: -1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated in the step above as its value. -1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated in the step above as its value. +1. `GRUNTWORK_CODE_ACCESS_TOKEN`. This should be assigned the _`GRUNTWORK_CODE_ACCESS`_ token generated by the `ci-read-only-user` in the [secrets section](#ci-read-only-user) as its value. +1. `INFRA_LIVE_ACCESS_TOKEN`. This should be assigned the _`INFRA_LIVE_ACCESS`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. +1. `PIPELINES_BOOTSTRAP_TOKEN`. This should be assigned the _`PIPELINES_BOOTSTRAP`_ token generated by the `ci-user` in the [secrets section](#ci-user) as its value. + + :::warning + For security purposes, you should delete: + 1. The `PIPELINES_BOOTSTRAP` Personal Access Token from the `ci-user` GitHub account and + 1. The `PIPELINES_BOOTSTRAP_TOKEN` GitHub Actions secret from the `infrastructure-pipelines` repository -Your organization is required to rotate tokens and update all secrets that use them. + after the bootstrap process is complete. + ::: diff --git a/docs/pipelines/upgrading/index.md b/docs/pipelines/upgrading/index.md index 0cf8fd8159..e0089b8b98 100644 --- a/docs/pipelines/upgrading/index.md +++ b/docs/pipelines/upgrading/index.md @@ -4,7 +4,7 @@ This migration guide is intended for those running the previous version of Grunt To accomplish this task, we'll be using ECS Deploy Runner to deploy Pipelines, then using Pipelines to destroy ECS Deploy Runner. -## What's new +## What's New ECS Deploy Runner was designed as a highly secure approach to CI/CD for infrastructure. With Pipelines, we adapted the security principles built into ECS Deploy Runner to embrace modern CI system functionality, leaning heavily on GitHub Actions native functionality to create a more streamlined setup experience, and better overall UX. The major changes in Pipelines include: @@ -14,61 +14,49 @@ ECS Deploy Runner was designed as a highly secure approach to CI/CD for infrastr ## Prerequisites -- [Boilerplate](https://github.com/gruntwork-io/boilerplate#install) installed on your system (requires Gruntwork subscription) - Ability to create repositories in your GitHub Organization - Ability to add users to your GitHub Organization and the Gruntwork Developer Portal - Permissions to create secrets in GitHub repositories - -## GitHub Personal Access Token Setup - -Pipelines uses several GitHub Personal Access Tokens (PATs) across workflows, with each PAT possessing only the minimal set of permissions necessary for its specific task. To uphold the principle of least privilege for each PAT, we advise maintaining two distinct users: the pre-existing CI user and a new CI Read Only user. The CI Read Only user should have access to read your `infrastructure-live` repository, `infrastructure-modules` repository (if applicable), and the Gruntwork Library. - -### Create a CI Read Only User - -Follow the steps on [signing up a new GitHub account](https://docs.github.com/en/get-started/signing-up-for-github/signing-up-for-a-new-github-account) to create a new user. If you have an email address for your existing CI user, we recommend using plus addressing, such as `ci+read-only@your-domain.com`, as the email address for the new account. - -Add the new user to your GitHub org, adding it to a Team with read only access to your `infrastructure-live` and `infrastructure-modules` repository (if applicable). Follow the steps on [adding a user to the Gruntwork Developer portal](../../developer-portal/invite-team.md) to grant the CI Read Only user access to the Gruntwork Library. - -### Create GitHub Personal Access Tokens for GitHub Actions Workflows - -Next, create the following PATs for the new CI Read Only user. We'll use these tokens shortly. - -- `GRUNTWORK_CODE_ACCESS_TOKEN`: Classic token with READ access to repos. This should should be associated with the `CI Read Only` user created in [creating a CI Read only user](#creating-a-ci-read-only-user). -- `PIPELINES_DISPATCH_TOKEN`: Fine-grained token that only has workflow permissions to execute workflows in the `infrastructure-pipelines` repository. This token should be associated with your existing CI user. -- `INFRA_LIVE_ACCESS_TOKEN`: Fine-grained token with READ and WRITE access to your `infrastructure-live` repository. This token should be associated with your existing CI user. +- [Terragrunt](https://terragrunt.gruntwork.io/) installed on your system ## Create your infrastructure-pipelines repository -Pipelines separates code (IaC) and deployment using two different git repositories. One repository holds both the deployment code and the AWS account access, and assigns write privileges only to a select group of admins. This repository uses OpenID Connect and AWS IAM roles to generate temporary session credentials, avoiding the need to store long-lasting secrets. +Pipelines separates code (IaC) and deployment using two different GitHub repositories. One repository holds both the deployment code and the AWS account access, and assigns write privileges only to a select group of admins. This repository uses OpenID Connect and AWS IAM roles to generate temporary session credentials, avoiding the need to store long-lasting secrets. We strongly recommend naming this repository `infrastructure-pipelines`. All code generated assumes the repository will be located at `/infrastructure-pipelines`, so using a different name will require manual work on your behalf. -### Create the repo +### Initialize infrastructure-pipelines repository + +Create a new Pipelines repository from the template below: -Navigate to the repositories tab of your organization or personal GitHub account in your web browser. +- [gruntwork-pipelines-standalone-template](https://github.com/gruntwork-io/gruntwork-pipelines-standalone-template) -1. Click the **New Repository** button. -1. In the **Repository name** field enter `infrastructure-pipelines` -1. Select the **Private** option -1. Do not initialize the repo with a README, gitignore, or license +1. Click `Use this template` +1. Click `Create a new repository` +1. Select your preferred organization from the owner dropdown +1. Name the repo anything you like and make note of the name. +1. Select the `Private` radio button +1. Click `Create Repository` + +![GitHub form for creating a new repository](/img/pipelines/tutorial/create_new_repo_form.png) :::info For a simple proof of concept, the default repo configuration will suffice. Before using this repository in a production environment, we recommend setting up a [branch protection rule](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) for your `main` branch. At a minimum, we recommend enabling requiring a pull request before merging with at least one reviewer required. ::: -### Use boilerplate to generate code +### Create Temporary Bootstrap Token +Create a temporary PAT with your own GitHub account so it has the ability to open PRs In your `infrastructure-pipelines` repository by using a classic PAT with `repo` and `workflow` access. Once you've generated this token, create a secret called `CUSTOMER_BOOTSTRAP_ACCESS_TOKEN` in `infrastructure-pipelines` and place this value in it. -Next, generate the `infrastructure-pipelines` repository code using [Boilerplate](https://github.com/gruntwork-io/boilerplate), a tool authored by Gruntwork for generating files and folders specially for DevOps use cases. Clone the newly created `infrastructure-pipelines` repository, `cd` into the repo directory, then use the following command replacing `` with the name of your GitHub organization and `` with the name of your infrastructure-live repository. +### Bootstrap the Pipelines Repo -```bash -boilerplate --template-url "git@github.com:gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/infrastructure-pipelines?ref=v1.0.0" \ - --output-folder . \ - --var InfraLiveRepoName="" \ - --var GithubOrg="" \ - --non-interactive -``` +In the newly created repository, go to Actions and run the **Infrastructure Pipelines Bootstrap** and enter in the name of your `infrastructure-live` repository. This will open up a pull request that you can merge. + +### Delete Temporary Bootstrap Token +Once the Pipelines repository is bootstrapped successfully, you can delete the `CUSTOMER_BOOTSTRAP_ACCESS_TOKEN`. + +## GitHub CI Machine Users and Secrets Setup -Push your changes up to the remote repository. +Pipelines utilizes two machine users, one for read-only and another elevated operations. Follow this [guide](../security/machine-users) to create them and and set up the correct access tokens in each of the repositories. ## Create AWS IAM roles for Pipelines @@ -76,50 +64,20 @@ In Pipelines, each of your accounts must have an AWS IAM role. This role shares For more information see [security hardening with OpenID Connect](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect) and [OpenID Connect in AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). -### Generate the code using boilerplate for each account +### Generate the Pipelines IAM code using GitHub Actions for each account -To simplify the migration process, we have developed a template that creates all code and files required to generate the IAM role, OpenID Connect identity provider, and IAM policies. +In `infrastructure-live`, open a pull request and copy this [GitHub Actions workflow file](https://github.com/gruntwork-io/gruntwork-infra-live-standalone-template/blob/main/.github/workflows/bootstrap.yml) into your `.github/workflows` directory. -Use the following command, replacing the values wrapped in `<>` with real values for your organization. The `AwsAccountName` variable should be the name of the top level directory that represents the AWS account that will use Pipelines (e.g, `dev`). You will need to run this command for each account that you would like to upgrade to Pipelines. - -Now `cd` into your `infrastructure-live` repository and run the following command: - -```bash -boilerplate --template-url "git@github.com:gruntwork-io/terraform-aws-architecture-catalog.git//blueprints/components/pipelines-edr-migration" \ - --output-folder . \ - --var InfraPipelinesRepoName="" \ - --var GithubOrg="" \ - --var AwsAccountName="" \ - --var RequestingTeamName="" \ - --var AwsAccountName="" \ - --var OrgNamePrefix="The name prefix to use for creating resources" \ - --non-interactive -``` - -Create a branch, commit your changes, and push your branch to the remote repository. Then create a pull request targeting your default branch (e.g., `main`). +Once that has been merged into `main`, run the **Infrastructure Live Bootstrap** for each account to generate the Terraform needed to create the IAM Roles and the GitHub OIDC connection to securely do deploys. Each workflow run will open a PR that needs to be merged. :::tip We recommend trying out Pipelines in non-production environments first, before rolling out to production environments. ::: -### Planning and applying - -Create a new branch for your changes, commit your changes to your branch, then push your branch. Next, create a PR to merge your branch into `main` (the default branch in your repository). The ECS Deploy Runner will detect the change and run a `plan` to create the IAM Roles, OpenID Connect providers, and all requires IAM policies. Review the `plan` output and confirm it looks as expected. +The ECS Deploy Runner will detect the change and run a `plan` to create the IAM Roles, OpenID Connect providers, and all requires IAM policies. Review the `plan` output and confirm it looks as expected. Once you have reviewed the `plan` output and gotten any necessary approvals, merge the PR. The EDR version of Pipelines will run `apply` to create the resources. -## Set up secrets - -Pipelines requires the use of GitHub personal access tokens (PAT) to allow workflows in the `infrastructure-live` to run workflows in the `infrastructure-pipelines` repository and workflows in the `infrastructure-pipelines` repository access to the code in `infrastructure-live` as well as repos for the Gruntwork Library. This section will guide you in establishing secrets in both repositories and comprehending the security implications of the multi-PAT configuration. - -### Infrastructure live - -Navigate to your `infrastructure-live` repository (it may be named `-infrastructure-live`). Select the **Settings** tab, select the **Secrets and variables** drop down on the left side panel, then select **Actions**. Create three secrets named `GRUNTWORK_CODE_ACCESS_TOKEN`, `PIPELINES_DISPATCH_TOKEN`, and `INFRA_LIVE_ACCESS_TOKEN`. Use the corresponding value from the tokens you created in [GitHub Personal Access Token Setup](#github-personal-access-token-setup) for the secret value. - -### Infrastructure pipelines - -Navigate to the `infrastructure-pipelines` repository. Select the **Settings** tab, select the **Secrets and variables** drop down on the left side panel, then select **Actions**. Create two secrets named `INFRA_LIVE_ACCESS_TOKEN` and `GRUNTWORK_CODE_ACCESS_TOKEN`. Use the corresponding value from the tokens you created in [GitHub Personal Access Token Setup](#github-personal-access-token-setup) for the secret value. - ## Updating your accounts file As part of a separate product improvement initiative, we have transitioned from the `accounts.json` file to `accounts.yml`. This change not only enhances readability but is also functionally better, with Terraform's built-in YAML formatter better meeting our requirement for regenerating a formatted file to facilitate the addition of new accounts through GitOps workflows. It's important to note that there is no functional distinction between these two file types in either the EDR version of Pipelines or Pipelines. @@ -181,6 +139,6 @@ Congratulations! If you have followed this guide, you will be deploying infrastr diff --git a/static/img/pipelines/security/PIPELINES_BOOTSTRAP.png b/static/img/pipelines/security/PIPELINES_BOOTSTRAP.png new file mode 100644 index 0000000000..28611e0ed7 Binary files /dev/null and b/static/img/pipelines/security/PIPELINES_BOOTSTRAP.png differ diff --git a/static/img/pipelines/tutorial/create_new_repo_form.png b/static/img/pipelines/tutorial/create_new_repo_form.png index f1fbb1900a..18167f08b0 100644 Binary files a/static/img/pipelines/tutorial/create_new_repo_form.png and b/static/img/pipelines/tutorial/create_new_repo_form.png differ diff --git a/static/img/pipelines/tutorial/infra_live_workflow.png b/static/img/pipelines/tutorial/infra_live_workflow.png new file mode 100644 index 0000000000..f03a63712f Binary files /dev/null and b/static/img/pipelines/tutorial/infra_live_workflow.png differ diff --git a/static/img/pipelines/tutorial/pipelines_run_workflow.png b/static/img/pipelines/tutorial/pipelines_run_workflow.png new file mode 100644 index 0000000000..660a7ccf57 Binary files /dev/null and b/static/img/pipelines/tutorial/pipelines_run_workflow.png differ