Updates to Pipeline Documentation (#256)

### Feature or Bugfix - Feature ### Detail - Adding updated documentation for data all pipelines as of v1m3m0 By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
data-dot-all · Dec 16, 2022 · 20458ef · 20458ef
1 parent 0e6c70d
commit 20458ef
Show file tree

Hide file tree

Showing 5 changed files with 50 additions and 30 deletions.
diff --git a/backend/dataall/cdkproxy/cdkpipeline/cdk_pipeline.py b/backend/dataall/cdkproxy/cdkpipeline/cdk_pipeline.py
@@ -16,12 +16,11 @@ class CDKPipelineStack:
     """
     Create a stack that contains CDK Continuous Integration and Delivery (CI/CD) pipeline.
 
-    The pipeline is based on CodePipeline pipelines
+    The pipeline is based on AWS DDK CICD CodePipeline pipelines
 
     - Defaults for source/synth - CodeCommit & cdk synth
     - blueprint with DDK application code added in the CodeCommit repository <https://github.com/awslabs/aws-ddk>
     - ability to define development stages: dev, test, prod
-    - ability to select gitflow or trunk-based as development strategy
     - Ability to connect to private artifactory to pull artifacts from at synth
     - Security best practices - ensures pipeline buckets block non-SSL, and are KMS-encrypted with rotated keys
     - data.all metadata as environment variables accesible at synth

diff --git a/documentation/userguide/docs/pictures/pipelines/github_template_create.png b/documentation/userguide/docs/pictures/pipelines/github_template_create.png
diff --git a/documentation/userguide/docs/pictures/pipelines/pip_cp_init.png b/documentation/userguide/docs/pictures/pipelines/pip_cp_init.png
diff --git a/documentation/userguide/docs/pictures/pipelines/pip_edit_form.png b/documentation/userguide/docs/pictures/pipelines/pip_edit_form.png
diff --git a/documentation/userguide/docs/pipelines.md b/documentation/userguide/docs/pipelines.md
@@ -38,28 +38,29 @@ data.all pipelines are created from the UI, under Pipelines. We need to fill the
 - Name, Description and tags
 - CICD Environment: AWS account and region where the CICD resources will be deployed.
 - Team, this is the Admin team of the pipeline. It belongs to the specified CICD Environment where the pipeline is defined as IaC
-- Development strategy: 
+- CICD strategy: This is the development strategy that determines the type of CICD Pipeline that is created by data.all. Currently the following 4 types are supported depending on your use case:
+    1. [**CDK Pipelines - Trunk-based**](#CDK-Pipelines-Overview) : A CICD pipeline based on [CDK Pipelines library](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html). It defines a DDK Core construct which deploys Continuous Integration and Delivery for your app. Specifically, it provisions a stack containing a self-mutating CDK code pipeline to deploy one or more copies of your CDK applications using CloudFormation with a minimal amount of effort on your part.
+    2. [**CodePipeline - Trunk-based**](#CodePipeline-pipelines---Trunk-based-or-GitFlow-Overview) : A CICD pipeline similar to CDK Pipelines and with a trunk-based approach but is not self-mutating.
+    3. [**CodePipeline - Gitflow**](#CodePipeline-pipelines---Trunk-based-or-GitFlow-Overview): A Gitflow branching strategy where each branch of the source repository has a corresponding CICD Pipeline that deploys resources for that branches environment.
+    4. [**GitHub Template**](#Github-Template-Pipelines-Overview) : This is a Bring-Your-Own-Template approach where users can specify they git clone path and deploy their own pipelines and IaC rather than using one of the previous 3 strategies. 
 
 Finally, we need to add **Development environments**. These are the AWS accounts and regions where the infrastructure defined in the CICD pipeline
 is deployed. 
 
 !!! warning "environment ID = data.all environment stage"
-      When creating the pipeline and adding development environments, you define the stage of the environment. The bootstrap `e` parameter needs to match the one that you define in the data.all UI.
+      When creating the pipeline and adding development environments, you define the stage of the environment. The ddk bootstrap `-e` parameter needs to match the one that you define in the data.all UI.
       In our example, we bootstraped with the parameters "dev" and "prod" and then we defined the stages as "dev" and "prod" correspondingly.
 
 
 ![create_pipeline](pictures/pipelines/pip_create_form.png#zoom#shadow)
 
-### CDK pipelines - Trunk-based
+---
 
-This CodePipeline pipeline is based on the [CDK Pipelines library](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html).
-As stated in the documentation, CDK Pipelines is an opinionated construct library. It is purpose-built to deploy one or more copies of your CDK applications using CloudFormation with a minimal amount of effort on your part.
+### CDK Pipelines Overview
 
 #### CodeCommit repository
 
-When a pipeline is created, a CloudFormation stack is deployed in the CICD environment AWS account. 
-It contains an AWS CodeCommit repository with the code of an AWS DDK application set up for a multi-account deployment,
-as explained in its [documentation](https://awslabs.github.io/aws-ddk/release/latest/how-to/multi-account-deployment.html).
+When a pipeline is created, an AWS CodeCommit repository with the code of an AWS DDK application is created in the CICD environment AWS account. It contains an  set up for a multi-account deployment, as explained in its [documentation](https://awslabs.github.io/aws-ddk/release/latest/how-to/multi-account-deployment.html).
 
 
 In the deployed repository, data.all pushes a `ddk.json` file with the details of the selected development environments:
@@ -88,7 +89,7 @@ In the deployed repository, data.all pushes a `ddk.json` file with the details o
     }
 }
 ```
-In addition, the `app.py` file is also written accordingly to the development environments selected in data.all UI.
+In addition, the `app.py` file is also written accordingly to the development environments selected in data.all UI. It will look similar to the following:
 
 ```
 
@@ -130,36 +131,29 @@ app.synth()
 
 ```
 #### CICD deployment
-data.all backend performs the first deployment of the CICD stack defined in the CodeCommit repository. The result is a
-CloudFormation template deploying a CICD pipeline having the aforementioned CodeCommit repository as source.
-This CodePipeline pipeline is based on the [CDK Pipelines library](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html). 
+data.all backend performs the first deployment of the CICD stack defined in the CodeCommit repository. The result is a CloudFormation template deploying a CICD pipeline having the aforementioned CodeCommit repository as source. This CodePipeline pipeline is based on the [CDK Pipelines library](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html). 
 
 ![create_pipeline](pictures/pipelines/pip_cdk_trunk.png#zoom#shadow)
 
-### CodePipeline pipelines - Trunk-based or GitFlow
+---
 
-For cases in which we need more control over the CICD pipeline, instead of using [CDK Pipelines library](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html) we can
-use [aws-codepipeline](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_codepipeline-readme.html) construct library. 
+### CodePipeline pipelines - Trunk-based or GitFlow Overview
+
+For cases in which we do not want to use [CDK Pipelines library](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.pipelines/README.html) we can instead
+use CodePipeline CICD Strategy which leverages the [aws-codepipeline](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_codepipeline-readme.html) construct library. 
 
 
 #### CodeCommit repository and CICD deployment
 When a pipeline is created, a CloudFormation stack is deployed in the CICD environment AWS account. It contains:
 
-- an AWS CodeCommit repository with the code of an AWS DDK application where we made some modifications to allow cross-account deployments.
+- an AWS CodeCommit repository with the code of an AWS DDK application (by running `ddk init`) with some modifications to allow cross-account deployments.
 - CICD CodePipeline(s) pipeline that deploy(s) the application
 
-In the first run of the pipeline we will perform some initialization actions from the pipeline itself (you don't need to do anything). In short, we initialize the DDK application by running `ddk init` 
-and we push the code back to our repository.
-
-This is the original repository:
-
-![created_pipeline](pictures/pipelines/pip_cp_init.png#zoom#shadow)
-
-This is the repository once it has been initialized in the commit "First Commit from CodeBuild - DDK application":
+The repository structure will look similar to:
 
 ![created_pipeline](pictures/pipelines/pip_cp_init2.png#zoom#shadow)
 
-We added the `Multiaccount` configuration class that allows us to define the deployment environment based on the `ddk.json`. 
+The added `Multiaccount` configuration class allows us to define the deployment environment based on the `ddk.json`. 
 Go ahead and customize this configuration further, for example you can set additional `env_vars`.
 
 Trunk-based pipelines append one stage after the other and read from the main branch of our repository:
@@ -174,6 +168,29 @@ The `dev` pipeline reads from the `dev` branch of the repository:
 
 ![created_pipeline](pictures/pipelines/pip_cp_gitflow2.png#zoom#shadow)
 
+--- 
+### Github Template Pipelines Overview
+
+This pipeline strategy takes a pre-defined IaC CDK Application that exists in a github repository and deploys the pipeline to be managed by data all. An AWS CodeCommit repository with the code of the github repository is created in the CICD environment AWS account.
+
+**NOTE: You may have to specify a access token in the HTTPS Clone Path of the Github Repository if the repository is private**
+
+data.all performs the inital deployment of this pipeline by running `cdk deploy` for the code now existing in AWS CodeCommit in the CICD environment. Adding development environments here is on the responsibility of the pipeline creator to align with the deployment environments specified in the cloned repository.
+
+![created_pipeline](pictures/pipelines/github_template_create.png#zoom#shadow)
+
+## Editing a Data All Pipeline
+
+For users who would like to promote their pipeline deployments to new environments managed by data all, you can do so by first bootstrapping the new environment(s) to be deployed to (as mentioned in the [Pre-requisites](#Pre-requisites)) and then adding and/or editing the development environments.
+
+![edit_pipeline](pictures/pipelines/pip_edit_form.png#zoom#shadow)
+
+Based on pipeline use case, editing a data all pipeline's development environments will:
+- **CDK Pipelines**: On update, the `ddk.json` and `app.py` will be edited to update the new development environment information. The self-mutating, CICD Pipeline will trigger and deploy to the new environments based on the source CodeCommit repository changes.
+- **CodePipelines - Trunk-based**: On update the `ddk.json` will be edited. A new `cdk deploy` will run to update the CICD CloudFormation Stack for the AWS CodePipeline to add the new stages required for the additional environment deployment(s) (as well as manual approval steps between stages in the code pipeline). You will see these updates to the CICD stack in CloudFormaiton of the CICD environment.
+- **CodePipelines - Gitflow**: On update the `ddk.json` will be edited. A new `cdk deploy` will run to update the CICD CloudFormation Stack to add the new AWS CodePipelines required for the additional environment deployment(s). You will see these updates to the CICD stack in CloudFormaiton of the CICD environment.
+- **Github Template Pipelines**: Editing development environments **will NOT** re-deploy the application or update the CodeCommit repository. Editing of template pipeline's development environment(s) is the responsibility of the pipeline creator for proper data all pipeline management.
+
 ## Which development strategy should I choose?
 
 **CDK pipelines - Trunk-based**
@@ -187,11 +204,15 @@ The `dev` pipeline reads from the `dev` branch of the repository:
 2. Developers working on the pipeline cannot modify the CICD pipeline
 3. Cross-account deployments require specific definition of the environment in the code.
 
-**Summary**
+**Github Template Pipelines**
+
+1. The aforementioned pipeline strategies do not align with your desired pipeline architecture
+2. You already have pipelines IaC written in AWS CDK and ready to be deployed rather than creating pipeline(s) and developing from scratch
 
-CDK pipelines are recommended for flexibility and for a robust cross-account application deployment, 
-whereas CodePipeline pipelines are recommended if you need to provide an immutable pipeline architecture or if you want to implement a GitFlow strategy.
 
+**Summary**
+
+CDK pipelines are recommended for flexibility and for a robust cross-account application deployment, whereas CodePipeline pipelines are recommended if you need to provide an immutable pipeline architecture or if you want to implement a GitFlow strategy.
 
 
 ## Cloning the repository