CI/CD Pipeline Project Using AWS Native SDLC Automation Tools

Project ToolBox 🧰

• CodeCommit CodeCommit is a secure, highly scalable, fully managed source control service that hosts private Git repositories.
• CodeBuild CodeBuild is a fully managed continuous integration service that compiles source code, runs tests, and produces ready-to-deploy software packages.
• CodeArtifact CodeArtifact allows you to store artifacts using popular package managers and build tools like Maven, Gradle, npm, Yarn, Twine, pip, NuGet, and SwiftPM.
• CodeDeploy CodeDeploy is a fully managed deployment service that automates software deployments to various compute services, such as EC2, ECS, Lambda, and on-premises.
• CodePipeline CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates.
• Amazon S3 Amazon S3 is an object storage service offering industry-leading scalability, data availability, security, and performance.
• EC2 EC2 allows users to rent virtual computers (EC2) to run their own workloads and applications.
• CloudWatch Metrics Amazon CloudWatch can load all the metrics in your account (both AWS resource metrics and application metrics that you provide) for search, graphing, and alarms.
• CloudWatch Logs You can use CloudWatch Logs to monitor applications and systems using log data.
• Amazon SNS Amazon SNS sends notifications two ways and provides high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven serverless applications.
• SonarCloud SonarCloud is a cloud-based code analysis service designed to detect coding issues in 26 different programming languages.

NOTE: a) Navigate to a Region on AWS that has all the AWS Code services (CodeCommit, CodeBuild, CodeArtifact, CodeDeploy and CodePipeline) b) You must Login as an IAM User before you can complete the below steps successfully (Login with a user that has Administrator Privileges)

1) Create a CodeCommit Project Repository

1.1) Navigate to CodeCommit to create a Project Repository

• Click on Create Repository
• Name your repository AWS-Native-CICD-Pipeline-Project
• Click Create

1.2) Configure SSH Connection On Local MacOs or Windows With CodeCommit

• Follow the steps provided in the Runbook below
• Runbook: https://docs.google.com/document/d/1vDu6TBcIrh2NbUtv5eayur1Cdo3Z8X5YSZ9t58ItlXo/edit?usp=sharing

1.3) Download the Project Zip Code From The Below Repository Link

• Project Code: https://github.com/jacodetunde/AWS-Native-CICD-Pipeline.git
• Unzip and Copy everything to the Code Commit Repository you just cloned
• Push the Code Upstream to Your CodeCommit Project Repository and Confirm you have everything in the Repository
• NOTE: Use the same Git commands you have always used to Push code to GitHub

2) Create A CodeBuild IAM Profile/Role

• Create a CodeBuild Service Role in IAM with Administrator Privilege
• Navigate to IAM
  • Click on Roles
  • Click on Create Role
  • Select Service Role
  • Search/Select CodeBuild
    • Click on Next
    • Attach Policy: AdministratorAccess
    • Click Next
    • Role Name: AWS-CodeBuild-Admin-Role
    • Click Create

3) Create A CodeDeploy IAM Profile/Role

• Create a CodeBuild Service Role in IAM with Administrator Privilege
• Navigate to IAM
  • Click on Roles
  • Click on Create Role
  • Select Service Role
  • Search/Select CodeDeploy
    • Click on Next
    • Attach Policy: AdministratorAccess
    • Click Next
    • Role Name: AWS-CodeDeploy-Deployment-Role
    • Click Create

4) Create An S3 Bucket Where The Build Artifact Will Be Stored

• Navigate to Amazon S3
• Click Create Bucket
  • Name: java-webapp-project-artifact-YOUR_ACCOUNT_ID
  • Region: Select Your working Region
  • Click: CREATE Bucket

5) Sign Up For SonarCloud Account

A) Sign up for SonarCloud using this URL: https://sonarcloud.io or https://sonarcloud.io/login

• Click on SIGN UP
• Click Sign Up With GITHUB
• Sign in with your “GitHub” Account
• Click on “Authorize SonarCloud”
  • NOTE: Once you Authorize SonarCloud, It’ll take you directly to Dashboard (Similar to Traditional SonarQube Server Dashboard but this is cloud based)

B) We have to Generate a Token which CodeBuild will use during the Maven Execution

• Click on your Profile at the Top right and click on My Account
• Click on Security
• Generate Tokens (Provide a Name): aws-native-cicd-pipeline-project
• Copy the Token and Save it somewhere (on your NOTEPAD)

C) Click on the “+” symbol at the Top Right

• Click on Analyze new project
• Click on create a project manually.
  • Click on Create another organization
    • Name (Must be Unique): yourfirst-or-lastname-aws-devops-org
    • Choose a plan: Select “Free Plan”
    • Click on Create Organization

D) Create a SonarCloud Project

• Click on Create Project
  • Project Key: aws-native-cicd-pipeline-project
  • Display name: This Will Populate Automatically
  • Public/Private: Public (because we’re using a free account)
  • Click on Next
  • Select Previous version
  • Click on Create Project NOTE **Save your Project Name as well on Notepad, Save your Organization name and the Sonarcloud url (https://sonarcloud.io). Make sure your Token has been saved also.

6) Create & Configure CodeArtifact Repository to Store and Manage All Application Maven Dependencies.

A) Create CodeArtifact Project Repository

• Navigate to AWS CodeArtifact
• Click on Repository
  • Click Create Repositoy
  • Repository Name: java-webapp-maven-repo
  • Public Upstream Repository: Select maven-central-store
  • Click on Next
    • AWS Account: Select This AWS Account
    • Domain Name: java-webapp-maven-repo
    • Click on Next
    • Click Create Repository
  • NOTE: Verify and Confirm both the Repository and the Repository Domain were created successfully

B) Configure Your CodeArtifact Project Repository With Maven POM.xml and Settings.xml

• Click on Repositories if you’ve not already
  • Click on maven-central-store
  • Click on View Connection Instructions
    • Step 1: Choose a package manager client: "Select (on the drop down):" mvn
    • Step 3: COPY and Run The export Command on your Local Terminal where awscli is installed
      • NOTE:NOTE:NOTE:NOTE!!
      • The command will look like this
      • BUT COPY YOUR OWN
      • Make sure your AWSCLI is configured (with a user with "Admin Priviledges")

      export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain java-webapp-maven-repo --domain-owner 213424289791 --region us-east-1 --query authorizationToken --output text`

      • Also RUN: echo $CODEARTIFACT_AUTH_TOKEN
      • NOTE:NOTE:
        • Copy the CODEARTIFACT_AUTH_TOKEN Encrypted Credential and SAVE on your NOTEPAD/Somewhere
        • We’re going to store this Token in SSM Parameter Store from where our CodeBuild Job is going pick it up

B.1) Update the Settings.xml File With CodeArtifact Repository Configurations

• Still on “View Connection Instructions” in maven-central-store
• Under Step 5:
  • COPY the Repository id and Paste it in the settings.xml file on line 29 at the time of this
  • COPY the Repository url and Paste on Line 18 and Line 30 in the settings.xml at the time of this
  • SAVE the changes made in the file

B.2) Update the POM.xml File With CodeArtifact Repository Configurations

• Still on “View Connection Instructions” in maven-central-store
• Under Step 5:
  • COPY the Repository id and Paste it in the POM.xml file on Line 75 at the time of this
  • COPY the Repository url and Paste on Line 76 in the POM.xml at the time of this
  • SAVE the changes made in the file
  • COMMIT the changes and PUSH to UpStream to CodeCommit

7) Store Your AWS CodeArtifact & SonarCloud Project Parameters/Values In SSM Parameter Store

• Navigate to SSM

• NOTE!! Make sure you create the parameters in the same Region as the bucket (same for all project resources)

• a) Click on Parameter Store

  • Click on Create Parameter
  • Name: CODEARTIFACT_AUTH_TOKEN
  • Type: Select Secure/String
  • Value: provide your CodeArtifact Token the one you copied when you ran echo $CODEARTIFACT_AUTH_TOKEN

• a) Click on Parameter Store

  • Click on Create Parameter
  • Name: Organization
  • Type: Select String
  • Value: provide your SnarCloud Org name

• b) Click on Parameter Store

  • Name: HOST
  • Type: Select String
  • Value: https://sonarcloud.io

• c) Click on Parameter Store

  • Name: Project
  • Type: Select String
  • Value: provide your SonaCloud Project name

• d) Click on Parameter Store

  • Name: sonartoken
  • Type: Select Secure/String
  • Value: provide your SonaCloud Project Token NOTE: Confirm that these same parameter names exist in your sonacloud_buildspec.yaml configuration.

7.1) Confirm That Have All Required Parameters Created With Their Respective Values

8) Create The Project Build Job in CodeBuild

• Navigate To The AWS CodeBuild Service
• Click on Create Build Project
  • Project Name: Java-Webapp-CB-Build-Job
  • Source Provider: Select AWS CodeCommit
  • Repository: Select AWS-Native-CICD-Pipeline-Project
  • Branch: master
  • Operating System: Ubuntu
  • Runtime: Standard
  • Image: MUST USE (aws/codebuild/standard:5.0) or else it’ll BREAK
  • Image version: Select Always use the latest for this runtime version
  • Environment type: Select Linux EC2
  • Service Role: Existing Service Role
    • Role name: Select AWS-CodeBuild-Admin-Role
    • Allow AWS CodeBuild to modify this service role so it can be used with this build project: Disable/Uncheck
  • Build Specifications: Pass buildspecs/buildspec.yml
  • Artifacts:
    • Type: Select Amazon S3
    • Bucket name: Select your bucket java-webapp-project-artifact-YOUR_ACCOUNT_ID
    • Artifacts packaging: Select Zip
  • Logs
    • CloudWatch Logs: Enable
    • Group Name: Java-Webapp-CodeBuild-Project-Logs
    • Stream name: Java-Webapp-CodeBuild-Build-Logs
  • CLICK: Click CREATE BUILD PROJECT

9) Create The SonarCloud Code Analysis Job in CodeBuild

• Navigate To The AWS CodeBuild Service
• Click on Create Build Project
  • Project Name: Java-Webapp-CB-SonarCloud-Job
  • Source Provider: Select AWS CodeCommit
  • Repository: Select AWS-Native-CICD-Pipeline-Project
  • Branch: master
  • Operating System: Ubuntu
  • Runtime: Standard
  • Image: MUST USE (aws/codebuild/standard:5.0) or else it’ll BREAK
  • Image version: Select Always use the latest for this runtime version
  • Environment type: Select Linux EC2
  • Service Role: Existing Service Role
    • Role name: Select AWS-CodeBuild-Admin-Role
    • Allow AWS CodeBuild to modify this service role so it can be used with this build project: Disable/Uncheck
  • Build Specifications: Pass buildspecs/sonarcloud_buildspec.yml
  • Artifacts:
    • Type: Select No Artifact
  • Logs
    • CloudWatch Logs: Enable
    • Group Name: Java-Webapp-CodeBuild-Project-Logs
    • Stream name: Java-Webapp-CodeBuild-SonarCloud-Logs
  • CLICK: Click CREATE BUILD PROJECT

10) Create Staging Deployment Area/Environment

• Navigate to EC2
• Click Launch Instances

  • Name: Stage-Env

  • Click Add additional tags

    • Click Add new tag
      • Key: Env
      • Value: Stage

  • Number of Instances: 1

  • AMI: Amazon Linux 2 (HVM)

  • Instance type: t2.micro

  • Key pair: Select an existing Key or Create New

  • Network Settings:

    • VPC: Default or a network that has Internet access
    • Auto-assign public IP: Enable
    • Firewall (security groups): Open the following Ports
      • Name: Tomcat-App-SG
      • Description: Tomcat-App-SG
      • Open Port 8080 to 0.0.0.0/0
      • Open Port 22 to eith your Network or Internet

  • Edvance Details:

    • IAM instance profile: Select an EC2 Admin Role
      • NOTE: If you do not have one, please go ahead and create before creating instance
      • NOTE: If Not Yours Will Break
    • User data:

    #!/bin/bash
    sudo yum update
    sudo yum install ruby -y
    sudo yum install wget -y
    cd /home/ec2-user
    wget https://aws-codedeploy-us-west-2.s3.us-west-2.amazonaws.com/latest/install
    chmod +x ./install
    sudo ./install auto
    sudo service codedeploy-agent status

    • Click Launch Instance

11) Create Production Deployment Area/Environment

• Navigate to EC2
• Click Launch Instances

  • Name: Prod-Env

  • Click Add additional tags

    • Click Add new tag
      • Key: Env
      • Value: Prod

  • Number of Instances: 1

  • AMI: Amazon Linux 2 (HVM)

  • Instance type: t2.micro

  • Key pair: Select an existing Key or Create New

  • Network Settings:

    • VPC: Default or a network that has Internet access
    • Auto-assign public IP: Enable
    • Firewall (security groups): Open the following Ports
      • Click on Select existing security group
      • Security group: Select Tomcat-App-SG

  • Edvance Details:

    • IAM instance profile: Select an EC2 Admin Role
      • NOTE: If you do not have one, please go ahead and create before creating instance
      • NOTE: If Not Yours Will Break
    • User data:

    #!/bin/bash
    sudo yum update
    sudo yum install ruby -y
    sudo yum install wget -y
    cd /home/ec2-user
    wget https://aws-codedeploy-us-west-2.s3.us-west-2.amazonaws.com/latest/install
    chmod +x ./install
    sudo ./install auto
    sudo service codedeploy-agent status

    • Click Launch Instance

11.1) Confirm that you have both the Stage and Prod Environments

12) Create CodeDeploy Application

• Navigate to CodeDeploy
• Click on Applications
  • Click Create Application
    • Name: Java-Webapp-CodeDeploy-Application
    • Compute Platform: EC2/On-premises
    • Click Create Application

13) Create A CodeDeploy Deployment Group To Deploy Staging Env

• Navigate to CodeDeploy
• Click on Applications
  • Click on Java-Webapp-CodeDeploy-Application
  • Click on Create deployment group
    • Deployment group name: Java-Webapp-CodeDeploy-Stage-DG
    • Service role: AWS-CodeDeploy-Deployment-Role
    • Deployment type: Select In-place
    • Environment configuration: Select Amazon EC2 instances
      • Key: Env
      • Value: Stage
    • Agent configuration with AWS Systems Manager: Now and schedule updates
      • Basic Scheduler
    • Deployment settings: Select CodeDeployDefault.AllAtOnce
      • NOTE: CONFIRM THAT YOU SELECTED CodeDeployDefault.AllAtOnce IF Not, yours will break.
    • Load balancer: Uncheck the box to Disable
    • Click Create deployment group

14) Create A CodeDeploy Deployment Group For The Production Env

• Navigate to CodeDeploy
• Click on Applications
  • Click on Java-Webapp-CodeDeploy-Application
  • Click on Create deployment group
    • Deployment group name: Java-Webapp-CodeDeploy-Prod-DG
    • Service role: AWS-CodeDeploy-Deployment-Role
    • Deployment type: Select In-place
    • Environment configuration: Select Amazon EC2 instances
      • Key: Env
      • Value: Prod
    • Agent configuration with AWS Systems Manager: Now and schedule updates
      • Basic Scheduler
    • Deployment settings: Select CodeDeployDefault.AllAtOnce
      • NOTE: CONFIRM THAT YOU SELECTED CodeDeployDefault.AllAtOnce IF Not, yours will break.
    • Load balancer: Uncheck the box to Disable
    • Click Create deployment group

15) Create The CI/CD Automation Pipeline With CodePipeline

• Navigate to CodePipeline
• Click on Create Pipeline

  • Name: AWS-Native-Java-Webapp-CICD-Pipeline-Automation

  • Service role: New service role

    • Role name: The name will populate automatically

  • Allow AWS CodePipeline to create a service role so it can be used with this new pipeline: Chech box to Enable

    • Click Next

  • SOURCE PROVIDER

  • Source provider: Select AWS CodeCommit

    • Repository name: AWS-Native-CICD-Pipeline-Project
    • Branch name: Master
    • Change detection options: Amazon CloudWatch Events (recommended)
    • Output artifact format: CodePipeline default
    • Click Next

  • BUILD PROVIDER

  • Build provider: AWS CodeBuild

    • Region: Your region will populate
    • Project name: Java-Webapp-CB-Build-Job
    • Build type: Single build
    • Click Next

  • DEPLOY PROVIDER

  • Deploy provider: AWS CodeDeploy

    • Region: Your region will populate
    • Application name: Java-Webapp-CodeDeploy-Application
    • Deployment group: Java-Webapp-CodeDeploy-Stage-DG
    • Click Next

  • Click CREATE PIPELINE

  • NOTE: Once you create the pipeline, it'll start Running Immediate. Just wait for all the various stages to complete

  • NOTE2: The Deployment Will Only Take Place In The Staging Environment (With Continuous Deployment)

16) Add The SAST Test Stage With SonarCloud

• Click on Edit to add the following Pipeline Stages;
  • The Testing Stage
  • The Manual Approval Stage
  • The Prod Deployment Stage
• Click on Add stage
• NOTE: Make sure to add this Stage in between the Build and Desploy Stage
• Stage name: SAST-Test-SonarCloud
• Click on Add action group

  • Action name: SAST-Test-SonarCloud

  • Action provider: AWS CodeBuild

  • Region: Select your project region

  • Input artifact: SourceArtifact

  • Project name: Select Your SonarCloud CodeBuild Job/Project

  • Build type: Single build

  • Click Done

  • Click on Done again

17) Add The Manual Approval Stage (To Achieve Continuous Delivery To Production)

• Click on Add stage

• Stage name: Manual-Approval

• Click on Add action group

  • Action name: Manual-Approval

  • Action provider: Manual approval

  • Click Done

  • Click on Done again

18) Add The Deploy To Production Stage With CodeDeploy

• Click on Add stage
• Stage name: Deploy-Prod
• Click on Add action group

  • Action name: Deploy-Prod

  • Action provider: AWS CodeDeploy

  • Region: Select your project region

  • Input artifact: BuildArtifact

  • Application name: Java-Webapp-CodeDeploy-Application

  • Deployment group: Java-Webapp-CodeDeploy-Prod-DG

  • Click Done

  • Click on Done to save changes

  • SCROLL UP and Click on SAVE

  • Click SAVE

19) RE-RUN YOUR PIPELINE and CONFIRM THE APP IS AVAILABLE IN STAGING ENV BEFORE APPROVING PRODUCTION

• CLICK on Release Change

19.1) Test To The Application Running In The Staging Environment

• Navigate to EC2
• Copy the Public IP Addresses of the Stage Instance and Try Accessing the Application
• URL: http://INSTANCE_PUBLIC_IP:8080/javawebapp

REVIEW AND APPROVE PRODUCTION DEPLOYMENT

• Once you Confirm that The Application is working as Expected...
• You Then APPROVE PROD Deployment

20) REVIEW ALL JOBS (Whle The Pipeline Is Running)

• Go through the CodeArtifact Downloaded Dependencies
• Go through the CodeBuild Build & Test Job Outputs
• Go through the CodeDeploy Stage & Prod Prod Deployment Results
• Go through the SonarCloud Project/Analysis etc

20A) CodeArtifact Maven Project Repository

i) CodeArtifact maven-central-store (These Dependencies Where All Downloaded From Maven Central and Stored Here)

ii) CodeBuild Project Logs (Build and Test Jobs)

20B) CodeBuild Build Job Results

20C) CodeBuild SonaCloud SAST Job Results

20D) SonaCloud SAST Test Results

20E) CodeDeploy Deployment Results (Stage&Prod)

• Navigate to CodeDeploy

21) CONFIRM THAT THE APPLICATION VALIDATE TEST PASSED

22) TEST ACCESS TO THE APPLICATION

• Navigate to EC2
• Copy the Public IP Addresses of the Instances and Try Accessing the Application
• URL: http://INSTANCE_PUBLIC_IP:8080/javawebapp

👨‍💻😃 CONGRATULATIONS TEAM!! CONGRATULATIONS TEAM!! 👨‍💻😃