CDK construct to deploy Fargate Gitlab runner

Many customers rely on GitLab and Gitlab CI/CD to automate their build, test, and deployment processes. Gitlab CI/CD uses a Gitlab Runner to run jobs in a pipeline. In this project , we use CDK to deploy a Gitlab/CI runner using AWS Fargate a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service.

Solution Architecture

• CDK construct to deploy Fargate Gitlab runner
• Solution Architecture
  • Limitations
• Deployment Guide
  • Requirement
  • Deployment
• Configuration options
  • Examples of advanced usage
    • Building Docker image
      • Deploying
    • Deploy an other task definition
    • Use a managed iam policy for your task_definiton execution role
    • Use a custom inline iam policy for your task_definiton execution role
    • Specify stacks name
• CHANGELOG
• LICENSE

Limitations

• This solution doesn't support Windows workloads

Deployment Guide

Requirement

• In order to deploy the CDK construct you need to have an environment provisioned with Python3.9, Pipenv and Pip3. Check your Python version:

  python --version 
  # Python 3.9.6
  pipenv --version
  # pipenv, version 2021.5.29
  pip3 --version
  # pip 21.1.3

• Install AWS CLI
• Install the AWS CDK version >= 2.0.0
• Install Docker In order to build to Docker images, you need to have Docker engine installed on your machine.
• AWS Account
• An AWS VPC, configure with a least a private subnet with a NAT Gateway attached. (see here to have a exemple Cloudformation stack)

Deployment

• Clone this repository
• Validate prerequisites for CDK here
• Bootstrap your AWS account to use CDK following the AWS guide here

cdk bootstrap aws://YOUR_ACCOUNT_ID/YOUR_REGION

• Export CDK environment variables

  • Linux or MacOS

  export CDK_DEFAULT_ACCOUNT=YOUR_ACCOUNT_ID
  export CDK_DEFAULT_REGION=YOUR_REGION

  • Windows

  setx CDK_DEFAULT_ACCOUNT YOUR_ACCOUNT_ID
  setx CDK_DEFAULT_REGION YOUR_REGION
  

• Enable service linked role for ECS

  If it is your first time using ECS, you may need to manually enable the service linked role.

  aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com

• Create Secret for Gitlab Token

  The Gitlab runner token should be stored in AWS Secret manager as a key/velue, with key=token and value can be retrieved from Gitlab setting from settings/cicd/runners.

  aws secretsmanager create-secret --name GitlabSecretToken \
  --description "Token used to register runner to Gitlab instance" \
  --secret-string '{ "token": "TOKEN_VALUE"}'

• Launch CDK stack

  The cdk application is composed on two stacks : - Main stack that deploys all the necessary resources for gitlab runner , this includes S3 bucket, IAM roles, ECS Cluster, and an autoscaling group for bastion instance. - Task definition stack, this is used to deploy fargate task definitions with the corresponding Docker images that are used in the ci/cd jobs. The docker images are built automticaly by CDK from the directory docker_images and pushed to ECR registry
  

  cd gitlab_ci_fargate_runner

  Copy config/app.yml-example to config/app.yml and replace values in the config file config/app.yml with your environment parameters

  ---
  app_name: Gitlabrunner
  bastion:
      gitlab_runner_version: "14.5.1" 
      mng_min_size: 1 #Default 1
      mng_max_size: 2 #Default number of availibity zones of provided vpc 
      gitlab_server: gitlab.com # modify with gitlab server
      concurrent_jobs: 2 # put here the desired concurent jobs
      gitlab_runner_token_secret_name: my_secret # Put here the name of the gitlab tokensecret name stored in secret manager
      runner_tags: my_tag # put here liset of tags of gitlab runner
      VpcId: vpc-idxxxxxx # Your VpcID
  task_definition:
      gitlab_runner_version: "14.5.1"
      docker_image_name: amazonlinux # put here the defaul docker image to use
  tags: # Put tags as key: value pair
      ProjectName: Demo
      CostCenter: IT

  ⚠️ Mac M1 users ⚠️ you have to add --platform=linux/amd64 after the FROM inside the Dockerfile to be able to build the images on your Mac ARM architecture. Fargate custom executor does currently support using Fargate on ARM. You can configure image in the docker_images folder

  FROM --platform=linux/amd64 amazonlinux:2.0.20210813.1
  ...

  Install python dependencies:

  cd gitlab-ci-fargate-runner
  pipenv install
  # start docker service , it is used to build image

  Validate Cloudformation Template and deploy :

  pipenv run cdk synth  --all 
  pipenv run cdk deploy --all

  This will deploy two stacks, GitlabrunnerBastionStack and {docker_image_name}TaskDefinitionStack. Where docker_image_name is the value set config/app.yml

  You can deploy an other task definition

  • Testing

    Add .gitlab_ci.yml to your project

    Replace value of :

    • FARGATE_TASK_DEFINITION with the value of the task definition id with the format : {docker_image_name}:{version}. An increment of version occurs at each changes of the taskdefiniton.
    • CI_TAGSs with the actual tags you specified in config/app : bastion.runner_tags

    ---
    default:
    tags:
    # This tell Gitlab to use our newly created runners.
    # this sould correspond to runner_tags config/app.yml above
        - CI_TAGS
    
    variables:
    # Add variable FARGATE_TASK_DEFINITION to the task definition revision deployed in the stack above.
    # This variable can also be redefined per job.
        FARGATE_TASK_DEFINITION: "amazonlinux:1"
    stages:         
      - build
      - test
      - deploy
    
    build-job: # This job runs in the build stage, which runs first.
      stage: build
      script:
        - echo "Compiling the code..."
        - echo "Compile complete."
    
    unit-test-job:   # This job runs in the test stage.
      stage: test    # It only starts when the job in the build stage completes successfully.
      script:
        - echo "Running unit tests "
        - echo "Code coverage is XX%"
    
    deploy-job:      # This job runs in the deploy stage.
      stage: deploy  # It only runs when *both* jobs in the test stage complete successfully.
      script:
        - echo "Deploying application..."
        - echo "Application successfully deployed."
    

    After commiting and pushing the code update, the pipline starts and uses the fargate runner to execute the jobs. The docker image used to execute the jobs is the one defined in Fargate task difinition. We can override the docker image, by creating a new task definition and set the variable FARGATE_TASK_DEFINITION in .gitlab-ci.yml.

    

  • Cleanup

    Run the command below to delete the resources that were created.

    • Delete the cdk stack

    pipenv run cdk destroy  --all  

    • Delete the secret created

    aws secretsmanager delete-secret --secret-id GitlabSecretToken \
        --recovery-window-in-days 7

Configuration options

• General

Configuration Key	CDK Context Key	Description	Required	Default value
app_name	-	Name of your application	Yes	-
tags	-	Key: value pair of Tags to apply to deployed resources	No	-

• Bastion

Configuration Key	CDK Context Key	Description	Required	Default value
gitlab_runner_version	-	Version of Gitlab Runner to use	Yes	-
desired_size	-	Number of desired instance Task in Fargate Service	No	1
gitlab_server	-	Host of the Gitlab Server instance	No	gitlab.com
concurrent_jobs	-	Number of concurent jobs	No	1
cpu	-	CPU Taskdefinition parameter see documentation	No	256
memory	-	Memory Taskdefinition parameter see documentation	No	512
gitlab_runner_token_secret_name	-	Name of the gitlab tokensecret name stored in secret manager	Yes	-
log_group_name	-	Name of the LogGroup create in Cloudwatch	No	/Gitlab/Runners/
VpcId	-	VPC Id where the Gitlab Runner will be deployed	Yes	-
stack_name	BastionStackName	Name of the resulting Cloudformation Stack	No	{app_name}BastionStack
runner_tags	-	Tags to add to runners	No	-

• Task Definition

Configuration Key	CDK Context Key	Description	Required	Default value
gitlab_runner_version	-	Version of Gitlab Runner to use	Yes	-
cpu	CPU	CPU Taskdefinition parameter see documentation	No	256
docker_image_name	DockerImageName	Name of the folder of the image (ex : amazonlinux) in docker_images folder	Yes	-
managed_policies	TaskManagedPolicies	Managed IAM policy Name	No	-
memory	Memory	Memory Taskdefinition parameter see documentation	No	512
iam_policy_template	TaskInlinePolicy	Path to inline policy to add to ExecutionTaskRolePolicy	No	-
log_group_name	-	Name of the LogGroup create in Cloudwatch	No	"/Gitlab/TaskDefinitions/{docker_image_name}/"
stack_name	TaskDefinitionStackName	Resulting Cloudformation StackName	No	root

Examples of advanced usage

Building Docker image

You can use GitLab CI/CD with Docker to create Docker images. To be able to build Docker images inside a container, the traditionnal way uses DinD (Docker-in-Docker) which require to run the builder container as privileged. This option is not available on AWS Fargate.

An other option to build Docker image inside a Docker container is to use Kaniko. Kaniko can build Docker image inside a Docker container without the privileged mode. Also, it works out-of-the-box with ECR.

Deploying

The goal here is to build an image with Gitlab-CI and Kaniko and then push it to an ECR repository

Please follow the Deployment Guide before the instructions below.

• Deploy the Kaniko task definition:

pipenv run cdk deploy -c DockerImageName=kaniko -c TaskManagedPolicies=AmazonEC2ContainerRegistryPowerUser kanikoTaskDefinitionStack

• Create an ECR repo

aws ecr create-repository \
  --repository-name kaniko-builder

• Configure your Git repository

  • In your Gitlab repository, create a directory named build and add your Dockerfile and docker-entrypoint.sh (you can use our example in docker_images.

  • Add this .gitlab-ci.yml at the root of your repo

  • Replace value of :

    • <AWS_ACCOUNT_ID> by the Account ID of your AWS account
    • <AWS_REGION> and the region where you created your ECR repository.
    • <CI_TAGS> with the actual tags you specified in config/app : bastion.runner_tags

---
default:
  tags:
  # This tell Gitlab to use our newly created runners.
  # this sould correspond to runner_tags config/app.yml above
    - CI_TAGS

variables:
  # Add variable FARGATE_TASK_DEFINITION to the task definition revision deployed in the stack above.
  # This variable can also be redefined per job.
  FARGATE_TASK_DEFINITION: "kaniko:1"

stages:         
  - build

docker_builder:
  stage: build
  variables:
    ENTRYPOINT: $CI_PROJECT_DIR/build
    IMAGE_TAG: myimage-$CI_COMMIT_SHORT_SHA
    REPO: <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/kaniko-builder
  before_script:
    - echo creating docker config for ecr
    - mkdir -p /root/.docker
    - echo '{"credsStore":"ecr-login"}' > /root/.docker/config.json
    - export PATH=$PATH:/kaniko
    # Source the AWS env vars to retrieve credentials use to push image to ECR
    - source /etc/profile.d/set-aws-env-vars.sh
  script:
    - echo "build image and push"
    - /kaniko/executor --context $ENTRYPOINT --dockerfile $ENTRYPOINT/Dockerfile --destination $REPO:$IMAGE_TAG

• Cleanup

  • Delete kaniko stack

  aws cloudformation delete-stack \
  --stack-name kanikoTaskDefinitionStack

  • Delete Kaniko ecr repo

  aws ecr delete-repository \
      --repository-name kaniko-builder \
      --force

Deploy an other task definition

You can deploy more task definitions stacks by running:

pipenv run cdk deploy -c DockerImageName=docker_image_name -c Memory=1024 -c CPU=512  {docker_image_name}TaskDefinitionStack

Where docker_image_name to the name of the docker image found in the directory docker_images

Use a managed iam policy for your task_definiton execution role

You can use the context variable TaskManagedPolicies or as a parameter task_definition.managed_policy in your app.config file

pipenv run cdk deploy -c DockerImageName=kaniko -c Memory=1024 -c CPU=512 -c TaskManagedPolicies=AmazonEC2ContainerRegistryPowerUser

Use a custom inline iam policy for your task_definiton execution role

pipenv run cdk deploy -c DockerImageName=python -c TaskInlinePolicy="./config/example_policy.json.j2"

Specify stacks name

• Deploy both stack

pipenv run cdk deploy -c BastionStackName="MY-CUSTOM-RUNNER" -c TaskDefinitionStackName="MyCustomImageTaskDefinition" --all

• Deploy only one stack

export StackName="MY-CUSTOM-RUNNER" 
pipenv run cdk deploy -c BastionStackName=$StackName $StackName

CHANGELOG

See the CHANGELOG file.

LICENSE

This project is licensed under the MIT-0 License. See the LICENSE file.