Skip to content
This repository was archived by the owner on May 25, 2022. It is now read-only.

Configuring CI CD for a project

Jump to bottom
Luis Sifu edited this page Mar 12, 2020 · 20 revisions

Setting up a project's CI/CD is easy, you just have to follow these steps:

  1. Create a Dockerfile
  2. Setup docker-compose
  3. Add gitlab-ci configuration
  4. Configure github & gitlab to use the pipeline
  5. Add CD with a Platform

1. Create a Dockerfile

Docker is a container system used by many, and is needed to simplify our test environment.

Creating a Dockerfile consist on detailing the steps needed to build our application and end up with a working environment.

Some general guidelines:

  • General structure:
    • Base image
    • Environment variables
    • Dependencies
    • Source
    • Start the application
  • Order is important! Each line has a cache, so only when a file changes the next steps are rerun, the rest use the cache. So put the most likely to change files last.
  • If your application has compiled files, consider using multi stage builds to reduce the size of your images.

Here is an example of a Dockerfile

# Base image to use, generally each language has an image already
# Alpine versions of images are smaller than their counterparts, 
# use them if you can.
# Use specific versions on your images, incompatibilities may 
# show up if you always use the latest version.
FROM elixir:1-8-alpine

# Environment variables should only be declared here only if
# they are needed to build the image and passed using ARG, 
# otherwise save them for when you actually start up the container.
ARG MIX_ENV
ENV MIX_ENV $MIX_ENV

WORKDIR /home/apollo

RUN mix local.hex --force
RUN mix local.rebar --force

# Fetch dependencies and other tools you'll need to build your application.
# Copy the files and run the command to fetch them.
# Sometimes your build requires fetching packages not provided by the
# base image, check your base image packet manager and use it to fetch them.
RUN apk --no-cache add --virtual native-deps \
  g++ gcc libgcc libstdc++ linux-headers autoconf automake make nasm python git

COPY mix.exs .
COPY mix.lock .

RUN mix deps.get

# Copy the source files of your application, as the configuration files are 
# less likely to change than the code, so put those first.
# Copy only the files you actually need to run (or test) the application, for
# example, you don't need to copy the Dockerfile into a container.
COPY .formatter.exs .
COPY config ./config
COPY priv ./priv
COPY test ./test
COPY lib ./lib

# Ensure your project is built before starting it, even if it your "start" 
# command does automatically. Prevents clogging of the log when starting, 
# and other  funky business when building a lot of images.
RUN mix compile

# Run the application, CMD is preferred over ENTRYPOINT as it 
# can be substituted more easily if needed (for example in tests).
CMD ["mix" "phx.server"]

You can read more on Dockerfiles in the Dockerfile Reference documentation

Some examples of Dockerfiles:

2. Setup docker-compose

Once the core of the application is dockerized, you also need to add the different services that the application needs to be able to function, such as the Database. For that, we use docker compose to start all of the services and link them together.

Docker compose uses a docker-compose.yml file to describe which services to start together with our application, as well as which ports, environment variables to setup.

Here is an example of a docker-compose.yml:

version: '3'
# Each key under services represents the name of the
# service docker-compose will assign to it as well
# as a container that will be started.
services:
  db:
    # remember to be specific on the version!
    image: postgres:11.3-alpine
    ports:
      - "5432:5432"
  elastic:
    image: elasticsearch6.8.0
    ports:
      - "9200:9200"
      - "9300:9300"
  api:
    # if you add "image" to the core service, it will
    # name and tag the image as the string provided.
    # compose also supports the ${VAR} syntax to
    # replace env vars, adding `:-default` sets up a
    # default value to the var, ie ${VAR:-default}
    image: "polaris:${DOCKER_TAG:-latest}"
    # Adding build makes it so compose takes our
    # Dockerfile and builds it instead of downloading
    # a preexisting image .
    build:
      context: ./
      # Supplies the ARG to the Dockerfile
      args:
        - MIX_ENV=${MIV_ENV:-dev}
    # Work around so that you can use "localhost"
    # in your application config.
    # WARNING: does not work on Mac
    network_mode: host
    ports:
      - "4000:4000"
    # Setups up the env vars for the container.
    # You can exec into it and check their values
    # if you want to see their values.
    environment:
      - MIX_ENV=${MIX_ENV:-dev}
      - DB_USER=${DB_USER}
      - DB_PASSWORD=${DB_PASSWORD}
      - DB_NAME=${DB_NAME}
      - DB_HOST=${DB_HOST}
      - DB_PORT=${DB_PORT}
      - DB_POOL_SIZE=${DB_POOL_SIZE}
      - ELASTIC_URL=${ELASTIC_URL}
    # Specifies the order or services to be run, ie
    # waits for DB and Elastic to be up (not ready!)
    # before starting the API.
    depends_on:
      - db
      - elastic

You can read up more on what you can use on the docker-compose.yml in the Compose reference documentation

Some examples of docker-compose.yml files:

3. Add gitlab-ci configuration

After the whole application is ready, you can then add a Pipeline using gitlab.

Making the Pipeline

A .gitlab-ci.yml file describes how each "stage" of your pipeline will work, generally we have 3 stages:

  • build
  • test
  • deploy

The "build" stage consist on building our previously defined image to be ready to be used on the rest of the pipeline.

The "test" stage runs the test suite of our application inside the container of the image we built. It should include all of the tools used, such as static code analyzers, linters and the actual test suites.

The "deploy" stage pushes our image to the Platform we use for Continuous Delivery and updates the Production (or Staging) environment with our latests changes. We will skip for now setting up this stage until we actually need to have one.

Heres an example of a .gitlab-ci.yml:

# Stages to use in our pipeline, note that
# "deploy" is disabled for now.
stages:
  - build
  - test
  # - deploy

# name of the Job
build_image:
  # in which stage to run on
  stage: build
  # ENV variables to be setup,
  # certain variables are available on
  # the CI, such as CI_COMMIT_SHORT_SHA
  # that can be used on our pipeline.
  variables:
    MIX_ENV: 'test'
    DOCKER_TAG: '$CI_COMMIT_SHORT_SHA'
  # The actual commands to run our application, 
  # you can also define a "before_script" for setup
  # and a "after_script" for cleanup as seen below.
  script:
    - "docker-compose build"
  # Specifies on which conditions to run this job
  # for example only when not in these two branches.
  except:
    - develop
    - master

run_tests:
  stage: test
  variables:
    MIX_ENV: 'test'
    DOCKER_TAG: '$CI_COMMIT_SHORT_SHA'
  # As hinted before, docker-compose up starts
  # all of the services, but does not wait for
  # them to be ready. The DB can be up but still
  # initializing when we ask it for requests, so
  # our pipeline may fail. To prevent this you 
  # can add a script to wait for the db to be ready
  # before starting the actual script.
  before_script:
    - "docker-compose up -d"
    - "./wait-for.sh db pg_isready"
  script:
    - "docker-compose exec -T api mix ecto.create"
    - "docker-compose exec -T api mix ecto.migrate"
    - "docker-compose exec -T api mix test"
  # After script acts as a finally clause, so even if
  # part of a script fails, this one will always be run.
  after_script:
    - "docker-compose down"

You can read further on the Gitlab Pipeline Reference for more information on what can you add to this file, as well as the full list of ENV variables provided by the runners that can be used on the pipeline.

Here's a reference implementation of the wait-for script:

#!/bin/bash

TRIES=0
MAX_TRIES=60
SERVICE=$1
shift
CMD=$@

until docker-compose exec -T $SERVICE $CMD; do
  TRIES=$((TRIES + 1))

  if [ $TRIES -eq $MAX_TRIES ]; then
    echo "Timed out after $TRIES tries"
    exit 1
  fi

  sleep 1
done

exit 0

Some examples of .gitlab-ci.yml:

Running the Pipeline

To test your pipeline, you have to have your repo on gitlab.

If your repo is on github, you'll have to mirror it.

  1. Go to the new project page on gitlab
  2. Click on "Import project"
  3. Click on "Github"
  4. List your Github Repositories
  5. Find the Repo you want to import from the list
    • Make sure the dropdown has "ecaresoft" instead of your username
  6. Click import
  7. After the project is imported, go to the Repo page
  8. Click on "Settings" > "Repository"
  9. Expand the "Mirroring repositories" tab
  10. Create a new pull mirror
    • Enter the repo url as: https://USERNAME@github.com/ecaresoft/REPO.git replacing USERNAME & REPO with their appropiate values
    • Select "Pull"
    • Select "Password"
    • Enter the password corresponding to the USERNAME
    • Set mirror user as your user
    • Check both "Overwrite diverged branches" & "Trigger pipelines for mirror updates" boxes
    • Click mirror repository

After your repo is on Gitlab, you'll have to configure the Runner to run the pipeline.

  1. Go to your Repo page
  2. Click on "Settings" > "CI/CD"
  3. Expand the "Runners" tab
  4. Disable "Shared Runners"
  5. Enable 1 of our runners on the left.

To trigger the pipeline and see the results of your work, push to the Repo (to a branch that contains a .gitlab-ci.yml) and see the Job being run

If you are using a Mirror, Gitlab takes a while to make the Pull of your changes, you can trigger it manually by clicking on the "Settings" > "Repo" > "Mirroring Repositories" and then click the "Refresh Icon" next to the pull mirror.

On the next step we'll see how to configure it so pipelines are triggered automatically when a push to github happens.

4. Configure github & gitlab to use the pipeline

If you are using github, you can configure it to integrate seamlessly with the Gitlab CI Pipeline.

Configure gitlab integration on Github

  1. Head over to the Repo page on gitlab
  2. Click on "Settings" > "Integrations"
  3. Under "Project services", select "Github"
  4. Fill out the form
    • Check "active"
    • Enter your github personal access token
    • Enter the url of your project, which should look something like this: https://github.com/ecaresoft/REPO, replacing REPO for the name of the project
    • Enable "Static status check names"
    • Click on "Test settings and save changes"

Trigger mirror pulls when pushing to Github

  1. Find out the Gitlab repo id:
    • Go to your projects' main page ("Project" > "Details")
    • Copy the id found under the name, it should look like this: * Project ID: 12847555
  2. Head over to Repo page on Github
  3. Click on "Settings" > "Webhooks"
  4. Click "Add Webhook"
  5. Fill out the form
    • Payload url should be like: https://gitlab.com/api/v4/projects/PROJECT_ID/mirror/pull?private_token=TOKEN, replacing PROJECT_ID for the Gitlab Repo id & TOKEN for your personal access token.
    • Leave "Content type" & "Secret" as is
    • Under "Which events would you like to trigger this webhook?", select "Let me select individual events."
    • Enable "Pull requests" only, disable "Pushes" if enabled
    • Leave "Active" checked
    • Click on "Add Webhook"

5. Add CD with a Platform

To add a Continuous delivery you'll need to setup the project on a Platform and update the .gitlab-ci.yml with the deploy instructions.

Setting up a Platform

To create new infrastructure, refer to the Infra Repo documentation

Updating gitlab pipeline to deploy

After the project has been setup, you should have a set of steps needed to deploy to it using CLI tool. This steps should be added to your projects' CI configuration so that they can be deployed automatically when integrating code into the production ready branches.

A production ready environment usually has different set of instructions when building & testing the application which result in custom job for each stage. Gitlab CI has the ability to support this via configuration in the .gitlab-ci.yml, like setting up the except and only clauses in a jobs' description.

stages:
  - build
  - test
  - deploy

build_image:
  stage: build
  variables:
    MIX_ENV: 'test'
    DOCKER_TAG: '$CI_COMMIT_SHORT_SHA'
  script:
    - "docker-compose build"
  # Note the except here, even if you have matching stages
  # this job will only run for non production ready branches.
  except:
    - develop
    - master
  
build_staging_image:
  stage: build
  variables:
    MIX_ENV: 'prod'
    DOCKER_TAG: 'staging'
  # These are added steps to be able to deploy to AWS, if your
  # application or platform need more steps on the build phase
  # you should add them here.
  before_script:
    - "$(aws ecr get-login --no-include-email --region us-west-2)"
  script:
    - "docker-compose build"
    - "docker tag polaris:staging $AWS_CONTAINER_REGISTRY:staging"
  after_script:
    - "docker logout $AWS_CONTAINER_REGISTRY"
  # Enable this job only for develop, this should match
  # against the default configuration (except: develop, master).
  only:
    - develop

# If you have another environment, you can add another job
# to build its image and then use it in another stage. 
# This is an example for the "production" env, you can
# compare it to the "staging" one above.
build_production_image:
  stage: build
  variables:
    MIX_ENV: 'prod'
    # Note the difference here
    DOCKER_TAG: 'production'
  before_script:
    - "$(aws ecr get-login --no-include-email --region us-west-2)"
  script:
    - "docker-compose build"
    - "docker tag polaris:staging $AWS_CONTAINER_REGISTRY:staging"
  after_script:
    - "docker logout $AWS_CONTAINER_REGISTRY"
  only:
    # And here
    - master

# The different between test stage are omitted. Only the 
# deploy example is here, but you can use the examples as
# reference on how to setup them up.
deploy_staging_image:
  stage: deployment
  before_script:
    - "$(aws ecr get-login --no-include-email --region us-west-2)"
  # Heres the example script to deploy to AWS Fargate
  # but this should match your platform CLI tool script
  # for deploying applications.
  script:
    - "docker push $AWS_CONTAINER_REGISTRY:staging"
    - "ecs-cli configure --cluster cirrus --region us-west-2 --default-launch-type 
      FARGATE --config-name cirrus"
    - "ecs-cli compose --project-name polaris-staging --file fargate-compose-staging.yml 
      service up --timeout 30 --cluster-config cirrus --force-deployment"
  after_script:
    - "docker logout $AWS_CONTAINER_REGISTRY"
  only:
    - develop

After you configure this, whenever you merge into a production ready branch a specific set of steps are run and then automatically deploys into your environment, completing the CI/CD cycle.

Clone this wiki locally