For access requirements go to Prerequisites.
If you want to update and/or build a Docker image, follow the Docker section.
For information about how the Concourse jobs are set up, go to Concourse
If you want to write or amend a Concourse pipeline, follow the Implementing Concourse pipeline section.
For information about the IAM role and policies that have been implemented go to IAM role.
Access to DockerHub - Ask the Engineering team to share credentials for the gdscyber account via LastPass.
Access to csw-concourse GitHub repo
Access to Concourse - Alphagov GitHub single sign-on
Read general info on deploying into PaaS using Concourse
Note: The Docker images that are defined in dockerfiles are now automatically built
using the cyber-security-concourse-base-image pipeline on the RE Concourse. There is no need to
manually build and deploy these images.
If updating the Dockerfiles for these images, remember to also update the tags files according
to Semantic Versioning. For example, if you add a new feature to the
cyber-chalice Dockerfile, but is otherwise backwards compatible, increase the minor version by one
(e.g. 2.1 -> 2.2).
The CSW Concourse implementation has two docker images in the Docker Hub:
-
cyber-chalice
The docker image is a ubuntu 18.04 installation with Python 3.6, Python virtual environment, nodejs and terraform installed.
Dockerfile for chalice:
The command below builds a docker image, which is tagged as latest version, the -t flag also tags it as version 2.1:
docker build --no-cache -t gdscyber/cyber-chalice -t gdscyber/cyber-chalice:2.1 .Check DockerHub for the latest version tag.
Afterwards to push to DockerHub run this command:
docker push gdscyber/cyber-chalice:2.1You also have to separately run either of the following for AWS ECS to pick up that this is the latest version:
docker push gdscyber/cyber-chalice docker push gdscyber/cyber-chalice:latest -
csw-concourse-worker
The docker image is a ubuntu 18.04 installation with Geckodriver and Firefox and Python environment inherited from cyber-chalice image
Dockerfile for csw-concourse-worker:
The command below builds a docker image for the csw-concourse-worker and tags it simultaneously as latest and version 1.3.2:
docker build --no-cache -t gdscyber/csw-concourse-worker -t gdscyber/csw-concourse-worker:1.3.2 .Then to push to DockerHub:
docker push gdscyber/csw-concourse-worker:1.3.2and
docker push gdscyber/csw-concourse-worker:latest or docker push gdscyber/csw-concourse-worker
Except for getsshkey and aws-assume-role, the below are all expect scripts.
-
- Allows the concourse worker to assume a given role to deploy to the staging / production accounts
-
- Accepts arguments and answers to yes/no prompts from the gulp environment.build task. Currently not in use as we are using existing environments.
-
- Accepts the "yes/no" prompt from the
usr/local/bin/gulp environment.deploycommand
- Accepts the "yes/no" prompt from the
-
- Loads ssh keys from SSM into
/root/.ssh/${ENVIRONMENT}& gives it the correct permissions in the docker container
- Loads ssh keys from SSM into
-
- This script takes the arguments provided and loads them into the gulp script prompts.
https://cd.gds-reliability.engineering/teams/cybersecurity-tools/pipelines/csw
-
Deploy job (incl. load)
-
Loads prefix and account ID from settings.json for the specified environment.
-
Does installation of dependencies and runs
aws-assume-role,getsshkey,loadcswanddeploycswscripts as above. -
Notifies Slack #cyber-security-service-health channel on successful exit or failure.
-
uat-deploy job
- task
csw-unit-test- installs and activates virtual environment, installs wheel and requirements-dev.txt. Then runs unittest. - task
csw-uat-deploy- loads 'uat' environment and runs deploy job. - task
csw-e2e-test- runs e2e test on the 'uat' environment ase2etest.user
- task
-
prod-deploy job
- only runs if
uat-deploypassed - task
csw-prod-deploy- runs deploy job in the 'prod' environment.
- only runs if
-
-
Build job - currently NOT IN USE
- The build job starts by installing latest Linux updates and generating RSA key pair.
- It installs and activates virtual environment and package dependencies and runs unittest.
- It then assumes the
concourserole and saves private and public keys to SSM parameter store. - It runs the
buildcswscript and notifies Slack if successful or failing.
-
Destroy job - currently NOT IN USE
- After assuming AWS Concourse role and fetching SSH keys this job activates virtual environment and installs dependencies.
- It runs
loadcswscript thengulp environment.cleanupfor the specified environment and notifies Slack on success/failure.
CSW Concourse pipeline uses variables which are uploaded into Concourse hosting account SSM parameter store. We have hidden from the pipeline code some variables like cyber staging and production account numbers, concourse role name and slack webhook url.
The cyber staging account ID:
```aws ssm put-parameter \
--name "/cd/concourse/pipelines/cybersecurity-tools/cyber-staging" \
--value "103495720024" \
--type SecureString \
--key-id "9044a24d-2e69-4058-ba72-52c43dec4979" \
--overwrite \
--region eu-west-2
```
Concourse role:
```aws ssm put-parameter \
--name "/cd/concourse/pipelines/cybersecurity-tools/cd-role" \
--value "cd-cybersecurity-tools-concourse-worker" \
--type SecureString \
--key-id "9044a24d-2e69-4058-ba72-52c43dec4979" \
--overwrite \
--region eu-west-2
```
Slack webhook configuration:
```
aws ssm put-parameter --cli-input-json '{"Type": "SecureString", "KeyId": "9044a24d-2e6
9-4058-ba72-52c43dec4979", "Name": "/cd/concourse/pipelines/cybersecurity-tools/slack-webhook-cyber", "Value"
: "https://hooks.slack.com/services/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" }' --overwrite --region eu
-west-2
```
The csw Concourse pipeline yaml file can be found at csw-concourse/pipelines/csw-pipeline.yml
Once you've made changes to the yaml file you login to concourse and set your target, in this case "cd":
fly login --target cd -c https://cd.gds-reliability.engineering -n cybersecurity-tools
The following command will create a new pipeline or amend the existing pipeline script:
fly -t cd sp -c csw-pipeline.yml -p csw
We restricted the policy assigned to the Concourse IAM role. It is likely that this will need to change in the future as new functionalities are added.
The policy lives in csw-infra/tools/concourse/main.tf file. To make changes to it you need to:
-
git clone the csw-infra repo
-
cd into tools/concourse
-
get
apply.tfvarsandbackend.tfvarsfiles from csw-configuration/tools/concourse/ and copy them to the csw-infra/tools/concourse folder wheremain.tffile is. -
make the needed changes in
main.tffile. -
amend bucket name in
apply.tfvarsandbackend.tfvarsdepending on whether you are implementing the changes in staging or production. -
then run Terraform as follows for both staging and production accounts:
aws-vault exec <prod or staging account> -- terraform init -backend-config=backend.tfvars -reconfigureaws-vault exec <prod or staging account> -- terraform plan -var-file apply.tfvarsaws-vault exec <prod or staging account> -- terraform apply -var-file apply.tfvars!!! Staging and production should always have the same policy.